Environment Handlers

Minecraft is an extremely complex environment which provides players with visual, auditory, and informational observation of many complex data types. Furthermore, players interact with Minecraft using more than just embodied actions: players can craft, build, destroy, smelt, enchant, manage their inventory, and even communicate with other players via a text chat.

To provide a unified interface with which agents can obtain and perform similar observations and actions as players, we have provided first-class for support for this multi-modality in the environment: the observation and action spaces of environments are gym.spaces.Dict spaces. These observation and action dictionaries are comprised of individual fields we call handlers.

Note

In the documentation of every environment we provide a listing of the exact gym.space of the observations returned by and actions expected by the environment’s step function. We are slowly building documentation for these handlers, and you can click those highlighted with blue for more information!

Spaces

Enum Spaces

Some observation and action spaces are Enum types. Examples include the equip observation and the equip action.

Observation and action spaces that are Enum are encoded as strings by default (e.g. “none”, “log”, and “sandstone#2”) when they are returned from env.step() and env.reset(), or yielded from minerl.data.DataPipeline.batch_iter().

When building an action to pass into env.step(act), the Enum component of the action dict can be encoded as either a string or an integer.

Tip

The Enum integer value that corresponds to each Enum string value can be accessed via Enum.values_map[string_value]. For example, to get the integer value corresponding to the equip action “dirt” in MineRLObtainDiamond or MineRLBasaltBuildVillageHouse, you can call env.action_space.spaces["equip"].values_map["dirt"].

Observations

Visual Observations - pov, third-person

pov : Box(width, height, nchannels)

An RGB image observation of the agent’s first-person perspective.

Type

np.uint8

third-person : Box(width, height, nchannels)

An RGB image observation of the agent’s third-person perspective.

Warning

This observation is not yet supported by any environment.

Type

np.uint8

compass-observation : Box(1)

The current position of the minecraft:compass object from 0 (behind agent left) to 0.5 in front of agent to 1 (behind agent right)

Note

This observation uses the default Minecraft game logic which includes compass needle momentum. As such it may change even when the agent has stoped moving!

Equip Observations - equipped_items

equipped_items.mainhand.type : Enum('none', 'air', ..., 'other'))

This observation is an Enum type. See Enum Spaces for more information.

The type of the item that the player has equipped in the mainhand slot. If the mainhand slot is empty then the value is ‘air’. If the mainhand slot contains an item not inside this observation space, then the value is ‘other’.

Type

np.int64

Shape

[1]

Actions

Camera Control - camera

camera : Box(2) [delta_pitch, delta_yaw]

This action changes the orientation of the agent’s head by the corresponding number of degrees. When the pov observation is available, the camera changes its orientation pitch by the first component and its yaw by the second component. Both delta_pitch and delta_yaw are limited to [-180, 180] inclusive

Type

np.float32

Shape

[2]

attack : Discrete(1) [attack]

This action causes the agent to attack.

Type

np.float32

Shape

[1]

Tool Control - equip and use

equip : Enum('none', 'air', ..., 'other'))

This is action is an Enum type. See Enum Spaces for more information.

This action equips the first instance of the specified item from the agents inventory to the main hand if the specified item is present, otherwise does nothing. air matches any empty slot in an agent’s inventory and functions as an un-equip, or equip-nothing action.

Type

np.int64

Shape

[1]

Note

equip 'none' and equip 'other' are both no-op actions. In other words, they leave the currently equipped item unchanged. However, in the MineRL dataset, other takes on a special meaning. other is the wildcard equip action that is recorded in the dataset whenever a player equipped an item that wasn’t included in this action space’s Enum.

Warning

env.step(act) typically will not process the equip action for two ticks (i.e., you will not see the observation value equipped_items change until two more calls to env.step.)

This is due to a limitation with the current version of Malmo, our Minecraft backend.

use : Discrete(1) [use]

This action is equivalent to right-clicking in Minecraft. It causes the agent to use the item it is holding in the mainhand slot, or to open doors or gates when it is facing an applicable Minecraft structure.

Type

np.int64

Shape

[1]