The Android XR SDK is now available in Developer Preview. We want your feedback! Visit our support page to reach out.

XR_ANDROID_composition_layer_passthrough_mesh OpenXR extension

Name String

XR_ANDROID_composition_layer_passthrough_mesh

Extension Type

Instance extension

Registered Extension Number

463

Revision

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2024-09-18

IP Status

No known IP claims.

Contributors

Grant Yoshida, Google

Kevin Moule, Google

Vasiliy Baranov, Google

Peter Chen, Google

Levana Chen, Google

Overview

For devices that support multiple environment blend modes, the system may provide passthrough configurations to show a user their physical environment from an immersive view.

This extension enables applications to project passthrough textures onto arbitrary geometry through an additional composition layer XrCompositionLayerPassthroughANDROID.

The passthrough layer characteristics are specified by the following parameters, in which the projection is represented by XrPassthroughLayerANDROID.

  XrPosef                      pose;
    XrVector3f                   scale;
    float                        opacity;
    XrPassthroughLayerANDROID    layer;

For full screen passthrough, applications can use Environment Blend Mode.

Inspect system capability

An application can inspect whether the system is capable of composition layer passthrough mesh by chaining an XrSystemPassthroughLayerPropertiesANDROID structure to the XrSystemProperties when calling xrGetSystemProperties.

typedef struct XrSystemPassthroughLayerPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsPassthroughLayer;
    uint32_t           maxMeshIndexCount;
    uint32_t           maxMeshVertexCount;
} XrSystemPassthroughLayerPropertiesANDROID;

Member Descriptions

type is the XrStructureType of this structure.
next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
supportsPassthroughLayer is an XrBool32, indicating if the current system supports composition layer passthrough mesh.
maxMeshIndexCount is a uint32_t returns the maximum count of indices that will be accepted for a passthrough mesh.
maxMeshVertexCount is a uint32_t returns the maximum count of vertices that will be accepted for a passthrough mesh.

If supportsPassthroughLayer returns XR_FALSE, the system does not support composition layer passthrough mesh, and therefore will receive XR_ERROR_FEATURE_UNSUPPORTED from xrCreatePassthroughLayerANDROID. The application should avoid using composition layer passthrough mesh when supportsPassthroughLayer is XR_FALSE.

If supportsPassthroughLayer returns XR_TRUE, the system supports composition layer passthrough mesh. In this case, maxMeshIndexCount and maxMeshVertexCount will return a non-zero number. An application should use maxMeshIndexCount and maxMeshVertexCount as the maximum values to set passthrough meshes when calling xrCreatePassthroughLayerANDROID and xrSetPassthroughLayerMeshANDROID, otherwise XR_ERROR_MESH_DATA_LIMIT_EXCEEDED_ANDROID may be returned to indicate the mesh data exceeds the supported limit.

Valid Usage (Implicit)

The XR_ANDROID_composition_layer_passthrough_mesh extension must be enabled prior to using XrSystemPassthroughLayerPropertiesANDROID
type must be XR_TYPE_SYSTEM_PASSTHROUGH_LAYER_PROPERTIES_ANDROID
next must be NULL or a valid pointer to the next structure in a structure chain

Passthrough Layer Composition

The XrCompositionLayerPassthroughANDROID contains the information needed to render a passthrough texture onto a triangle mesh when calling xrEndFrame. XrCompositionLayerPassthroughANDROID is an alias type for the base struct XrCompositionLayerBaseHeader used in XrFrameEndInfo.

typedef struct XrCompositionLayerPassthroughANDROID {
    XrStructureType              type;
    const void*                  next;
    XrCompositionLayerFlags      layerFlags;
    XrSpace                      space;
    XrPosef                      pose;
    XrVector3f                   scale;
    float                        opacity;
    XrPassthroughLayerANDROID    layer;
} XrCompositionLayerPassthroughANDROID;

Member Descriptions

type is the XrStructureType of this structure.
next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
layerFlags is a bitmask of XrCompositionLayerFlags describing flags to apply to the layer.
space is the XrSpace in which the pose of the layer mesh is evaluated over time.
pose is an XrPosef defining the position and orientation of the layer mesh in the reference frame of the space.
scale is an XrVector3f defining the scale of the layer mesh.
opacity is a float defining the opacity of the passthrough texture in range [0, 1].
layer is the XrPassthroughLayerANDROID previously created by xrCreatePassthroughLayerANDROID.

The application can create an XrCompositionLayerPassthroughANDROID structure with the created layer and the corresponding meshes provided by XrPassthroughLayerMeshANDROID.

A pointer to XrCompositionLayerPassthroughANDROID may be submitted in xrEndFrame as a pointer to the base structure XrCompositionLayerBaseHeader, in the chosen layer order, to request the runtime to composite a passthrough layer into the final frame output.

Valid Usage (Implicit)

The XR_ANDROID_composition_layer_passthrough_mesh extension must be enabled prior to using XrCompositionLayerPassthroughANDROID
type must be XR_TYPE_COMPOSITION_LAYER_PASSTHROUGH_ANDROID
next must be NULL or a valid pointer to the next structure in a structure chain
layerFlags must be 0 or a valid combination of XrCompositionLayerFlagBits values
space must be a valid XrSpace handle
layer must be a valid XrPassthroughLayerANDROID handle
Both of layer and space must have been created, allocated, or retrieved from the same XrSession

Create a passthrough layer handle

The XrPassthroughLayerANDROID handle represents a passthrough layer which defines the behavior of XrCompositionLayerPassthroughANDROID.

XR_DEFINE_HANDLE(XrPassthroughLayerANDROID)

An application can create an XrPassthroughLayerANDROID handle by calling xrCreatePassthroughLayerANDROID. The returned XrPassthroughLayerANDROID handle can be subsequently used in API calls.

XrResult xrCreatePassthroughLayerANDROID(
    XrSession                                   session,
    const XrPassthroughLayerCreateInfoANDROID*  createInfo,
    XrPassthroughLayerANDROID*                  layer);

Parameter Descriptions

session is an XrSession in which the passthrough layer will be created for.
createInfo is a pointer to an XrPassthroughLayerCreateInfoANDROID structure specifying initial passthrough layer parameters. This field can also be chained to an XrPassthroughLayerMeshANDROID structure to set the mesh at the same time.
layer is a pointer to a handle in which the created XrPassthroughLayerANDROID is returned.

The application should specify the number of passthrough mesh indices in the XrPassthroughLayerCreateInfoANDROID::vertexCapacity and XrPassthroughLayerCreateInfoANDROID::indexCapacity less than or equal to the maximum values returned by XrSystemPassthroughLayerPropertiesANDROID::maxMeshIndexCount and XrSystemPassthroughLayerPropertiesANDROID::maxMeshVertexCount when calling xrGetSystemProperties. xrCreatePassthroughLayerANDROID will return a XR_ERROR_MESH_DATA_LIMIT_EXCEEDED_ANDROID error if the count of mesh indices defined bycreateInfo is greater than the maximum values.

The XrPassthroughLayerANDROID handle must be eventually freed using the xrDestroyPassthroughLayerANDROID function.

Valid Usage (Implicit)

The XR_ANDROID_composition_layer_passthrough_mesh extension must be enabled prior to calling xrCreatePassthroughLayerANDROID
session must be a valid XrSession handle
createInfo must be a pointer to a valid XrPassthroughLayerCreateInfoANDROID structure
layer must be a pointer to an XrPassthroughLayerANDROID handle

Return Codes

Success

XR_SUCCESS
XR_SESSION_LOSS_PENDING

Failure

XR_ERROR_FUNCTION_UNSUPPORTED
XR_ERROR_VALIDATION_FAILURE
XR_ERROR_RUNTIME_FAILURE
XR_ERROR_HANDLE_INVALID
XR_ERROR_INSTANCE_LOST
XR_ERROR_SESSION_LOST
XR_ERROR_OUT_OF_MEMORY
XR_ERROR_LIMIT_REACHED
XR_ERROR_SIZE_INSUFFICIENT
XR_ERROR_MESH_DATA_LIMIT_EXCEEDED_ANDROID

The XrPassthroughLayerCreateInfoANDROID structure is defined as:

typedef struct XrPassthroughLayerCreateInfoANDROID {
    XrStructureType    type;
    const void*        next;
    uint32_t           vertexCapacity;
    uint32_t           indexCapacity;
} XrPassthroughLayerCreateInfoANDROID;

Member Descriptions

type is the XrStructureType of this structure.
next is NULL or a pointer to the next structure in a structure chain. XrPassthroughLayerMeshANDROID can be provided in the next chain to specify an initial mesh for the passthrough layer when calling xrCreatePassthroughLayerANDROID.
vertexCapacity is an uint32_t representing the maximum capacity of the vertex buffer for this layer's mesh, or 0 if unspecified. If specified, the XrPassthroughLayerMeshANDROID::vertexCount of any mesh set for this layer must be less than or equal to the vertexCapacity.
indexCapacity is an uint32_t representing the maximum capacity of the index buffer for this layer's mesh, or 0 if unspecified. If specified, the XrPassthroughLayerMeshANDROID::indexCount of any mesh set for this layer must be less than or equal to the indexCapacity.

Valid Usage (Implicit)

The XR_ANDROID_composition_layer_passthrough_mesh extension must be enabled prior to using XrPassthroughLayerCreateInfoANDROID
type must be XR_TYPE_PASSTHROUGH_LAYER_CREATE_INFO_ANDROID
next must be NULL or a valid pointer to the next structure in a structure chain. See also: XrPassthroughLayerMeshANDROID

An application can use xrDestroyPassthroughLayerANDROID function to release the passthrough layer and the underlying resources.

XrResult xrDestroyPassthroughLayerANDROID(
    XrPassthroughLayerANDROID                   layer);

Parameter Descriptions

layer is the XrPassthroughLayerANDROID to be destroyed.

Valid Usage (Implicit)

The XR_ANDROID_composition_layer_passthrough_mesh extension must be enabled prior to calling xrDestroyPassthroughLayerANDROID
layer must be a valid XrPassthroughLayerANDROID handle

Thread Safety

Access to layer, and any child handles, must be externally synchronized

Return Codes

Success

XR_SUCCESS

Failure

XR_ERROR_FUNCTION_UNSUPPORTED
XR_ERROR_RUNTIME_FAILURE
XR_ERROR_HANDLE_INVALID

Set passthrough layer mesh

An application can use xrSetPassthroughLayerMeshANDROID function to set the mesh for a passthrough layer.

XrResult xrSetPassthroughLayerMeshANDROID(
    XrPassthroughLayerANDROID                   layer,
    const XrPassthroughLayerMeshANDROID*        mesh);

Parameter Descriptions

layer is an XrPassthroughLayerANDROID handle in which to update with the given mesh.
- mesh is a pointer to an XrPassthroughLayerMeshANDROID structure specifying the information of the mesh.

The application should specify the number of passthrough mesh indices in the XrPassthroughLayerMeshANDROID::vertexCount and XrPassthroughLayerMeshANDROID::indexCount less than or equal to the maximum values returned by XrSystemPassthroughLayerPropertiesANDROID::maxMeshIndexCount and XrSystemPassthroughLayerPropertiesANDROID::maxMeshVertexCount when calling xrGetSystemProperties. If the count of mesh indices given by mesh from xrSetPassthroughLayerMeshANDROID is greater than the maximum values, then XR_ERROR_MESH_DATA_LIMIT_EXCEEDED_ANDROID will be returned.

If the mesh buffer capacity is specified by the XrPassthroughLayerCreateInfoANDROID::vertexCapacity and XrPassthroughLayerCreateInfoANDROID::indexCapacity when creating the layer using xrCreatePassthroughLayerANDROID, then the XR_ERROR_SIZE_INSUFFICIENT error will be returned on xrSetPassthroughLayerMeshANDROID if the count of mesh indices defined by mesh is greater than the capacity.

Valid Usage (Implicit)

The XR_ANDROID_composition_layer_passthrough_mesh extension must be enabled prior to calling xrSetPassthroughLayerMeshANDROID
layer must be a valid XrPassthroughLayerANDROID handle
mesh must be a pointer to a valid XrPassthroughLayerMeshANDROID structure

Return Codes

Success

XR_SUCCESS
XR_SESSION_LOSS_PENDING

Failure

XR_ERROR_FUNCTION_UNSUPPORTED
XR_ERROR_VALIDATION_FAILURE
XR_ERROR_RUNTIME_FAILURE
XR_ERROR_HANDLE_INVALID
XR_ERROR_INSTANCE_LOST
XR_ERROR_SESSION_LOST
XR_ERROR_OUT_OF_MEMORY
XR_ERROR_LIMIT_REACHED
XR_ERROR_SIZE_INSUFFICIENT
XR_ERROR_MESH_DATA_LIMIT_EXCEEDED_ANDROID

The XrPassthroughLayerMeshANDROID structure is defined as:

typedef struct XrPassthroughLayerMeshANDROID {
    XrStructureType          type;
    const void*              next;
    XrWindingOrderANDROID    windingOrder;
    uint32_t                 vertexCount;
    const XrVector3f*        vertices;
    uint32_t                 indexCount;
    const uint16_t*          indices;
} XrPassthroughLayerMeshANDROID;

Member Descriptions

type is the XrStructureType of this structure.
next is NULL or a pointer to the next structure in a structure chain.
windingOrder is the XrWindingOrderANDROID of the mesh triangles, which will be used for backface culling when rendering the mesh.
vertexCount is an uint32_t representing the number of vertices in the mesh. When XrPassthroughLayerCreateInfoANDROID::vertexCapacity is specified, the vertexCount must be less than or equal to the vertexCapacity.
- vertices is a pointer to an array of XrVector3f which contains the vertex positions of the triangle mesh.
indexCount is an uint32_t representing the number of indices in the triangle mesh. The last indexCount % 3 indices, if any, won't be drawn. When XrPassthroughLayerCreateInfoANDROID::indexCapacity is specified, the indexCount must be less than or equal to the indexCapacity.
indices is a pointer to an array of uint16_t which contains the indices of the triangle mesh.

Valid Usage (Implicit)

The XR_ANDROID_composition_layer_passthrough_mesh extension must be enabled prior to using XrPassthroughLayerMeshANDROID
type must be XR_TYPE_PASSTHROUGH_LAYER_MESH_ANDROID
next must be NULL or a valid pointer to the next structure in a structure chain
windingOrder must be a valid XrWindingOrderANDROID value
If vertexCount is not 0, vertices must be a pointer to an array of vertexCount XrVector3f structures
If indexCount is not 0, indices must be a pointer to an array of indexCount uint16_t values

The XrWindingOrderANDROID enumeration identifies the winding order of a mesh's triangles, used by the runtime for backface culling when rendering the mesh of the passthrough layer.

typedef enum XrWindingOrderANDROID {
    XR_WINDING_ORDER_UNKNOWN_ANDROID = 0,
    XR_WINDING_ORDER_CW_ANDROID = 1,
    XR_WINDING_ORDER_CCW_ANDROID = 2
} XrWindingOrderANDROID;

Enumerant Descriptions

XR_WINDING_ORDER_UNKNOWN_ANDROID — Winding order of the mesh's triangles is not known.
XR_WINDING_ORDER_CW_ANDROID — Winding order of the mesh's triangles is clockwise.
XR_WINDING_ORDER_CCW_ANDROID — Winding order of the mesh's triangles is counter-clockwise.

Example code for passthrough layer composition

The following example code demonstrates how to create a passthrough layer and use it in compositing.

XrInstance instance; // previously initialized
XrSystemId systemId; // previously initialized
XrSession session; // previously initialized
XrSpace space; // previously initialized

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreatePassthroughLayerANDROID xrCreatePassthroughLayerANDROID; // previously initialized
PFN_xrDestroyPassthroughLayerANDROID xrDestroyPassthroughLayerANDROID; // previously initialized
PFN_xrSetPassthroughLayerMeshANDROID xrSetPassthroughLayerMeshANDROID; // previously initialized

// Inspect passthrough mesh system properties
XrSystemPassthroughLayerPropertiesANDROID passthroughLayerSystemProperties{
  XR_TYPE_SYSTEM_PASSTHROUGH_LAYER_PROPERTIES_ANDROID};
XrSystemProperties systemProperties{
  XR_TYPE_SYSTEM_PROPERTIES, &passthroughLayerSystemProperties};
CHK_XR(xrGetSystemProperties(instance, systemId, &systemProperties));
if (!passthroughLayerSystemProperties.supportsPassthroughLayer) {
    // the system does not support composite layer passthrough mesh.
    return;
}

// The initial mesh for the layer.
XrPassthroughLayerMeshANDROID mesh = {
  .type = XR_TYPE_PASSTHROUGH_LAYER_MESH_ANDROID,
  .windingOrder = XR_WINDING_ORDER_CW_ANDROID,
  .vertexCount = 4,
  .vertices = {
    { 0, 0, 0 }, { 0, 1, 0 }, { 1, 1, 0 }, { 1, 0, 0 }
  },
  .indexCount = 6,
  .indices = {
    0, 1, 2,
    0, 2, 3
  },
};

// Create the layer. Layers are expected to persist across frames.
XrPassthroughLayerCreateInfoANDROID create_info = {
  .type = XR_TYPE_PASSTHROUGH_LAYER_CREATE_INFO_ANDROID,
  .next = &mesh,
  .vertexCapacity = 0,
  .indexCapacity = 0,
};
XrPassthroughLayerANDROID layer;
CHK_XR(xrCreatePassthroughLayerANDROID(session, &create_info, &layer));

// Create a composition layer. Composition layers are submitted per frame.
XrCompositionLayerPassthroughANDROID passthrough_layer = {
  .type = XR_TYPE_COMPOSITION_LAYER_PASSTHROUGH_ANDROID,
  .next = nullptr,
  .layerFlags = 0,
  .space = space,
  .pose = {
    .orientation = { 0.0f, 0.0f, 0.0f, 1.0f }
    .position = { 0.0f, 0.0f, 0.0f }
  },
  .scale = { 1.0f, 1.0f, 1.0f },
  .opacity = 1.0f,
  .layer = layer
};

while (1) {
    // ...
    // For every frame in frame loop
    // ...

    // Submit composition layer in xrEndFrame.
    std::vector<XrCompositionLayerBaseHeader*> layers = {
        ...,
        &passthrough_layer,
        ...,
    };
    XrFrameEndInfo end_frame_info = { XR_TYPE_FRAME_END_INFO, nullptr };
    end_frame_info.layerCount = (uint32_t)layers.size();
    end_frame_info.layers = layers.data();
    CHK_XR(xrEndFrame(session, &end_frame_info));

    // Update the layer. Results can be seen the next time a passthrough composition
    // layer is submitted.
    mesh.indexCount = 9;
    const uint16_t new_index_buffer[] = {
        0, 1, 2,
        0, 2, 3,
        0, 1, 2
    };
    mesh.indexBuffer = &new_index_buffer[0];
    CHK_XR(xrSetPassthroughLayerMeshANDROID(&layer, &mesh));

    // ...
    // Finish frame loop
    // ...
}

// Clean up.
CHK_XR(xrDestroyPassthroughLayerANDROID(layer));