XR_ANDROID_face_tracking OpenXR extension

Name String

XR_ANDROID_face_tracking

Extension Type

Instance extension

Registered Extension Number

459

Revision

1

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2024-09-06

IP Status

No known IP claims.

Contributors

Spencer Quin, Google

Jared Finder, Google

Levana Chen, Google

Overview

This extension enables applications to get weights of blend shapes and render facial expressions in XR experiences.

This extension is intended to provide the information needed to create realistic avatars and expressive representations of users in virtual space. The application can check facial calibration activation before getting weights of blend shapes.

Face Tracker

A face tracker is a sensing device that tracks facial expression through user-facing image streams and camera calibration. The main purpose of this extension is to map user facial expressions to their avatars in a virtual scene.

Face tracking data is sensitive personal information and is closely linked to personal privacy and integrity. It is strongly recommended that applications storing or transferring face tracking data always ask the user for active and specific acceptance to do so.

Caution:

Permissions Android applications must have the android.permission.FACE_TRACKING permission listed in their manifest. The android.permission.FACE_TRACKING permission is considered a dangerous permission. The application must request the permission at runtime to use these functions:

• xrCreateFaceTrackerANDROID
• xrDestroyFaceTrackerANDROID
• xrGetFaceStateANDROID
• xrGetFaceCalibrationStateANDROID

(protection level: dangerous)

• Applications will receive XR_ERROR_PERMISSION_INSUFFICIENT when creating an active face tracker until the application has been allowed access to the face tracker.
• When getting face states by using xrGetFaceStateANDROID, XrFaceStateANDROID::isValid won't return XR_TRUE unless the application access has been allowed

Create a face tracker handle

XR_DEFINE_HANDLE(XrFaceTrackerANDROID)

The XrFaceTrackerANDROID handle represents a face tracker for face tracking.

This handle can be used to access face tracking data using other functions in this extension.

The xrCreateFaceTrackerANDROID function is defined as:

XrResult xrCreateFaceTrackerANDROID(
    XrSession                                   session,
    const XrFaceTrackerCreateInfoANDROID*       createInfo,
    XrFaceTrackerANDROID*                       faceTracker);

Parameter Descriptions

• session is an XrSession handle in which the face tracker will be active.
• createInfo is the XrFaceTrackerCreateInfoANDROID used to specify the face tracker.
• faceTracker is the returned XrFaceTrackerANDROID handle.

An application can create an XrFaceTrackerANDROID handle using xrCreateFaceTrackerANDROID function.

If the system does not support face tracking, xrCreateFaceTrackerANDROID will return XR_ERROR_FEATURE_UNSUPPORTED.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrCreateFaceTrackerANDROID
• session must be a valid XrSession handle
• createInfo must be a pointer to a valid XrFaceTrackerCreateInfoANDROID structure
• faceTracker must be a pointer to an XrFaceTrackerANDROID handle

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED

The XrFaceTrackerCreateInfoANDROID structure is described as follows:

typedef struct XrFaceTrackerCreateInfoANDROID {
    XrStructureType    type;
    void*              next;
} XrFaceTrackerCreateInfoANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

The XrFaceTrackerCreateInfoANDROID structure describes the information to create an XrFaceTrackerANDROID handle.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to using XrFaceTrackerCreateInfoANDROID
• type must be XR_TYPE_FACE_TRACKER_CREATE_INFO_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain

The xrDestroyFaceTrackerANDROID function is defined as:

XrResult xrDestroyFaceTrackerANDROID(
    XrFaceTrackerANDROID                        faceTracker);

Parameter Descriptions

• faceTracker is an XrFaceTrackerANDROID previously created by xrCreateFaceTrackerANDROID.

The xrDestroyFaceTrackerANDROID function releases the faceTracker and the underlying resources when the face tracking experience is over.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrDestroyFaceTrackerANDROID
• faceTracker must be a valid XrFaceTrackerANDROID handle

Thread Safety

• Access to faceTracker, and any child handles, must be externally synchronized

Return Codes

Success

• XR_SUCCESS

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_HANDLE_INVALID

Check face calibration

The xrGetFaceCalibrationStateANDROID function is defined as:

XrResult xrGetFaceCalibrationStateANDROID(
    XrFaceTrackerANDROID                        faceTracker,
    XrBool32*                                   faceIsCalibratedOutput);

Parameter Descriptions

• faceTracker is an XrFaceTrackerANDROID previously created by xrCreateFaceTrackerANDROID.
• faceIsCalibratedOutput indicates if the face tracker has been calibrated or not.

An application can check face calibration state using xrGetFaceCalibrationStateANDROID function.

If the system does not support face calibration, xrGetFaceCalibrationStateANDROID will return XR_ERROR_FEATURE_UNSUPPORTED. Otherwise, faceIsCalibratedOutput may be set to XR_TRUE to reflect the face calibration state.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrGetFaceCalibrationStateANDROID
• faceTracker must be a valid XrFaceTrackerANDROID handle
• faceIsCalibratedOutput must be a pointer to an XrBool32 value

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED

Get facial expressions

The xrGetFaceStateANDROID function returns blend shapes of facial expressions at a given time.

XrResult xrGetFaceStateANDROID(
    XrFaceTrackerANDROID                        faceTracker,
    const XrFaceStateGetInfoANDROID*            getInfo,
    XrFaceStateANDROID*                         faceStateOutput);

Parameter Descriptions

• faceTracker is an XrFaceTrackerANDROID previously created by xrCreateFaceTrackerANDROID.
• getInfo is a pointer to XrFaceStateGetInfoANDROID describing information to obtain face expressions.
• faceStateOutput is a pointer to XrFaceStateANDROID receiving the returned face tracking state and facial expressions.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrGetFaceStateANDROID
• faceTracker must be a valid XrFaceTrackerANDROID handle
• getInfo must be a pointer to a valid XrFaceStateGetInfoANDROID structure
• faceStateOutput must be a pointer to an XrFaceStateANDROID structure

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED
• XR_ERROR_TIME_INVALID

The XrFaceStateGetInfoANDROID structure describes the information to obtain facial expressions.

typedef struct XrFaceStateGetInfoANDROID {
    XrStructureType    type;
    void*              next;
    XrTime             time;
} XrFaceStateGetInfoANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
• time is an XrTime at which the facial expressions are requested.

Applications should request a time equal to the predicted display time for the rendered frame.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to using XrFaceStateGetInfoANDROID
• type must be XR_TYPE_FACE_STATE_GET_INFO_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain

XrFaceStateANDROID structure returns the face tracking state and facial expressions.

typedef struct XrFaceStateANDROID {
    XrStructureType               type;
    void*                         next;
    uint32_t                      parametersCapacityInput;
    uint32_t                      parametersCountOutput;
    float*                        parameters;
    XrFaceTrackingStateANDROID    faceTrackingState;
    XrTime                        sampleTime;
    XrBool32                      isValid;
} XrFaceStateANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
• parametersCapacityInput is a uint32_t describing the capacity of the parameters array, or 0 to indicate a request to retrieve the required capacity.
• parametersCountOutput is a uint32_t describing the number of parameters, or the required capacity in the case that parametersCapacityInput is insufficient.
• parameters is a pointer to an application-allocated array of float that will be filled with weights of facial expression blend shapes.
• faceTrackingState is the XrFaceTrackingStateANDROID of validity status of face tracking.
• sampleTime is an XrTime time at which the returned expressions are tracked or extrapolated to. It equals the time at which the expression weights were requested if the extrapolation at the time was successful.
• isValid indicates if the data is valid even if it is not from the current frame.
• See the Buffer Size Parameters section for a detailed description of retrieving the required parameters size.

The application can set parametersCapacityInput to XR_FACE_PARAMETER_COUNT_ANDROID to get facial expressions which are indexed by XrFaceParameterIndicesANDROID.

The returned parameters represent the weights of blend shapes of current facial expressions.

The parameters array updates will be ordered so that the application can index elements using the corresponding facial expression enum (e.g. XrFaceParameterIndicesANDROID).

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to using XrFaceStateANDROID
• type must be XR_TYPE_FACE_STATE_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain
• If parametersCapacityInput is not 0, parameters must be a pointer to an array of parametersCapacityInput float values
• faceTrackingState must be a valid XrFaceTrackingStateANDROID value

The XrFaceTrackingStateANDROID enumeration identifies the different states of the face tracker.

typedef enum XrFaceTrackingStateANDROID {
    XR_FACE_TRACKING_STATE_PAUSED_ANDROID = 0,
    XR_FACE_TRACKING_STATE_STOPPED_ANDROID = 1,
    XR_FACE_TRACKING_STATE_TRACKING_ANDROID = 2
} XrFaceTrackingStateANDROID;

The enums have the following meanings:

Enum	Description
XR_FACE_TRACKING_STATE_PAUSED_ANDROID	Indicates that face tracking is paused but may be resumed in the future.
XR_FACE_TRACKING_STATE_STOPPED_ANDROID	Tracking has stopped but the client still has an active face tracker.
XR_FACE_TRACKING_STATE_TRACKING_ANDROID	The face is tracked and its pose is current.

Conventions of blend shapes

This extension defines 63 blend shapes through XR_FACE_PARAMETER_COUNT_ANDROID, for the reduced G-Nome format. Each parameter in this enum is an index into a blend shape array whose values are type of float and the runtime normalizes to 1 - 0.

typedef enum XrFaceParameterIndicesANDROID {
    XR_FACE_PARAMETER_INDICES_BROW_LOWERER_L_ANDROID = 0,
    XR_FACE_PARAMETER_INDICES_BROW_LOWERER_R_ANDROID = 1,
    XR_FACE_PARAMETER_INDICES_CHEEK_PUFF_L_ANDROID = 2,
    XR_FACE_PARAMETER_INDICES_CHEEK_PUFF_R_ANDROID = 3,
    XR_FACE_PARAMETER_INDICES_CHEEK_RAISER_L_ANDROID = 4,
    XR_FACE_PARAMETER_INDICES_CHEEK_RAISER_R_ANDROID = 5,
    XR_FACE_PARAMETER_INDICES_CHEEK_SUCK_L_ANDROID = 6,
    XR_FACE_PARAMETER_INDICES_CHEEK_SUCK_R_ANDROID = 7,
    XR_FACE_PARAMETER_INDICES_CHIN_RAISER_B_ANDROID = 8,
    XR_FACE_PARAMETER_INDICES_CHIN_RAISER_T_ANDROID = 9,
    XR_FACE_PARAMETER_INDICES_DIMPLER_L_ANDROID = 10,
    XR_FACE_PARAMETER_INDICES_DIMPLER_R_ANDROID = 11,
    XR_FACE_PARAMETER_INDICES_EYES_CLOSED_L_ANDROID = 12,
    XR_FACE_PARAMETER_INDICES_EYES_CLOSED_R_ANDROID = 13,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_DOWN_L_ANDROID = 14,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_DOWN_R_ANDROID = 15,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_LEFT_L_ANDROID = 16,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_LEFT_R_ANDROID = 17,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_RIGHT_L_ANDROID = 18,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_RIGHT_R_ANDROID = 19,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_UP_L_ANDROID = 20,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_UP_R_ANDROID = 21,
    XR_FACE_PARAMETER_INDICES_INNER_BROW_RAISER_L_ANDROID = 22,
    XR_FACE_PARAMETER_INDICES_INNER_BROW_RAISER_R_ANDROID = 23,
    XR_FACE_PARAMETER_INDICES_JAW_DROP_ANDROID = 24,
    XR_FACE_PARAMETER_INDICES_JAW_SIDEWAYS_LEFT_ANDROID = 25,
    XR_FACE_PARAMETER_INDICES_JAW_SIDEWAYS_RIGHT_ANDROID = 26,
    XR_FACE_PARAMETER_INDICES_JAW_THRUST_ANDROID = 27,
    XR_FACE_PARAMETER_INDICES_LID_TIGHTENER_L_ANDROID = 28,
    XR_FACE_PARAMETER_INDICES_LID_TIGHTENER_R_ANDROID = 29,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_DEPRESSOR_L_ANDROID = 30,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_DEPRESSOR_R_ANDROID = 31,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_PULLER_L_ANDROID = 32,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_PULLER_R_ANDROID = 33,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_LB_ANDROID = 34,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_LT_ANDROID = 35,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_RB_ANDROID = 36,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_RT_ANDROID = 37,
    XR_FACE_PARAMETER_INDICES_LIP_PRESSOR_L_ANDROID = 38,
    XR_FACE_PARAMETER_INDICES_LIP_PRESSOR_R_ANDROID = 39,
    XR_FACE_PARAMETER_INDICES_LIP_PUCKER_L_ANDROID = 40,
    XR_FACE_PARAMETER_INDICES_LIP_PUCKER_R_ANDROID = 41,
    XR_FACE_PARAMETER_INDICES_LIP_STRETCHER_L_ANDROID = 42,
    XR_FACE_PARAMETER_INDICES_LIP_STRETCHER_R_ANDROID = 43,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_LB_ANDROID = 44,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_LT_ANDROID = 45,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_RB_ANDROID = 46,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_RT_ANDROID = 47,
    XR_FACE_PARAMETER_INDICES_LIP_TIGHTENER_L_ANDROID = 48,
    XR_FACE_PARAMETER_INDICES_LIP_TIGHTENER_R_ANDROID = 49,
    XR_FACE_PARAMETER_INDICES_LIPS_TOWARD_ANDROID = 50,
    XR_FACE_PARAMETER_INDICES_LOWER_LIP_DEPRESSOR_L_ANDROID = 51,
    XR_FACE_PARAMETER_INDICES_LOWER_LIP_DEPRESSOR_R_ANDROID = 52,
    XR_FACE_PARAMETER_INDICES_MOUTH_LEFT_ANDROID = 53,
    XR_FACE_PARAMETER_INDICES_MOUTH_RIGHT_ANDROID = 54,
    XR_FACE_PARAMETER_INDICES_NOSE_WRINKLER_L_ANDROID = 55,
    XR_FACE_PARAMETER_INDICES_NOSE_WRINKLER_R_ANDROID = 56,
    XR_FACE_PARAMETER_INDICES_OUTER_BROW_RAISER_L_ANDROID = 57,
    XR_FACE_PARAMETER_INDICES_OUTER_BROW_RAISER_R_ANDROID = 58,
    XR_FACE_PARAMETER_INDICES_UPPER_LID_RAISER_L_ANDROID = 59,
    XR_FACE_PARAMETER_INDICES_UPPER_LID_RAISER_R_ANDROID = 60,
    XR_FACE_PARAMETER_INDICES_UPPER_LIP_RAISER_L_ANDROID = 61,
    XR_FACE_PARAMETER_INDICES_UPPER_LIP_RAISER_R_ANDROID = 62
} XrFaceParameterIndicesANDROID;

Note: In this table, a shortened form of the XrFaceParameterIndicesANDROID names for ease of reading. BROW_LOWERER_R, for example, maps to XR_FACE_PARAMETER_INDICES_BROW_LOWERER_L_ANDROID.

Index	Name	Reference Image
0	BROW_LOWERER_L	BROW_LOWERER_L = 0.0	BROW_LOWERER_L = 1.0
1	BROW_LOWERER_R	BROW_LOWERER_R = 0.0	BROW_LOWERER_R = 1.0
2	CHEEK_PUFF_L	CHEEK_PUFF_L = 0.0	CHEEK_PUFF_L = 1.0
3	CHEEK_PUFF_R	CHEEK_PUFF_R = 0.0	CHEEK_PUFF_R = 1.0
4	CHEEK_RAISER_L	CHEEK_RAISER_L = 0.0	CHEEK_RAISER_L = 1.0
5	CHEEK_RAISER_R	CHEEK_RAISER_R = 0.0	CHEEK_RAISER_R = 1.0
6	CHEEK_SUCK_L	CHEEK_SUCK_L = 0.0	CHEEK_SUCK_L = 1.0
7	CHEEK_SUCK_R	CHEEK_SUCK_R = 0.0	CHEEK_SUCK_R = 1.0
8	CHIN_RAISER_B	CHIN_RAISER_B = 0.0	CHIN_RAISER_B = 1.0
9	CHIN_RAISER_T	CHIN_RAISER_T = 0.0	CHIN_RAISER_T = 1.0
10	DIMPLER_L	DIMPLER_L = 0.0	DIMPLER_L = 1.0
11	DIMPLER_R	DIMPLER_R = 0.0	DIMPLER_R = 1.0
12	EYES_CLOSED_L	EYES_CLOSED_L = 0.0	EYES_CLOSED_L = 1.0
13	EYES_CLOSED_R	EYES_CLOSED_R = 0.0	EYES_CLOSED_R = 1.0
14	EYES_LOOK_DOWN_L	EYES_LOOK_DOWN_L = 0.0	EYES_LOOK_DOWN_L = 1.0
15	EYES_LOOK_DOWN_R	EYES_LOOK_DOWN_R = 0.0	EYES_LOOK_DOWN_R = 1.0
16	EYES_LOOK_LEFT_L	EYES_LOOK_LEFT_L = 0.0	EYES_LOOK_LEFT_L = 1.0
17	EYES_LOOK_LEFT_R	EYES_LOOK_LEFT_R = 0.0	EYES_LOOK_LEFT_R = 1.0
18	EYES_LOOK_RIGHT_L	EYES_LOOK_RIGHT_L = 0.0	EYES_LOOK_RIGHT_L = 1.0
19	EYES_LOOK_RIGHT_R	EYES_LOOK_RIGHT_R = 0.0	EYES_LOOK_RIGHT_R = 1.0
20	EYES_LOOK_UP_L	EYES_LOOK_UP_L = 0.0	EYES_LOOK_UP_L = 1.0
21	EYES_LOOK_UP_R	EYES_LOOK_UP_R = 0.0	EYES_LOOK_UP_R = 1.0
22	INNER_BROW_RAISER_L	INNER_BROW_RAISER_L = 0.0	INNER_BROW_RAISER_L = 1.0
23	INNER_BROW_RAISER_R	INNER_BROW_RAISER_R = 0.0	INNER_BROW_RAISER_R = 1.0
24	JAW_DROP	JAW_DROP = 0.0	JAW_DROP = 1.0
25	JAW_SIDEWAYS_LEFT	JAW_SIDEWAYS_LEFT = 0.0	JAW_SIDEWAYS_LEFT = 1.0
26	JAW_SIDEWAYS_RIGHT	JAW_SIDEWAYS_RIGHT = 0.0	JAW_SIDEWAYS_RIGHT = 1.0
27	JAW_THRUST	JAW_THRUST = 0.0	JAW_THRUST = 1.0
28	LID_TIGHTENER_L	LID_TIGHTENER_L = 0.0	LID_TIGHTENER_L = 1.0
29	LID_TIGHTENER_R	LID_TIGHTENER_R = 0.0	LID_TIGHTENER_R = 1.0
30	LIP_CORNER_DEPRESSOR_L	LIP_CORNER_DEPRESSOR_L = 0.0	LIP_CORNER_DEPRESSOR_L = 1.0
31	LIP_CORNER_DEPRESSOR_R	LIP_CORNER_DEPRESSOR_R = 0.0	LIP_CORNER_DEPRESSOR_R = 1.0
32	LIP_CORNER_PULLER_L	LIP_CORNER_PULLER_L = 0.0	LIP_CORNER_PULLER_L = 1.0
33	LIP_CORNER_PULLER_R	LIP_CORNER_PULLER_R = 0.0	LIP_CORNER_PULLER_R = 1.0
34	LIP_FUNNELER_LB	LIP_FUNNELER_LB = 0.0	LIP_FUNNELER_LB = 1.0
35	LIP_FUNNELER_LT	LIP_FUNNELER_LT = 0.0	LIP_FUNNELER_LT = 1.0
36	LIP_FUNNELER_RB	LIP_FUNNELER_RB = 0.0	LIP_FUNNELER_RB = 1.0
37	LIP_FUNNELER_RT	LIP_FUNNELER_RT = 0.0	LIP_FUNNELER_RT = 1.0
38	LIP_PRESSOR_L	LIP_PRESSOR_L = 0.0	LIP_PRESSOR_L = 1.0
39	LIP_PRESSOR_R	LIP_PRESSOR_R = 0.0	LIP_PRESSOR_R = 1.0
40	LIP_PUCKER_L	LIP_PUCKER_L = 0.0	LIP_PUCKER_L = 1.0
41	LIP_PUCKER_R	LIP_PUCKER_R = 0.0	LIP_PUCKER_R = 1.0
42	LIP_STRETCHER_L	LIP_STRETCHER_L = 0.0	LIP_STRETCHER_L = 1.0
43	LIP_STRETCHER_R	LIP_STRETCHER_R = 0.0	LIP_STRETCHER_R = 1.0
44	LIP_SUCK_LB	LIP_SUCK_LB = 0.0	LIP_SUCK_LB = 1.0
45	LIP_SUCK_LT	LIP_SUCK_LT = 0.0	LIP_SUCK_LT = 1.0
46	LIP_SUCK_RB	LIP_SUCK_RB = 0.0	LIP_SUCK_RB = 1.0
47	LIP_SUCK_RT	LIP_SUCK_RT = 0.0	LIP_SUCK_RT = 1.0
48	LIP_TIGHTENER_L	LIP_TIGHTENER_L = 0.0	LIP_TIGHTENER_L = 1.0
49	LIP_TIGHTENER_R	LIP_TIGHTENER_R = 0.0	LIP_TIGHTENER_R = 1.0
50	LIPS_TOWARD	LIPS_TOWARD = 0.0	JAW_DROP = 1.0 and LIPS_TOWARD = 1.0
51	LOWER_LIP_DEPRESSOR_L	LOWER_LIP_DEPRESSOR_L = 0.0	LOWER_LIP_DEPRESSOR_L = 1.0
52	LOWER_LIP_DEPRESSOR_R	LOWER_LIP_DEPRESSOR_R = 0.0	LOWER_LIP_DEPRESSOR_R = 1.0
53	MOUTH_LEFT	MOUTH_LEFT = 0.0	MOUTH_LEFT = 1.0
54	MOUTH_RIGHT	MOUTH_RIGHT = 0.0	MOUTH_RIGHT = 1.0
55	NOSE_WRINKLER_L	NOSE_WRINKLER_L = 0.0	NOSE_WRINKLER_L = 1.0
56	NOSE_WRINKLER_R	NOSE_WRINKLER_R = 0.0	NOSE_WRINKLER_R = 1.0
57	OUTER_BROW_RAISER_L	OUTER_BROW_RAISER_L = 0.0	OUTER_BROW_RAISER_L = 1.0
58	OUTER_BROW_RAISER_R	OUTER_BROW_RAISER_R = 0.0	OUTER_BROW_RAISER_R = 1.0
59	UPPER_LID_RAISER_L	UPPER_LID_RAISER_L = 0.0	UPPER_LID_RAISER_L = 1.0
60	UPPER_LID_RAISER_R	UPPER_LID_RAISER_R = 0.0	UPPER_LID_RAISER_R = 1.0
61	UPPER_LIP_RAISER_L	UPPER_LIP_RAISER_L = 0.0	UPPER_LIP_RAISER_L = 1.0
62	UPPER_LIP_RAISER_R	UPPER_LIP_RAISER_R = 0.0	UPPER_LIP_RAISER_R = 1.0

Example code for face tracking

The following example code demonstrates how to get all weights for facial expression blend shapes.

XrSession session; // previously initialized, for example created at app startup.

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateFaceTrackerANDROID xrCreateFaceTrackerANDROID; // previously initialized
PFN_xrDestroyFaceTrackerANDROID xrDestroyFaceTrackerANDROID; // previously initialized
PFN_xrGetFaceStateANDROID xrGetFaceStateANDROID; // previously initialized
PFN_xrGetFaceCalibrationStateANDROID xrGetFaceCalibrationStateANDROID; // previously initialized

XrFaceTrackerANDROID faceTracker;
XrFaceTrackerCreateInfoANDROID
        createInfo{.type = XR_TYPE_FACE_TRACKER_CREATE_INFO_ANDROID,
                    .next = nullptr};
CHK_XR(xrCreateFaceTrackerANDROID(session, &createInfo, &faceTracker));

// If the system supports face calibration:
bool isCalibrated;
CHK_XR(xrGetFaceCalibrationStateANDROID(faceTracker, &isCalibrated));
if (!isCalibrated) {
    // Redirect the user to system calibration setting.
}

XrFaceStateANDROID faceState;
float faceExpressionParameters[XR_FACE_PARAMETER_COUNT_ANDROID];
faceState.type = XR_TYPE_FACE_STATE_ANDROID;
faceState.next = nullptr;
faceState.parametersCapacityInput = XR_FACE_PARAMETER_COUNT_ANDROID;
faceState.parameters = faceExpressionParameters;

while (1) {
    // ...
    // For every frame in the frame loop
    // ...
    XrFrameState frameState; // previously returned from xrWaitFrame

    XrFaceStateGetInfoANDROID faceGetInfo{
            .type = XR_TYPE_FACE_STATE_GET_INFO_ANDROID,
            .next = nullptr,
            .time = frameState.predictedDisplayTime,
    };

    CHECK_XR(xrGetFaceStateANDROID(faceTracker, &faceGetInfo, &faceState));
    if (faceState.isValid) {
        for (uint32_t i = 0; i < XR_FACE_PARAMETER_COUNT_ANDROID; ++i) {
            // parameters[i] contains a weight of specific blend shape
        }
    }
}

// after usage
CHK_XR(xrDestroyFaceTrackerANDROID(faceTracker));

New Object Types

• XrFaceTrackerANDROID

New Enum Constants

• XR_FACE_PARAMETER_COUNT_ANDROID

XrObjectType enumeration is extended with:

• XR_OBJECT_TYPE_FACE_TRACKER_ANDROID

XrStructureType enumeration is extended with:

• XR_TYPE_FACE_TRACKER_CREATE_INFO_ANDROID
• XR_TYPE_FACE_STATE_GET_INFO_ANDROID
• XR_TYPE_FACE_STATE_ANDROID

New Enums

• XrFaceParameterIndicesANDROID
• XrFaceTrackingStateANDROID

New Structures

• XrFaceTrackerCreateInfoANDROID
• XrFaceStateGetInfoANDROID
• XrFaceStateANDROID

New Functions

• xrCreateFaceTrackerANDROID
• xrDestroyFaceTrackerANDROID
• xrGetFaceStateANDROID
• xrGetFaceCalibrationStateANDROID

Issues

Version History

• Revision 1, 2024-09-05 (Levana Chen)
  • Initial extension description