Work with hands using ARCore for Jetpack XR

ARCore for Jetpack XR can provide information about the user's detected hands, and gives pose information for hands and their associated joints. This hand data can be used to attach entities and models to a user's hands, for example, a tool menu:

Create an ARCore for Jetpack XR session

Access hand information through an ARCore for Jetpack XR session. See Understand a Session's lifecycle to obtain a Session.

Retrieve hand data

Hand data is available for left and right hands separately. Use each hand's state to access pose positions for each joint:

Hand.left(session).state.collect { // or Hand.right(session)
  // Hand state has been updated.
  // Use the state of hand joints to update an entity's position.
}

Hands have the following properties:

• isActive: whether or not the hand is being tracked.

• handJoints: a map of hand joints to poses. Hand joint poses are specified by the OpenXR standards.

Use hand data in your app

The positions of a user's hand joints can be used to anchor 3D objects to a user's hands, for example, to attach a model to the left palm:

Hand.left(session)?.state?.collect { leftHandState ->
  val palmPose = leftHandState.handJoints[HandJointType.PALM] ?: return@collect

  // the down direction points in the same direction as the palm
  val angle = Vector3.angleBetween(palmPose.rotation * Vector3.Down, Vector3.Up)
  palmEntity.setHidden(angle > Math.toRadians(40.0))

  val transformedPose =
    session.perceptionSpace.transformPoseTo(
      palmPose,
      session.activitySpace,
    )
  val newPosition = transformedPose.translation + transformedPose.down*0.05f
  palmEntity.setPose(Pose(newPosition, transformedPose.rotation))
}

Or to attach a model to your right hand's index finger tip:

Hand.right(session)?.state?.collect { rightHandState ->
  val tipPose = rightHandState.handJoints[HandJointType.INDEX_TIP] ?: return@collect

  // the forward direction points towards the finger tip.
  val angle = Vector3.angleBetween(tipPose.rotation * Vector3.Forward, Vector3.Up)
  indexEntity.setHidden(angle > Math.toRadians(40.0))

  val transformedPose =
    session.perceptionSpace.transformPoseTo(
      tipPose,
      session.activitySpace,
    )
  val position = transformedPose.translation + transformedPose.forward * 0.03f
  val rotation = Quaternion.fromLookTowards(transformedPose.up, Vector3.Up)
  indexEntity.setPose(Pose(position, rotation))
}