Add spatial environments to your app

In the Jetpack XR SDK, spatial environments are immersive surroundings that you can add to your app to customize the background of the virtual scene. Spatial environments are only visible when an app is in Full Space.

Overview of spatial environments

A SpatialEnvironment is used to manage an app's spatial environment preferences. It is a composite of a standalone skybox image and glTF-specified geometry. Only a single skybox image and a single glTF geometry file can be set at a time.

A skybox represents the image a user sees around them in the virtual scene, creating the illusion of a distant background environment, like a sky, mountains, or cityscape. The user cannot interact with or get closer to the skybox. The Jetpack XR SDK supports spherical skyboxes in the OpenEXR standard. In addition to providing an immersive background for your app, an EXR skybox also provides image based lighting (IBL) to 3D models loaded by your app. For more information, refer to the guide for working with 3D models.

Spatial environments can also include 3D geometry content in the glTF standard. Environment geometry loaded this way will automatically be aligned with the real-world floor. Environment geometry is a great way to add realism to your environment through foreground and midground elements that blend into the skybox with the parallax effect.

In the design guidance for spatial environments, you can learn about the different types of assets you can use to create spatial environments and how to create safe and enjoyable spatial environments.

You can set your app's spatial environment to one of these three configurations:

• A combination of a skybox image and glTF geometry.
• A passthrough surface, where the environment that is displayed is a live feed from the device's outward facing cameras. At full opacity, this passthrough surface completely occludes the skybox and geometry.
• A mixed configuration, where the passthrough surface is not at full opacity, nor is it at zero opacity. In this case, the passthrough surface becomes semi-transparent and alpha blends with the skybox and geometry behind it.

Note: When you set preferences for the SpatialEnvironment, the methods don't necessarily take effect immediately. Instead, these methods set a preference that will be applied when the device enters a state where the app environment can be changed. In the code snippets, you'll see listeners for handling these state changes.

Spatial capabilities for spatial environments

• SpatialCapabilities: Represents the spatial capabilities of the current session. Certain spatial capabilities are relevant to spatial environments.

• SPATIAL_CAPABILITY_PASSTHROUGH_CONTROL: Indicates whether or not the activity can enable or disable passthrough at the current time.

• SPATIAL_CAPABILITY_APP_ENVIRONMENT: Indicates whether or not the activity can set its own spatial environment at the current time.

Import and load spatial environment resources

glTF and EXR resources for spatial environments are loaded asynchronously in the Session class. These files must be stored in the assets folder.

Create a glTF resource

A glTF resource can be created as a GltfModel, where the glTF is loaded from a local file. A GltfModel can be used as part of a spatial app environment.

// assume that session is a Session that has been previously created
val environmentGeometryFuture = session.createGltfResourceAsync("DayGeometry.glb")

val environmentGeometry = environmentGeometryFuture.await()

Create an EXR image resource

An EXR image resource can be created as an ExrImage, where the EXR is loaded from a local file. An ExrImage can be used as part of a spatial app environment for drawing skyboxes.

// assume that session is a Session that has been previously created
val skyboxExrFuture = session.createExrImageResourceAsync("BlueSkybox.exr")

val skyboxExr = skyboxExrFuture.await()

Set the SpatialEnvironmentPreference for your app

setSpatialEnvironmentPreference is used to set the preferred spatial environment for an app. This method only sets a preference and does not cause an immediate change unless isSpatialEnvironmentPreferenceActive is already true. Once the device enters a state where the XR background can be changed and the SpatialCapabilities.SPATIAL_CAPABILITY_APP_ENVIRONMENT capability is available, the preferred spatial environment for the application will be automatically displayed.

Setting the preference to null will disable the preferred spatial environment for the app, meaning the default system environment will be displayed instead.

If the given SpatialEnvironmentPreference is not null, but all of its properties are null, then the spatial environment will consist of a black skybox and no geometry.

To get notified of changes to the SpatialEnvironment state, use addOnSpatialEnvironmentChangedListener.

Basic usage

This code snippet creates the environment geometry and skybox resources and then sets the spatial environment preference. This preference will be remembered, and it will be applied when the app has the capability to set its own environment.

// Assume that session is a Session that has been previously created

// Create a GLTFResource
val environmentGeometry = session.createGltfResourceAsync("DayGeometry.glb").await()

// Create an ExrImage for the skybox
val skyboxExr = session.createExrImageResourceAsync("BlueSkybox.exr").await()

val spatialEnvironmentPreference = SpatialEnvironmentPreference(skyboxExr, environmentGeometry)

val preferenceResult = session.spatialEnvironment.setSpatialEnvironmentPreference(spatialEnvironmentPreference)

if (preferenceResult ==  SpatialEnvironment.SetSpatialEnvironmentPreferenceChangeApplied()) {
   // The environment was successfully updated and is now visible, and any listeners
   // specified using addOnSpatialEnvironmentChangedListener will be notified.
} else if (preferenceResult == SpatialEnvironment.SetSpatialEnvironmentPreferenceChangePending()) {
    // The environment is in the process of being updated. Once visible, any listeners
   // specified using addOnSpatialEnvironmentChangedListener will be notified.
}

Advanced usage

For more advanced use cases where you need finer control over the environment, you can incorporate SpatialCapabilities checks and implement an addOnSpatialEnvironmentChangedListener to determine when you want to set the spatial environment preference.

Set the PassthroughOpacityPreference for the spatial environment for your app

One of the components of an app's immersive virtual background is a passthrough surface. In this case the background that is displayed is a live feed from the device's outward facing cameras.

setPassthroughOpacityPreference is used to set the preferred passthrough opacity for an app. This method only sets a preference and does not cause an immediate change unless the SpatialCapabilities.SPATIAL_CAPABILITY_PASSTHROUGH_CONTROL capability is available. Once the device enters a state where the passthrough opacity can be changed, and the SpatialCapabilities.SPATIAL_CAPABILITY_PASSTHROUGH_CONTROL capability is available, the preferred passthrough opacity for the application will be automatically applied.

The values for passthrough opacity preference range from 0.0f (zero opacity, where the passthrough surface is not visible) to 1.0f (full opacity, where the passthrough surface hides the spatial environment). The setPassthroughOpacityPreference parameter is a nullable float. Setting the value to null indicates that the app has no passthrough opacity preference, and will return passthrough control to the system.

Basic usage

This code snippet sets the passthrough opacity preference. This preference will be remembered, and it will be applied when the app has the capability to set passthrough opacity.

// Assume that session is a Session that has been previously created

val preferenceResult = session.spatialEnvironment.setPassthroughOpacityPreference(1.0f)

if (preferenceResult ==  SpatialEnvironment.SetPassthroughOpacityPreferenceChangeApplied()) {
  // The passthrough opacity request succeeded and should be visible now, and any listeners specified using addOnPassthroughOpacityChangedListener
  // will be notified
} else if (preferenceResult == SpatialEnvironment.SetPassthroughOpacityPreferenceChangePending()) {
  // The passthrough opacity preference was successfully set, but not
  // immediately visible. The passthrough opacity change will be applied
  // when the activity has the
  // SpatialCapabilities.SPATIAL_CAPABILITY_PASSTHROUGH_CONTROL capability.
  // Then, any listeners specified using addOnPassthroughOpacityChangedListener
  // will be notified
}

Advanced usage

For more advanced use cases where you need finer control over the passthrough opacity, you can incorporate SpatialCapabilities checks and implement an addOnPassthroughOpacityChangedListener to determine when you want to set the passthrough opacity preference.

Asset optimization

When creating assets for setting your users' SpatialEnvironment, you'll want to ensure that your assets achieve high-quality resolution while maintaining a reasonable file size. You'll want to ensure that your glb uses mipmaps and ktx2 textures. You'll also want to be sensitive to the polycount in your glb files, as a high polycount can lead to unnecessary power consumption. The majority of the file size for most SpatialEnvironment instances comes from the image used for the Skybox. To ensure your images are optimized, run the assets through an optimization tool (for example, ktx).

Determine the current passthrough opacity

val currentPassthroughOpacity =  session.spatialEnvironment.getCurrentPassthroughOpacity()

See also

• Spatial environments design guidance