After you've requested and been granted the necessary permissions, your app can access the hardware on the audio glasses or display glasses. The key to accessing the glasses' hardware (instead of the phone's hardware), is to use a projected context.
There are two primary ways to get a projected context, depending on where your code is executing:
Get a projected context if your code is running in an projected activity
If your app's code is running from within your projected activity, its own activity context is already a projected context. In this scenario, calls made within that activity can already access the glasses' hardware.
Get a projected context for code running in a phone app component
If a part of your app outside of your projected activity (such as a phone
activity or a service) needs to access the glasses' hardware, it must explicitly
obtain a projected context. To do this, use the
createProjectedDeviceContext method:
@RequiresApi(Build.VERSION_CODES.BAKLAVA) @OptIn(ExperimentalProjectedApi::class, ExperimentalCoroutinesApi::class) fun monitorProjectedDeviceConnectivity( context: Context, onConnected: suspend (projectedContext: Context) -> (() -> Unit)? ) { val lifecycleScope = (context as? LifecycleOwner)?.lifecycleScope ?: CoroutineScope(Dispatchers.Main + SupervisorJob()) lifecycleScope.launch { var cleanupAction: (() -> Unit)? = null try { // Before creating a projected context, verify that the projected device // is connected to the host device (phone). // While this flow collects true, the active projected context remains valid. ProjectedContext.isProjectedDeviceConnected(context, coroutineContext) .distinctUntilChanged() .collectLatest { isConnected -> // Clean up any previous session resources prior to state transitions. cleanupAction?.invoke() cleanupAction = null if (!isConnected) { Log.i(TAG, "Projected device disconnected. Cleaning up hardware resources...") return@collectLatest } // Inside a phone Activity or Service, create a projected context. // Note: If you are already inside a projected Activity, the default context // is already a projected context. val projectedContext = runCatching { ProjectedContext.createProjectedDeviceContext(context) }.onFailure { e -> Log.e(TAG, "Failed to create projected context", e) }.getOrNull() ?: return@collectLatest // Use the new, valid `projectedContext` to initialize system services (e.g., CameraManager). Log.i(TAG, "Projected device connected. Initializing hardware...") cleanupAction = onConnected(projectedContext) } } finally { // Guarantee final resource cleanup on Activity destruction or scope cancellation. cleanupAction?.invoke() cleanupAction = null } } }
Check for validity
Wrap the createProjectedDeviceContext call within the
ProjectedContext.isProjectedDeviceConnected. While this method returns
true, the projected context remains valid to the connected device, and your
phone app activity or service (such as a CameraManager) can access the glasses
hardware.
Clean up on disconnect
The projected context is tied to the lifecycle of the connected device, so it is
destroyed when the device disconnects. When the device disconnects,
ProjectedContext.isProjectedDeviceConnected returns false. Your app
should listen for this change and clean up any system services (such as a
CameraManager) or resources that your app created using that projected
context.
Re-initialize on reconnect
When the glasses reconnect, your app can obtain another projected context
instance using createProjectedDeviceContext, and then re-initialize any
system services or resources using the new projected context.
Record audio with the glasses' microphone
You can record audio from the glasses using Bluetooth Hands-Free Profile (HFP). In this mode, the glasses connect to the phone using standard Headset and Advanced Audio Distribution Profile (A2DP) profiles, functioning like a typical Bluetooth peripheral. If your app is already designed for standard Bluetooth recording, you can use this method to record audio from the glasses without any additional development.
Record audio using Bluetooth HFP
To record audio using Bluetooth HFP, first request the required runtime
permissions, and then record the audio using the AudioManager API, as
described in the following sections.
Request permissions
As with any standard Bluetooth audio device, the RECORD_AUDIO,
BLUETOOTH_CONNECT, and other related permissions are controlled by the
phone and not the connected device (such as audio glasses or display glasses).
Follow these steps to request the permissions:
Declare following permissions in your app's manifest file:
Request both the
RECORD_AUDIOandBLUETOOTH_CONNECTpermissions at runtime using the standard Android permission flow.
Use AudioManager to route audio
After the user has granted your app the necessary runtime permissions, use the
AudioManager API to set the communication device to
TYPE_BLUETOOTH_SCO to route the audio through Bluetooth HFP. This
directs the system to retrieve audio from the Bluetooth peripheral.
val audioManager = context.getSystemService(AudioManager::class.java) ?: return val hfpDevice = audioManager.availableCommunicationDevices .firstOrNull { it.type == AudioDeviceInfo.TYPE_BLUETOOTH_SCO } ?: return // HFP input is limited to 16 kHz, with mono or stereo channel configurations. val sampleRate = 16_000 val channelMask = AudioFormat.CHANNEL_IN_MONO val encoding = AudioFormat.ENCODING_PCM_16BIT // Use a small multiple of the minimum buffer size to keep chunks short for low latency. val minBufferSize = AudioRecord.getMinBufferSize(sampleRate, channelMask, encoding) if (minBufferSize <= 0) return val audioRecord = AudioRecord.Builder() .setAudioSource(MediaRecorder.AudioSource.VOICE_COMMUNICATION) .setAudioFormat( AudioFormat.Builder() .setSampleRate(sampleRate) .setChannelMask(channelMask) .setEncoding(encoding) .build() ) .setBufferSizeInBytes(minBufferSize * 2) .build() try { // Route voice communication audio through the Bluetooth device. if (!audioManager.setCommunicationDevice(hfpDevice)) return audioRecord.startRecording() // Read audio with audioRecord.read() on a background thread. } finally { // Stop and release when done. if (audioRecord.recordingState == AudioRecord.RECORDSTATE_RECORDING) audioRecord.stop() audioRecord.release() audioManager.clearCommunicationDevice() }
Capture an image with the glasses' camera
To capture an image with the glasses' camera, set up and bind the CameraX's
ImageCapture use case to the glasses' camera using the correct
context for your app:
/** * Filters [CameraSelector.DEFAULT_BACK_CAMERA] by dynamic [cameraId], verifies availability, * and binds an [ImageCapture] use case targeting a 16:9 resolution to [lifecycleOwner]. * * Note: Depending on your application's needs, [ImageCapture] can be swapped out for other * standard CameraX use cases: * - Use [ImageAnalysis] for real-time streaming workloads (e.g., OCR, AI vision, or VPS). * - Use [VideoCapture] for video recording pipelines. */ @OptIn(ExperimentalCamera2Interop::class) fun bindGlassesImageCapture( cameraProvider: ProcessCameraProvider, cameraId: String, lifecycleOwner: LifecycleOwner, preferredResolution: Size = Size(1920, 1080) ): ImageCapture? { // Within a projected context, the glasses' primary outward-facing camera maps to // `DEFAULT_BACK_CAMERA`. Narrow it down by the dynamic `cameraId` because the projected // camera's ID may change when recreated. val cameraSelector = CameraSelector.Builder() .addCameraFilter { cameraInfos -> // Filter the `DEFAULT_BACK_CAMERA` candidates by our target dynamic ID. CameraSelector.DEFAULT_BACK_CAMERA.filter(cameraInfos).filter { info -> Camera2CameraInfo.from(info).cameraId == cameraId } } .build() // Pre-binding check: Verify the selected camera is registered and available on the device. // The projected camera may be unavailable (for example, if the user took off the glasses). if (!cameraProvider.hasCamera(cameraSelector)) { Log.w(TAG, "Glasses camera (ID: $cameraId) is not currently available.") return null } val resolutionSelector = ResolutionSelector.Builder() .setAspectRatioStrategy(AspectRatioStrategy.RATIO_16_9_FALLBACK_AUTO_STRATEGY) .setResolutionStrategy( ResolutionStrategy( preferredResolution, ResolutionStrategy.FALLBACK_RULE_CLOSEST_LOWER ) ) .build() // Initialize the ImageCapture use case. val imageCapture = ImageCapture.Builder() .setResolutionSelector(resolutionSelector) .build() return try { // Unbind any previously bound use cases before rebinding to the lifecycle owner. cameraProvider.unbindAll() // Bind the configured use case. You can swap `imageCapture` out for an `ImageAnalysis` // (e.g., from `setupCameraStreamingFps`) or `VideoCapture` instance here to match your use case. // `bindToLifecycle()` can throw IllegalArgumentException if use cases // are incompatible, or IllegalStateException if the LifecycleOwner is destroyed. cameraProvider.bindToLifecycle( lifecycleOwner, cameraSelector, imageCapture ) Log.i(TAG, "ImageCapture successfully bound for camera ID $cameraId") imageCapture } catch (exc: IllegalArgumentException) { Log.e(TAG, "Invalid camera configuration or use case combination for ID $cameraId", exc) null } catch (exc: IllegalStateException) { Log.e(TAG, "Camera is already bound to another lifecycle or closed", exc) null } }
Key points about the code
- Obtains an instance of the
ProcessCameraProviderusing the projected device context. - Within the projected context's scope, the glasses' primary, outward-pointing
camera maps to the
DEFAULT_BACK_CAMERAwhen selecting a camera. - A pre-binding check uses
cameraProvider.hasCamera(cameraSelector)to verify that the selected camera is available on the device before proceeding. - Uses Camera2 Interop with
Camera2CameraInfoto read the underlyingCameraCharacteristics#SCALER_STREAM_CONFIGURATION_MAP, which can be useful for advanced checks on supported resolutions. - A custom
ResolutionSelectoris built to precisely control the output image resolution forImageCapture. - Creates an
ImageCaptureuse case that is configured with a customResolutionSelector. - Binds the
ImageCaptureuse case to the activity's lifecycle. This automatically manages the opening and closing of the camera based on the activity's state (for example, stopping the camera when the activity is paused). - Configuring for single-stream hardware: The glasses camera pipeline is
constrained to a single active stream at a time. When
ImageCaptureis bound alone, CameraX attaches an internal repeating stream (MeteringRepeating) for focus and metering by default, which causes the single-stream pipeline to stall. To bindImageCaptureby itself without a preview stream, disable the forced repeating stream by settingsetRepeatingStreamForced(false)onCameraXConfig.Builder(for example, by implementingCameraXConfig.Providerin yourApplicationclass or configuringProcessCameraProviderbefore initialization).
After the glasses' camera is set up, you can capture an image with the CameraX's
ImageCapture class. Refer to the CameraX's documentation to learn about using
takePicture to capture an image.
Capture a stream with the glasses' camera
To capture streams instead of an image, replace the ImageCapture component
from the capture image example with VideoCapture or
ImageAnalysis components and change the capture logic.
For more information about these APIs how to use them, see the CameraX video capture and CameraX image analysis documentation.
Choose a resolution and frame rate
You should query the device to find its supported resolutions and their corresponding maximum frame rates. The following table lists examples of values you might see, depending on your app's use case. To conserve battery power and reduce heat on the glasses, choose the lowest resolution and frame rate that work for your use case.
| Use case | Resolution | Frame rate |
|---|---|---|
| Text recognition (OCR) or world sensing | 1920 x 1080 | 1 FPS |
| AI video streaming (multimodal assistants) | 1024 x 768 | 1 FPS |
| Localization and visual positioning (VPS) | 640 x 360 or 640 x 480 | 10 FPS |
| Video calling | 640 x 360 or 640 x 480 | 24 FPS |
Query the maximum frame rate for each resolution
CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES reports the device's overall
frame rate bounds, which can be higher than the streaming rate that a given
resolution supports. To get the actual maximum for each resolution on audio and
display glasses, read its minimum frame duration and convert it to FPS with
StreamConfigurationMap.getOutputMinFrameDuration(format, size):
@androidx.annotation.OptIn(ExperimentalCamera2Interop::class) fun getMaxFpsBySize(cameraInfo: CameraInfo, format: Int): Map<Size, Int> { val map = Camera2CameraInfo.from(cameraInfo).getCameraCharacteristic( CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP ) ?: return emptyMap() return map.getOutputSizes(format).associateWith { size -> val minFrameDurationNs = map.getOutputMinFrameDuration(format, size) if (minFrameDurationNs > 0) (1_000_000_000L / minFrameDurationNs).toInt() else 0 } }
Next, use that value to calculate the minFps and maxFps bounds for your
target frame rate range. Cap the lower bound (minFps) at the resolution's
maximum streaming frame rate calculated from getOutputMinFrameDuration, and
set a higher upper bound (maxFps) as a catch-up window:
val maxFpsBySize = getMaxFpsBySize(cameraInfo, ImageFormat.YUV_420_888) val maxSupportedFps = maxFpsBySize[size] ?: return ImageAnalysis.Builder().build() // minFps is the target streaming frame rate and should be at most maxSupportedFps. val minFps = minOf(desiredFps, maxSupportedFps) // Set the upper bound to 2x minFps so the stream can catch up if frames are delayed. val maxFps = minFps * 2
Key points about the code
- Use the format that matches your use case. For
ImageAnalysis, that'sYUV_420_888. - For
VideoCapture, use theClassoverloads instead:getOutputSizes(MediaRecorder::class.java)andgetOutputMinFrameDuration(MediaRecorder::class.java, size). - Specify a range where
maxFpsis greater thanminFps(such asminFps*2) rather than setting both bounds to the same value likeRange(minFps, minFps). The lower bound (minFps) sets the target streaming frame rate and should be at most the value calculated fromgetOutputMinFrameDuration, while the upper bound (maxFps) acts as a catch-up window for delayed frames to prevent persistent latency. For example, [1,2] or [1,3] for a 1 FPS stream, and [24,30] or [24,48] for a 24 FPS stream.
Configure the frame rate
To configure the frame rate range based on your CameraX use case:
// Create and return the ImageAnalysis builder with proper frame rate ranges // ImageAnalysis: OCR, AI streaming, and VPS val analysisBuilder = ImageAnalysis.Builder() .setResolutionSelector(resolutionSelector) Camera2Interop.Extender(analysisBuilder) .setCaptureRequestOption( CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(minFps, maxFps) ) return analysisBuilder.build()
This example shows configuration for both VideoCapture and ImageAnalysis,
but you can implement whichever use cases your app supports.
Understand how the stream behaves
Keep the following system behaviors in mind while working with capture streams:
- Unsupported resolutions are rejected: If you request a size that isn't in the list, Camera2 rejects the configuration before the stream opens.
- Requesting frame rates above the maximum: If you request a minimum frame rate above the maximum supported for your resolution, the session still opens, but the target frame rate from the glasses is clamped to that supported maximum while the upper bound continues to act as the catch-up rate limit. For example, requesting [30,60] at 640 x 480 clamps the target frame rate to 24 FPS (behaving like [24,60]).
- Thermal throttling: When the glasses get warm, the frame rate drops but the resolution stays the same. Build your app to handle lower or uneven frame rates, for example by checking frame timestamps rather than assuming a fixed rate.
Access a phone's hardware from a projected activity
A projected activity can also access the phone's hardware (such as the
camera or microphone) by using createHostDeviceContext(context) to get
the host device's (phone) context:
@OptIn(ExperimentalProjectedApi::class) fun getHostDeviceContext(context: Context): Context { // Obtain the host device (phone) context. // The `context` argument passed in can be any context (projected, host, Service, or App). return ProjectedContext.createHostDeviceContext(context) }
When accessing hardware or resources that are specific to the host device (phone) in a hybrid app (an app containing both mobile and glasses experiences), you must explicitly select the correct context to make sure your app can access the correct hardware:
- Use the
Activitycontext from the phoneActivityor theProjectedContext.createHostDeviceContextto get the phone's context. - Don't use
getApplicationContextbecause the application context can incorrectly return the glasses' context if a projected activity was the most-recently-launched component.