Handle device orientation with Vulkan pre-rotation

This article describes how to efficiently handle device rotation in your Vulkan application by implementing pre-rotation.

Vulkan allows you to specify much more information about rendering state than OpenGL. With that power comes some new responsibilities; you're expected to implement things explicitly that were handled by the driver in OpenGL. One of these is device orientation and its relationship to render surface orientation. Currently, there are 3 ways that Android can handle reconciling the render surface of the device with the device orientation:

  1. The device has a Display Processing Unit (DPU) that can efficiently handle surface rotation in hardware. (on supported devices only)
  2. The Android OS can handle surface rotation by adding a compositor pass that will have a performance cost depending on how the compositor has to deal with rotating the output image.
  3. The application itself can handle the surface rotation by rendering a rotated image onto a render surface that matches the current orientation of the display.

What Does This Mean For Your Apps?

There isn't currently a way for an application to know whether surface rotation handled outside of the application will be free. Even if there is a DPU to take care of this for you, there will still likely be a measurable performance penalty to pay. If your application is CPU bound, this becomes a power issue due to the increased GPU usage by the Android Compositor (which is usually running at a boosted frequency). If your application is GPU bound, then the Android Compositor can also preempt your application's GPU work, causing additional performance loss.

When running shipping titles on the Pixel 4XL, we have seen that SurfaceFlinger (the higher-priority task that drives the Android Compositor) both regularly preempts the application’s work causing 1-3ms hits to frametimes, as well as puts increased pressure on the GPU’s vertex/texture memory as the Compositor has to read the entire framebuffer to do its composition work.

Handling orientation properly stops GPU preemption by SurfaceFlinger almost entirely, while the GPU frequency drops 40% as the boosted frequency used by the Android Compositor is no longer needed.

To ensure surface rotations are handled properly with as little overhead as possible, (as seen in the case above) we recommend implementing method 3, which is known as pre-rotation. This tells the Android OS that your app handles the surface rotation. You can do so by passing surface transform flags that specify the orientation during swapchain creation. This stops the Android Compositor from doing the rotation itself.

Knowing how to set the surface transform flag is important for every Vulkan application, since applications tend to either support multiple orientations or support a single orientation where the render surface is in a different orientation to what the device considers its identity orientation. For example, a landscape-only application on a portrait-identity phone, or a portrait-only application on a landscape-identity tablet.

In this article we will describe in detail how to implement pre-rotation and handle device rotation in your Vulkan application.

Modify AndroidManifest.xml

To handle device rotation in your app, begin by changing the application’s AndroidManifest.xml file to tell Android that your app will handle orientation and screen size changes. This prevents Android from destroying and recreating the Android Activity and calling the onDestroy() function on the existing window surface when an orientation change occurs. This is done by adding the orientation (to support API level <13) and screenSize attributes to the activity’s configChanges section:

<activity android:name="android.app.NativeActivity"
          android:configChanges="orientation|screenSize">

If your application fixes its screen orientation using the screenOrientation attribute you do not need to do this. Also if your application uses a fixed orientation then it will only need to setup the swapchain once on application startup/resume.

Get the Identity Screen Resolution and Camera Parameters

The next thing that needs to be done is to detect the device’s screen resolution associated with the VK_SURFACE_TRANSFORM_IDENTITY_BIT_KHR value. This resolution is associated with the identity orientation of the device, and is therefore the one that the swapchain will always need to be set to. The most reliable way to get this is to make a call to vkGetPhysicalDeviceSurfaceCapabilitiesKHR() at application startup and storing the returned extent. You'll want to swap the width and height based on the currentTransform that is also returned in order to ensure that you are storing the identity screen resolution:

VkSurfaceCapabilitiesKHR capabilities;
vkGetPhysicalDeviceSurfaceCapabilitiesKHR(physDevice, surface, &capabilities);

uint32_t width = capabilities.currentExtent.width;
uint32_t height = capabilities.currentExtent.height;
if (capabilities.currentTransform & VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR ||
    capabilities.currentTransform & VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR) {
  // Swap to get identity width and height
  capabilities.currentExtent.height = width;
  capabilities.currentExtent.width = height;
}

displaySizeIdentity = capabilities.currentExtent;

displaySizeIdentity is a VkExtent2D structure that we use to store said identity resolution of the app's window surface in the display’s natural orientation.

Detect Device Orientation Changes (Android 10+)

The most reliable way to detect an orientation change in your application is to verify whether the vkQueuePresentKHR() function returns VK_SUBOPTIMAL_KHR. For example:

auto res = vkQueuePresentKHR(queue_, &present_info);
if (res == VK_SUBOPTIMAL_KHR){
  orientationChanged = true;
}

Detecting Device Orientation Changes (Pre-Android 10)

For devices running older versions of Android below 10, a different implementation is needed since VK_SUBOPTIMAL_KHR is not supported.

Using Polling

On pre-Android 10 devices you can poll the current device transform every pollingInterval frames, where pollingInterval is a granularity decided on by the programmer. The way you do this is by calling vkGetPhysicalDeviceSurfaceCapabilitiesKHR() and then comparing the returned currentTransform field with that of the currently stored surface transformation (in this code example stored in pretransformFlag).

currFrameCount++;
if (currFrameCount >= pollInterval){
  VkSurfaceCapabilitiesKHR capabilities;
  vkGetPhysicalDeviceSurfaceCapabilitiesKHR(physDevice, surface, &capabilities);

  if (pretransformFlag != capabilities.currentTransform) {
    window_resized = true;
  }
  currFrameCount = 0;
}

On a Pixel 4 running Android 10, polling vkGetPhysicalDeviceSurfaceCapabilitiesKHR() took between .120-.250ms and on a Pixel 1XL running Android 8, polling took .110-.350ms.

Using Callbacks

A second option for devices running below Android 10 is to register an onNativeWindowResized() callback to call a function that sets the orientationChanged flag, signaling to the application an orientation change has occurred:

void android_main(struct android_app *app) {
  ...
  app->activity->callbacks->onNativeWindowResized = ResizeCallback;
}

Where ResizeCallback is defined as:

void ResizeCallback(ANativeActivity *activity, ANativeWindow *window){
  orientationChanged = true;
}

The drawback to this solution is that onNativeWindowResized() only ever gets called on 90 degree orientation changes (going from landscape to portrait or vice-versa), so for example an orientation change from landscape to reverse- landscape will not trigger the swapchain recreation, requiring the Android compositor to do the flip for your application.

Handling the Orientation Change

To handle the orientation change, call the orientation change routine at the top of the main rendering loop when the orientationChanged variable is set to true. For example:

bool VulkanDrawFrame() {
 if (orientationChanged) {
   OnOrientationChange();
}

And within the OnOrientationChange() function you will do all the work necessary to recreate the swapchain. This involves destroying any existing instances of Framebuffer and ImageView; recreating the swapchain while destroying the old swapchain (which will be discussed next); and then recreating the Framebuffers with the new swapchain’s DisplayImages. Note that attachment images (depth/stencil images for example) usually do not need to be recreated as they are based on the identity resolution of the pre-rotated swapchain images.

void OnOrientationChange() {
 vkDeviceWaitIdle(getDevice());

 for (int i = 0; i < getSwapchainLength(); ++i) {
   vkDestroyImageView(getDevice(), displayViews_[i], nullptr);
   vkDestroyFramebuffer(getDevice(), framebuffers_[i], nullptr);
 }

 createSwapChain(getSwapchain());
 createFrameBuffers(render_pass, depthBuffer.image_view);
 orientationChanged = false;
}

And at the end of the function you reset the orientationChanged flag to false to show that you have handled the orientation change.

Swapchain Recreation

In the previous section we mention having to recreate the swapchain. The first steps to doing so involves getting the new characteristics of the rendering surface:

void createSwapChain(VkSwapchainKHR oldSwapchain) {
   VkSurfaceCapabilitiesKHR capabilities;
   vkGetPhysicalDeviceSurfaceCapabilitiesKHR(physDevice, surface, &capabilities);
   pretransformFlag = capabilities.currentTransform;

With the VkSurfaceCapabilities struct populated with the new information, you can now check to see whether an orientation change has occurred by checking the currentTransform field. You'll store it for later in the pretransformFlag field as you will be needing it for later when you make adjustments to the MVP matrix.

To do so, specify the following attributes in the VkSwapchainCreateInfo struct:

VkSwapchainCreateInfoKHR swapchainCreateInfo{
  ...
  .sType = VK_STRUCTURE_TYPE_SWAPCHAIN_CREATE_INFO_KHR,
  .imageExtent = displaySizeIdentity,
  .preTransform = pretransformFlag,
  .oldSwapchain = oldSwapchain,
};

vkCreateSwapchainKHR(device_, &swapchainCreateInfo, nullptr, &swapchain_));

if (oldSwapchain != VK_NULL_HANDLE) {
  vkDestroySwapchainKHR(device_, oldSwapchain, nullptr);
}

The imageExtent field will be populated with the displaySizeIdentity extent that you stored at application startup. The preTransform field will be populated with the pretransformFlag variable (which is set to the currentTransform field of the surfaceCapabilities). You also set the oldSwapchain field to the swapchain that will be destroyed.

MVP Matrix Adjustment

The last thing that needs to be done is to apply the pre-transformation by applying a rotation matrix to your MVP matrix. What this essentially does is apply the rotation in clip space so that the resulting image is rotated to the current device orientation. You can then simply pass this updated MVP matrix into your vertex shader and use it as normal without the need to modify your shaders.

glm::mat4 pre_rotate_mat = glm::mat4(1.0f);
glm::vec3 rotation_axis = glm::vec3(0.0f, 0.0f, 1.0f);

if (pretransformFlag & VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR) {
  pre_rotate_mat = glm::rotate(pre_rotate_mat, glm::radians(90.0f), rotation_axis);
}

else if (pretransformFlag & VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR) {
  pre_rotate_mat = glm::rotate(pre_rotate_mat, glm::radians(270.0f), rotation_axis);
}

else if (pretransformFlag & VK_SURFACE_TRANSFORM_ROTATE_180_BIT_KHR) {
  pre_rotate_mat = glm::rotate(pre_rotate_mat, glm::radians(180.0f), rotation_axis);
}

MVP = pre_rotate_mat * MVP;

Consideration - Non-Full Screen Viewport and Scissor

If your application is using a non-full screen viewport/scissor region, they will need to be updated according to the orientation of the device. This requires that you enable the dynamic Viewport and Scissor options during Vulkan’s pipeline creation:

VkDynamicState dynamicStates[2] = {
  VK_DYNAMIC_STATE_VIEWPORT,
  VK_DYNAMIC_STATE_SCISSOR,
};

VkPipelineDynamicStateCreateInfo dynamicInfo = {
  .sType = VK_STRUCTURE_TYPE_PIPELINE_DYNAMIC_STATE_CREATE_INFO,
  .pNext = nullptr,
  .flags = 0,
  .dynamicStateCount = 2,
  .pDynamicStates = dynamicStates,
};

VkGraphicsPipelineCreateInfo pipelineCreateInfo = {
  .sType = VK_STRUCTURE_TYPE_GRAPHICS_PIPELINE_CREATE_INFO,
  ...
  .pDynamicState = &dynamicInfo,
  ...
};

VkCreateGraphicsPipelines(device, VK_NULL_HANDLE, 1, &pipelineCreateInfo, nullptr, &mPipeline);

The actual computation of the viewport extent during command buffer recording looks like this:

int x = 0, y = 0, w = 500, h = 400;

glm::vec4 viewportData;

switch (device->GetPretransformFlag()) {
  case VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR:
    viewportData = {bufferWidth - h - y, x, h, w};
    break;
  case VK_SURFACE_TRANSFORM_ROTATE_180_BIT_KHR:
    viewportData = {bufferWidth - w - x, bufferHeight - h - y, w, h};
    break;
  case VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR:
    viewportData = {y, bufferHeight - w - x, h, w};
    break;
  default:
    viewportData = {x, y, w, h};
    break;
}

const VkViewport viewport = {
    .x = viewportData.x,
    .y = viewportData.y,
    .width = viewportData.z,
    .height = viewportData.w,
    .minDepth = 0.0F,
    .maxDepth = 1.0F,
};

vkCmdSetViewport(renderer->GetCurrentCommandBuffer(), 0, 1, &viewport);

The x and y variables define the coordinates of the top left corner of the viewport, while w and h define the width and height of the viewport respectively. The same computation can be used to also set the scissor test, and is included below for completeness:

int x = 0, y = 0, w = 500, h = 400;
glm::vec4 scissorData;

switch (device->GetPretransformFlag()) {
  case VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR:
    scissorData = {bufferWidth - h - y, x, h, w};
    break;
  case VK_SURFACE_TRANSFORM_ROTATE_180_BIT_KHR:
    scissorData = {bufferWidth - w - x, bufferHeight - h - y, w, h};
    break;
  case VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR:
    scissorData = {y, bufferHeight - w - x, h, w};
    break;
  default:
    scissorData = {x, y, w, h};
    break;
}

const VkRect2D scissor = {
    .offset =
        {
            .x = (int32_t)viewportData.x,
            .y = (int32_t)viewportData.y,
        },
    .extent =
        {
            .width = (uint32_t)viewportData.z,
            .height = (uint32_t)viewportData.w,
        },
};

vkCmdSetScissor(renderer->GetCurrentCommandBuffer(), 0, 1, &scissor);

Consideration - Fragment Shader Derivatives

If your application is using derivative computations such as dFdx and dFdy, additional transformations may be needed to account for the rotated coordinate system as these computations are executed in pixel space. This requires the app to pass some indication of the preTransform into the fragment shader (such as an integer representing the current device orientation) and use that to map the derivative computations properly:

  • For a 90 degree pre-rotated frame
    • dFdx needs to be mapped to dFdy
    • dFdy needs to be mapped to -dFdx
  • For a 270 degree pre-rotated frame
    • dFdx needs to be mapped to -dFdy
    • dFdy needs to be mapped to dFdx
  • For a 180 degree pre-rotated frame,
    • dFdx needs to be mapped to -dFdx
    • dFdy needs to be mapped to -dFdy

Conclusion

In order for your application to get the most out of Vulkan on Android, implementing pre-rotation is a must. The most important takeaways from this article are:

  • Ensure that during swapchain creation or recreation, the pretransform flag is set so that it matches the flag returned by the Android operating system. This will avoid the compositor overhead.
  • Keep the swapchain size fixed to the identity resolution of the app's window surface in the display’s natural orientation.
  • Rotate the MVP matrix in clip space to account for the devices orientation since the swapchain resolution/extent no longer update with the orientation of the display.
  • Update viewport and scissor rectangles as needed by your application

Sample App: Minimal Android pre-rotation