Google is committed to advancing racial equity for Black communities. See how.

Non Native Apps Accessibility Support on Android TV

Most Android apps are built with Android native components, while some of them build with 3rd party frameworks or components. Like some apps built with Custom View. And usually, you may draw your UX with non native tools, like OpenGL, Canvas etc.

If so most accessibility services (Talkback, SwitchAccess) may not work well on those kinds of apps. Some issues below may be raised with Talkback ON:

  • The accessibility focus, the green rectangle, disappears on your app.
  • The accessibility focus selected the boundary of the whole screen.
  • The accessibility focus is not movable.
  • Four direction DPAD keys do not take effect, even if you set handler.

In that case, you need to make sure that your app exposes its AccessibilityNodeInfo tree to the accessibility services.

Root Cause: DPAD Events consumed by accessibility services

The main reason for the issue is that the DPAD key events have been consumed by accessibility services. And those will not pass over to your App. Yes, the issue is raised by the Android Accessibility Suite, itself.

Dpad events consumption

As the figure above, when Talkback ON, the DPAD event will not pass to the DPAD handler created by App developer. That's the reason why your App cannot receive the DPAD events.. Accessibility services receives the DPAD event and then it controls the accessibility focus moving. However, because the non-native Android components created by non-native Android framework, the informations of those components is not exposed to accessibility services. As a result, the accessibility services does not know what and where UI components are on the screen, so the accessibility focus cannot move.

In conclusion, the reasons why apps developed with custom view do not work with Talkback are:

  • DPAD key events consumed by accessibility services
  • Accessibility services do not know what and where the UI components are on the screen

And it also impact the SwitchAccess service. Because the SwitchAccess navigation also depends on the AccessibilityNodeInfo tree.

To solve the issue, we should focus on those two parts.

Why the DPAD key events are being consumed

On non accessibility mode, Android TV only moves focus on necessary focusable elements, such as buttons, links and icons skipping all elements that the sighted user can read. But on accessibility mode, accessibility focus should also move to text only elements and speak them out. To solve this, the Accessibility Services on TV need to intercept the keypad events to move both input and accessibility focus in sync.

Exposing information to accessibility services.

As we mentioned in the last section, to the accessibility services, the UI components, like button, link, list, text description drawed by non-native framework, are unknown components. The services does not know their location and anything else information. Thus to solve the issue, we should tell the accessibility services all it need to know.

The AccessibilityNodeInfo is the class to store the information for each component. And then we can use ExploreByTouchHelper to define and expose all components information to the services. And then use setAccessibilityDetegate to set the ExploreByTouchHelper object.

Here is the presentation video published that in Google I/O 2013. You can also refer to the Android doc, Populate accessibility events to get more details.

What you should do is to create a new class to inherit the ExploreByTouchHelper. And then override its 4 methods in the ExploreByTouchHelper here:

Kotlin

// Return the virtual view ID whose view is covered by the input point (x, y).
protected fun getVirtualViewAt(x:Float, y:Float):Int

// Fill the virtual view ID list into the input parameter virutalViewIds.
protected fun getVisibleVirtualViews(virtualViewIds:List<Int>)

// For the view whose virtualViewId is the input virtualViewId, populate the
// accessibility node information into the AccessibilityNodeInfoCompat parameter.
protected fun onPopulateNodeForVirtualView(virtualViewId:Int, @NonNull node:AccessibilityNodeInfoCompat)

// Set the accessibility handling when perform action.
protected fun onPerformActionForVirtualView(virtualViewId:Int, action:Int, @Nullable arguments:Bundle):Boolean

Java

// Return the virtual view ID whose view is covered by the input point (x, y).
protected int getVirtualViewAt(float x, float y)

// Fill the virtual view ID list into the input parameter virutalViewIds.
protected void getVisibleVirtualViews(List<Integer> virtualViewIds)

// For the view whose virtualViewId is the input virtualViewId, populate the
// accessibility node information into the AccessibilityNodeInfoCompat parameter.
protected void onPopulateNodeForVirtualView(int virtualViewId, @NonNull AccessibilityNodeInfoCompat node)

// Set the accessibility handling when perform action.
protected boolean onPerformActionForVirtualView(int virtualViewId, int action, @Nullable Bundle arguments)

For more details, you can refer to the ExploreByTouchHelper or alternative useful class AccessibilityNodeProvider and the demo source code.

What information you need to expose

To make sure the component visible to accessibility services, you have to build its own AccessibilityNodeInfo for each component. And make sure below items:

  1. Required AccessibilityNodeInfo.getBoundsInScreen() to set the position of the component.

  2. Required AccessibilityNodeInfo.setVisibleToUser() should be true to make the virtual node visible.

  3. Required AccessibilityNodeInfo.getContentDescription() should set the content description for the Talkback to announce.

  4. AccessibilityNodeInfo.setClassName() should be set to allow services distinguish the component type.

  5. When overriding performAction() method, we should send AccessibilityEvent out.

  6. If we want to implement more ACTION types, like ACTION_CLICK, invoke AccessibilityNodeInfo.addAction(ACTION_CLICK) to add the action. And also add handling logic in performAction() method.

  7. Depends on your need, set setFocusable, setClickable, setScrollable and the similar methods as true.

  8. The more methods you set in AccessibilityNodeInfo, the more information the accessibility services know your components more.

Best Practices

Here is the best practices document. This document builds a simple App with CustomView and friendly accessibility support.