Most Android apps are built with Android native components, while some of them build with 3rd party frameworks or components. Like some apps built with Custom View. And usually, you may draw your UX with non native tools, like OpenGL, Canvas etc.
If so most accessibility services (Talkback, SwitchAccess) may not work well on those kinds of apps. Some issues below may be raised with Talkback ON:
- The accessibility focus, the green rectangle, disappears on your app.
- The accessibility focus selected the boundary of the whole screen.
- The accessibility focus is not movable.
- Four direction DPAD keys do not take effect, even if you set handler.
In that case, you need to make sure that your app exposes its AccessibilityNodeInfo tree to the accessibility services.
Root Cause: DPAD Events consumed by accessibility services
The main reason for the issue is that the DPAD key events have been consumed by accessibility services. And those will not pass over to your App. Yes, the issue is raised by the Android Accessibility Suite, itself.
As the figure above, when Talkback ON, the DPAD event will not pass to the DPAD handler created by App developer. That's the reason why your App cannot receive the DPAD events.. Accessibility services receives the DPAD event and then it controls the accessibility focus moving. However, because the non-native Android components created by non-native Android framework, the informations of those components is not exposed to accessibility services. As a result, the accessibility services does not know what and where UI components are on the screen, so the accessibility focus cannot move.
In conclusion, the reasons why apps developed with custom view do not work with Talkback are:
- DPAD key events consumed by accessibility services
- Accessibility services do not know what and where the UI components are on the screen
And it also impact the SwitchAccess service. Because the SwitchAccess navigation also depends on the AccessibilityNodeInfo tree.
To solve the issue, we should focus on those two parts.
Why the DPAD key events are being consumed
On non accessibility mode, Android TV only moves focus on necessary focusable elements, such as buttons, links and icons skipping all elements that the sighted user can read. But on accessibility mode, accessibility focus should also move to text only elements and speak them out. To solve this, the Accessibility Services on TV need to intercept the keypad events to move both input and accessibility focus in sync.
Exposing information to accessibility services.
As we mentioned in the last section, to the accessibility services, the UI components, like button, link, list, text description drawed by non-native framework, are unknown components. The services does not know their location and anything else information. Thus to solve the issue, we should tell the accessibility services all it need to know.
The AccessibilityNodeInfo is the class to store the information for each component. And then we can use ExploreByTouchHelper to define and expose all components information to the services. And then use setAccessibilityDetegate to set the ExploreByTouchHelper object.
What you should do is to create a new class to inherit the ExploreByTouchHelper. And then override its 4 methods in the ExploreByTouchHelper here:
// Return the virtual view ID whose view is covered by the input point (x, y). protected fun getVirtualViewAt(x:Float, y:Float):Int // Fill the virtual view ID list into the input parameter virutalViewIds. protected fun getVisibleVirtualViews(virtualViewIds:List<Int>) // For the view whose virtualViewId is the input virtualViewId, populate the // accessibility node information into the AccessibilityNodeInfoCompat parameter. protected fun onPopulateNodeForVirtualView(virtualViewId:Int, @NonNull node:AccessibilityNodeInfoCompat) // Set the accessibility handling when perform action. protected fun onPerformActionForVirtualView(virtualViewId:Int, action:Int, @Nullable arguments:Bundle):Boolean
// Return the virtual view ID whose view is covered by the input point (x, y). protected int getVirtualViewAt(float x, float y) // Fill the virtual view ID list into the input parameter virutalViewIds. protected void getVisibleVirtualViews(List<Integer> virtualViewIds) // For the view whose virtualViewId is the input virtualViewId, populate the // accessibility node information into the AccessibilityNodeInfoCompat parameter. protected void onPopulateNodeForVirtualView(int virtualViewId, @NonNull AccessibilityNodeInfoCompat node) // Set the accessibility handling when perform action. protected boolean onPerformActionForVirtualView(int virtualViewId, int action, @Nullable Bundle arguments)
What information you need to expose
To make sure the component visible to accessibility services, you have to build its own AccessibilityNodeInfo for each component. And make sure below items:
Required AccessibilityNodeInfo.getBoundsInScreen() to set the position of the component.
Required AccessibilityNodeInfo.setVisibleToUser() should be true to make the virtual node visible.
Required AccessibilityNodeInfo.getContentDescription() should set the content description for the Talkback to announce.
AccessibilityNodeInfo.setClassName() should be set to allow services distinguish the component type.
If we want to implement more ACTION types, like ACTION_CLICK, invoke AccessibilityNodeInfo.addAction(ACTION_CLICK) to add the action. And also add handling logic in performAction() method.
The more methods you set in AccessibilityNodeInfo, the more information the accessibility services know your components more.
Here is the best practices document. This document builds a simple App with CustomView and friendly accessibility support.