Review of existing tests' ability to be automated

[zcorpan]

Background

In the ARIA-AT Automation Design Doc, we discussed that the existing ARIA AT tests may need some changes to allow for automation without ambiguity in the test instructions. This issue reviews the existing tests and suggests changes we think are necessary for automation. These changes could make the tests clearer for manual testers as well.

View these suggestions as a starting point for discussion; they are not set in stone.

alert

Test instructions look OK wrt automation.

breadcrumb

Test instructions look OK wrt automation.

checkbox

PR: #447 (WIP)

• Should split “navigate forwards” and “navigate backwards” to separate tests.
• For tests that navigate forwards, insert a link before the checkbox and set focus to it with the setup script. Similar to checkbox-tri-state.
• For tests that navigate backwards, insert a link after the checkbox and set focus to it with the setup script. Similar to checkbox-tri-state.
• Remove “Note: it should be in the unchecked state.” from the instructions. This is covered in the success criteria.

checkbox-tri-state

• Remove “Note: it should be in the partially checked state.” from the instructions. This is covered in the success criteria.
• Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"

combobox-autocomplete-both-updated

• Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"

combobox-select-only

• Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"

command-button

Test instructions look OK wrt automation.

disclosure-faq

• Remove “Note: it should be in the collapsed state.” from the instructions. This is covered in the success criteria.
• Should use command sequences: “you may need to press Down Arrow multiple times”

disclosure-navigation

• Should use command sequences: "you may need to press certain commands (such as Down Arrow) multiple times"

horizontal-slider

This test is currently skipped in automation because key press instructions are ambiguous.

menu-button-actions

Test instructions look OK wrt automation.

menu-button-actions-active-descendant

Test instructions look OK wrt automation.

menubar-editor

• Split “Up Arrow / Down Arrow” into separate tests, with a link before or after that has initial focus.

minimal-data-grid

• Uses command sequences ("T followed by Down Arrow"), but not fully? "you may need to press certain commands (such as Down Arrow) multiple times"

modal-dialog

Test instructions look OK wrt automation.

radiogroup-aria-activedescendant

• Should use command sequences: “you may need to press Down Arrow multiple times”

radiogroup-roving-tabindex

• Should use command sequences: “you may need to press Down Arrow multiple times”

rating-slider

This test is currently skipped in automation because key press instructions are ambiguous.

tabs-manual-activation

• Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"

toggle-button

Test instructions look OK wrt automation.

vertical-temperature-slider

This test is currently skipped in automation because key press instructions are ambiguous.

[jscholes]

“Insure NVDA is in browse mode by pressing Escape. Note: This command has no effect if NVDA is already in browse mode.”

See #382.

[zcorpan]

I've updated the OP with my findings for the current state of the tests.

The main thing is that some of the tests aren't using command sequences yet.

menubar-editor also uses the old "Up Arrow / Down Arrow" pattern in one test.