Automated testing: how should we handle non-speech output, such as earcons?

[mfairchild365]

Our current approach for automated testing simply looks at speech output, but support for some features might not be communicated via speech.

For example, on the HTML side of things, VO on MacOS will use earcons to convey the boundaries of a native select listbox.

I'm struggling to come up with other examples where non-speech output like earcons are used to convey information. In my experience, this is the the exception rather than the norm. But that doesn't mean we shouldn't test it.

Does anyone have other examples?

[sinabahram]

sorry to take a hard line on this, but this is a basic equity question. I find this to be a complete failure of any assistive technology that only uses sound. If they only use sound, how do they expect a deaf-blind user to use it? Any screen reader should fully fail any test, what-so-ever, that does not have the option of conveying this information via speech as well, because that means deaf-blind users won't have access. We should not have sound testing in scope what-so-ever for years, because it's not necessary.

[mfairchild365]

Fair enough, @sinabahram. I had assumed that such earcons or non-speech output would be conveyed via refreshable braille display in some way, but I agree that isn't a safe assumption.

[sinabahram]

If they are, that's good, though, because then we should be fine on capturing it. If there is an edge case where it is only a sound effect through speech and also a label in Braille, sure, that's an edge case, but I can't think of a single such implementation, and plus that's again a terrible SR design, I feel.

For example, in Jaws, one of the first things I do is switch the sound off for forms mode, forcing it to say it with speech.

[mfairchild365]

Thanks, @sinabahram. I honestly know very little about how refreshable braille displays work and how screen readers support them.

I think this warrants further investigation as it might affect assertion writing, test writing, and automation.

It sounds like:

• For automation, if our implementation is only intercepting speech output, we might need to look elsewhere for non-speech information. Perhaps a braille driver, or somehow configure the screen reader to send that information as speech. If we need to configure the screen reader to send the information as speech, this raises a yellow flag for me because we will have non-default settings.
• For test writing and assertion writing, is it okay if support for a given feature is conveyed via earcon? If not, is that documented and clear to manual testers?

curious what @mcking65 thinks about this.

[WestonThayer]

This issue came up in today's AT Automation meeting (@jugglinmike tested with Windows screen readers and confirmed that the 3rd party "automation voice" approach, described in w3c/aria-at-automation-driver#1) does not capture earcons — the screen reader appears to handle the audio itself).

I brought up that NVDA currently does not send live region updates to refreshable braille displays, there's a discussion about the behavior here: nvaccess/nvda#7756. In terms of equity, the issue seems similar to earcons. We were curious to get other's thoughts on how to test live regions.

[jscholes]

@WestonThayer If a screen reader was to only convey some information via an earcon, it would fail the assertion being tested for. So I claim that the fact that they wouldn't be captured by the automated test runner is not a problem that needs to be solved. If you disagree and have one or more specific examples of where capturing the playback of a specific earcon would be necessary, let's talk through it/them.

In terms of braille output: the ARIA-AT project is currently entirely focused on speech output. It's what we tell testers to use to gauge whether or not an assertion has passed, it's the output we ask them to paste into the testing form, etc. So again, is this a problem that needs to be solved in the short-term? I don't know where braille support lies on the project roadmap, as it would also imply testing braille-display-specific commands.

In the long-term, we can either:

• create a fake HID braille display; or
• hook into the braille processing at the individual screen reader level, e.g. via custom/virtual braille drivers.

Finally, live regions: taking braille out of the equation, that becomes an easy one. Trigger the live region announcement on click/press of a button, and assert that the known/expected text is spoken. At a high level, that's what we'll need to do for pretty much all tests anyway. At a more specific level, it's also what we do for tests targeted at the APG's Alert example.

[WestonThayer]

Great! Thank you for the clarity. I agree with all your points.