most css-counter-styles test results aren't shown

[heycam]

https://wpt.fyi/css/css-counter-styles shows only one test, but there are more tests in sub-directories under there.

[jugglinmike]

Results aren't available for those files because we don't run those tests. Here's what the WPT CLI reports for the contents of that directory (pardon the debugging information):

$ ./wpt run --list-tests --include css/css-counter-styles firefox
Using certutil /usr/bin/certutil
Using webdriver binary /home/mike/projects/bocoup/google-wpt/web-platform-tests/_venv/bin/geckodriver
Using cached test prefs from /home/mike/projects/bocoup/google-wpt/web-platform-tests/_venv/profiles/59.0.2-stable.cache
 0:01.78 INFO Updating test manifest /home/mike/projects/bocoup/google-wpt/web-platform-tests/MANIFEST.json
INFO:manifest:Skipping manifest download because existing file is recent
INFO:manifest:Updating manifest
DEBUG:manifest:Opening manifest at /home/mike/projects/bocoup/google-wpt/web-platform-tests/MANIFEST.json
/css/css-counter-styles/broken-symbols.htm
$

According to the expected format of the supported test types, this is expected. That's because searching for the supported patterns only returns the one file:

$ grep -E '(mis)?match|testharness' css/css-counter-styles/ -rl
css/css-counter-styles/broken-symbols.htm
$ find css/css-counter-styles/ -name '*visual*' -o -name '*manual*' -o -name '*any*' -o -name '*worker*' -o -name '*window*'
$

These tests appear to require a human operator to validate the visual rendering. If that's true, then they are currently out-of-scope for reporting on https://wpt.fyi. Also in that case: we should re-name the tests to honor the documented convention.

@heycam or @gsnedders: can you fill me in on how these tests are expected to be executed?

[heycam]

Oh, indeed you're right. It looks like those tests are all manual tests. We should probably convert them to reftests at some point, but it makes sense they're not run.

[gsnedders]

Also in that case: we should re-name the tests to honor the documented convention.

We have thousands of tests in css/ that don't follow the larger convention; we deliberately added support for <meta name=flags> when merging the test suites.

Oh, indeed you're right. It looks like those tests are all manual tests. We should probably convert them to reftests at some point, but it makes sense they're not run.

See web-platform-tests/wpt#8122 to add replacements from m-c. Note we also have https://wpt.fyi/css/vendor-imports/mozilla/mozilla-central-reftests/counter-styles-3 (and see https://bugzilla.mozilla.org/show_bug.cgi?id=1423971 for why that doesn't get merged in to the rest of the testsuite…).

[gsnedders]

See also web-platform-tests/wpt.fyi#56