How to separate tests?

[sellout]

I have a package that has some initialization-time stuff that needs to be handled carefully. What I’ve done is split up my tests so I have the standard library tests, and then a number of separate test files that are each intended to be run in a fresh Emacs instance, so I can make sure different user init styles all work correctly. What I first tried with Eldev is

;; in Eldev
(setf eldev-test-fileset `(:and ,eldev-test-fileset (:not "./tests/initialization")))

# test the standard library functionality
eldev test

# run each initialization test on its own
init_tests=($(IFS=$'\n' find ./tests/initialization -type f -name '*.el'))
for file in "''${init_tests[@]}"; do
  eldev test -f "$file"
done

But, since the initialization tests aren’t in the test fileset, eldev skips them, even though they’re listed by name.

However, if I leave them in the fileset, then

1. contributors running eldev test will get a bunch of failures in those initialization tests and
2. I have to explicitly enumerate the files for the “standard” tests.

I’m just curious how I should approach this. I feel like I’m probably overlooking or misunderstanding something about filesets, but I couldn’t find anything relevant in the manual.

How would you go about handling this situation?

[doublep]

You could maybe base on a somewhat-similar separation in Eldev's own file Eldev.

I have tests that are self-contained (use only Eldev itself) and tests that depend on external projects, e.g. Buttercup or Relint. The latter I call "integration tests"; you can just search the file for "integration". The two added commands are actually only sugarcoating, the important thing are the fileset definitions and the test selection option.

This doesn't include "one test per Emacs instance" of course, but as I have understood, you figured out that part yourself.

[sellout]

Thanks, that did it!

Here’s what I ended up with in my Eldev:

;; This needs to be set to have “eldev test” run only the standard tests
;; if no `local-test-selection' option is provided.
(setf eldev-test-fileset
  `(:and ,eldev-test-fileset (:not "./tests/initialization")))

(eldev-defoption local-test-selection (type)
  "Select tests to run; type can be `main' or `integration'"
  :options        (-T --test-type)
  :for-command    test
  :value          TYPE
  :default-value  'main
  (pcase (intern type)
    ('main ())
    ('integration (setf eldev-test-fileset "./tests/initialization"))
    (_ (signal 'local-wrong-option-usage `("unknown test type `%s'" ,type)))))

One thing that confused me at first is that that :default-value only applies to the command if an option is provided with no value – the body doesn’t get called at all if the option isn’t explicitly provided to the command. Which is fine – it just meant I had to set up eldev-test-fileset outside of eldev-defoption for the default case. But it could be documented.

[doublep]

Are the docstring improvements in commit afeacd6 good enough?

[sellout]

Yeah, that sounds good to me, thanks!