Black-box testing vs White-box testing

[smol-ninja]

Flow follows the white-box testing approach.

From ChatGPT

What is white-box testing?

White-box testing requires understanding the internal structure of the code. It focuses on code paths, loops, and conditions, ensuring that all parts of the code work as expected. White-box testing is often used for unit and integration testing where in-depth knowledge of the code is necessary.

Should testing of a code be dependent on the internal logic/implementation?

No, testing of a code should not depend on the internal logic or implementation. Instead, it should focus on testing the external behavior of the code—this is called black-box testing. Tests should validate that the code meets the specified requirements and produces the expected results for given inputs, regardless of how the code is implemented internally. This makes the tests resilient to changes in the code structure, as long as the external behavior remains consistent.

What is black-box testing?

Black-box testing focuses on testing the external behavior of the software without knowledge of the internal code. It checks whether the input produces the expected output without considering the actual implementation. This method is useful for functional testing and UI testing, where you are more interested in how the system behaves rather than how it is built

Currently, all Flow tests (concrete, fuzz, invariant) follow white-box testing, which means if we change an internal implementation of a function without affecting the APIs (inputs, outputs) and expected returned values, we would also be required to change a lot of tests. IMO a good testing system should test for the behavior of the system and not rely on the internal implementation of the functions.

A good approach would be to have white-box testing in concrete whereas black-box testing in fuzz and invariant, though more knowledge is needed in this domain.

So, starting this discussion to brainstorm on these lines, gather resources, understand it better and eventually refactor our tests to use a mix of black and white box techniques.

Resources

• https://chatgpt.com/share/670e4dd3-e2e8-8000-8dd2-d372b26a5761

cc @sablier-labs/engineers as it may be relevant to Solana and other repos.

[PaulRBerg]

Interesting concept.

However, I think that the line is fuzzier (pun-intended) between white-box and black-box testing than it may seem at first blush.

Presumably, black-box testing involves writing middleware that converts the internal logic into some abstracted interfaces. So then the maintenance moves to the middlewares.

I could be wrong, though. A PR that shows how black-box testing works in practice would be helpful.