OWASP ASVS overpromising on Level 1 requirements being testable without source code

[kwwall]

These comments are based off the latest stable ASVS release, v4.0.3. Also, this is solely my opinion so take it FWIW. Also, apologies if the ASVS OWASP Slack channel is the preferred venue for this sort of discussion, but I just find Slack harder to follow threads in than here.

The OWASP ASVS v4.0.3 states:

Level 1 controls can be checked either automatically by tools or simply manually without access to source code.

And, then on page 11, it repeats the same sentiment:

Level 1 is the only level that is completely penetration testable using humans. All others require access to documentation, source code, configuration, and the people involved in the development process. However, even if L1 allows "black box" (no documentation and no source) testing to occur, it is not an effective assurance activity and should be actively discouraged.

I always knew that there were unstated exceptions. For example, there are logging-related requirements such as requirement ID 7.1.2:

Verify that the application does not log other sensitive data as defined under local privacy laws or relevant security policy.

In theory at least, you can do that verification if you can ensure in a black-box pen-test that you have triggered all the log events, but in practice really is not feasible. (Not to mention that most pen-testers do not have access to the logs.) So, I was long aware of those special cases, but the seemed minimal, so it was never a big deal.

But while going through an exercise and scrutinizing every L1 requirement, I just found a bunch of requirements where verifying them without access to source code is IMHO, just a pipe dream. For instance, I think that most of the Level 1 requirements in the "Validation, Sanitization and Encoding" section are not generally testable in any reasonable fashion in most typical cases without access to source code unless when it is written "Verify that ..." it is only referring to a few cases and certainly not all of them, and probably not even most of them.

So, in that sense, I think those 2 quotes that I lead with from ASVS v4.0.3 need to be tempered a bit for 5.0.0 or otherwise we need to revise a lot of L1 requirements to get rid of false expectations. And since I think we are (and should be!) reluctant to remove the 'level 1' checkbox for many of these security requirements because they are so absolutely fundamental as an absolute minimal baseline, maybe another alternative is to simply add an additional column to the 5.0.0 tables that denotes whether the requirement is realistically testable without access to source code. In practice, since most of us do have source code access, maybe that doesn't matter as much as we think it does or should. But right now, this over-promising seems almost a case of ultra-hype and therefore (unintentionally) misleading.

I'd be happy to go into a few specific examples if someone wants more details. But if no one shares this as a concern, it may be a moot point and a waste of my time to explain details that no one cares about, so I shall wait to see if anyone asks for specific examples.

FWIW,
-kevin

[cmlh]

#1127 is the parent issue of this discussion.

[kwwall]

Thanks for the pointer.

[elarlang]

more precise and opened issue on the topic is #1495

[tghosth]

From a 4.0.3 perspective, L1 should be about 90-95% pen testable although that does depend a bit on how much assurance you want to get.

For 5.0 it is not 100% clear if we will keep the same approach or not, we have seen various organizations say that L1 is not enough for them anyway which makes the point a little moot.

You may also be interested in this document where someone gave a perspective on the information required to test various requirements