-
-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Introduce a way to suppress violations #119
base: main
Are you sure you want to change the base?
Conversation
|
|
||
Furthermore, we are adding one more reason to exit with an error code (see "Maintaining a lean baseline"). This might have some negative side-effects to wrapper scripts that assume that error messages are available when that happens. We could introduce a different exit code, to differentiate between exiting due to unresolved errors or ignored errors that do not occur anymore. | ||
|
||
## Alternatives |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the alternative we've introduced at Canva is that we designate specific rules as "in migration" and we only consider reports from those rules if they exist in changed files (according to git
comparison against the main branch).
With this system developers must address lint errors if they touch a file but otherwise they can be ignored.
This does require integration with the relevant source control system - though we've found it works quite well.
Thanks for the RFC @softius. Can you please sign the CLA as requested in this comment? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for putting this together. It sound good in theory, but you haven't provided much in the way of implementation details. Please dig into the code a bit to see how this might be implemented.
@octogonz Would really like your feedback on this based on your work on Rush Stack Bulk. |
The feature that we designed is part of @rushstack/eslint-patch. Kevin Yang summarized it in this blog post. A key design question is how to keep bulk suppressions stable across Git merges, since they are stored separately from the associated source code. From a cursory glance, it sounds like this RFC is proposing to count the number of errors per file path. We prototyped that design, as well as a design that tracks line #'s, and also a design that tracks formatted messages ( Ultimately we settled on a JSON file using a There's probably still a lot that can be improved about our approach. However I can say that Microsoft and TikTok are using it successfully in two very large TypeScript monorepos, where style guide migrations involve hundreds of thousands of source files and megabytes of JSON. So it's worth studying even if you don't adopt it. 🙂 As far as future improvements, a feature I'd like to see is an API that enables the VS Code extension to highlight suppressed violations specially. Currently people either highlight them as normal errors (which I find confusing, since you have to run the CLI to find out whether you really need to fix it or not) or else don't highlight them at all. |
@octogonz thanks for the insights. I think we're past the point of adopting eslint-bulk wholesale, but would welcome your feedback on this RFC to see if there are any salient points that have been missed or problem areas you've encountered. |
Agreed, the I've been really busy lately, but I'd like to help out with the RFC. Let me make some time to read through @softius's doc in more detail, then follow up with feedback. |
@softius just a reminder that there are some suggestions for you to apply. Please, let us know if you need any help. |
I really support this RFC - we've been running our own baseline package (just a wrapper around eslint), which takes the same approach. We've been using this implementation on about 30+ prod projects with no complaints from our team. Hopefully, this implementation can assist with the RFC |
Co-authored-by: Nicholas C. Zakas <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for the delay in getting back to this. Had a lot of post-v9 work.
I think there are some helpful ideas that eslint-bulk has that we should consider here:
- Using the term "suppressions" instead of "baseline" - I think this clarifies what is actually happening. While people may not understand what generating a baseline is, most will understand when something is suppressed.
- A way to indicate which rules should be suppressed. For instance, if I just want to suppress "no-eval", I should be able to do that. That may mean rethinking the command line flags. So maybe
--suppress-rule <ruleId>
and--suppress-all
for all rules?
@Humni that's great! Do you have any specific feedback on the implementation described in this proposal? How does it compare to what you're already using? |
Hi Nicholas, thanks for checking out the eslint-bulk project!
Other names we considered were exemptions, pardons, and legacy-suppressions. On a semi-related note, the "baselines" feature seems to very closely resemble the already existing eslint --max-warnings flag, which might be a source of confusion if the two features are both retained. |
@kevin-y-ang I'm not sure I follow. Can you explain what you mean by this? |
At a conceptual level, both the proposed baseline feature and the --max-warnings feature set a threshold of X errors where the system will alarm if there are >X errors. At our company, we previously used The message essentially being conveyed here is "Okay there are already X number of ESLint warnings in this package but we won't allow any more", while the message being conveyed with the baseline feature seems to be a more granular version of this: "Okay there are already X number of ESLint errors/warnings for rule R in file F but we won't allow any more." Anyway, it's just a similarity I see, it's not a big deal. |
Probably the main decision to make for this RFC which style of error matching you use. These would be:
With our current implementation, we use the Exact Error Line Matching, which does have some short falls. It works pretty well, the vast majority of PRs don't require any rework. There are cases where if you modify a line at the top of a file though, will cause all of the existing errors to be exposed. This approach was chosen due to simplicity. For the most part this does push a project to having a smaller and smaller baseline, however it does lead to developers touching the baseline "too much" due to them not wanting to fix errors not directly related to their PR. We're currently exploring moving to a context aware error matching, so it only comes up if the nearest 3 lines of code are modified - LuminateOne/eslint-baseline#8. This would provide a solution to the issue above, but it's not yet tested but seems like a sensible approach (if it can be done in a performant way). I would highly recommend an error count approach the same as PHPStan (as per comments from @ondrejmirtes), as that seems to sit with the development team better. Only other question "feature request" I would add into this is to allow for "flushing" a baseline of excess/resolved errors only (and not add any new ones). Just a QOL improvement though as it's typically easy enough to see new lines added into the diff in the baseline. |
@Humni gotcha, that's helpful. So it sounds like the current RFC, which uses error counting, is what you'd prefer. 👍 Keep in mind, too, that we generally build things incrementally. Our goal here is to get a baseline (no pun intended) of support for suppressions such that we can continue to build on top of it as feedback comes in. |
@softius are you still working on this? There is some outstanding feedback to review. |
Co-authored-by: Francesco Trotta <[email protected]>
/** | ||
* Checks the provided suppressions against the lint results. | ||
* | ||
* For each error included in `results`, checks if the error and the file are in `suppressions`. | ||
* If yes, the count is decreased by 1 and ignores the error unless the count has reached zero. | ||
* Otherwise, it keeps the error. | ||
* | ||
* It returns the lint results that are not in the suppressions file, | ||
* as well as the unmatched suppressions. | ||
* | ||
* @param {LintResult[]} results The lint results. | ||
* @param {SuppressedViolations} suppressions The suppressions. | ||
* @returns {{ | ||
* results: LintResult[], | ||
* unmatched: SuppressedViolations | ||
* }} | ||
*/ | ||
applySuppressions(results, suppressions) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the update. Maybe I'm misunderstanding something, so bear with me. It would be helpful if you could add a draft implementation to show how this function will work.
To clarify, this is an example I would like to understand. Suppose you have a file test.js where the no-undef
rule would normally report three errors:
/* eslint no-undef: "error" */
a = b + c;
What will the function return when the suppression file specifies a count
of 2? This is what the arguments will look like:
const results = [{
filePath: '/my-project/test.js',
messages: [{
ruleId: 'no-undef',
severity: 2,
message: "'a' is not defined.",
line: 3,
column: 1,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 2
},
{
ruleId: 'no-undef',
severity: 2,
message: "'b' is not defined.",
line: 3,
column: 5,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 6
},
{
ruleId: 'no-undef',
severity: 2,
message: "'c' is not defined.",
line: 3,
column: 9,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 10
}],
suppressedMessages: [],
errorCount: 3,
fatalErrorCount: 0,
warningCount: 0,
fixableErrorCount: 0,
fixableWarningCount: 0,
source: '/* eslint no-undef: "error" */\n\na = b + c;\n',
usedDeprecatedRules: []
}];
const suppressions = {
"test.js": {
"no-undef": {
count: 2
}
}
};
You can see that the input results
contains details about each error, while the suppressions are just numbers. What will the call to applySuppressions(results, suppressions)
return in this case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case it will return an updated copy of results and an empty object for unmatched (because all suppressions are matched).
// results
const results = [{
filePath: '/my-project/test.js',
messages: [
{
ruleId: 'no-undef',
severity: 2,
message: "'c' is not defined.",
line: 3,
column: 9,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 10
}],
suppressedMessages: [],
errorCount: 1,
fatalErrorCount: 0,
warningCount: 0,
fixableErrorCount: 0,
fixableWarningCount: 0,
source: '/* eslint no-undef: "error" */\n\na = b + c;\n',
usedDeprecatedRules: []
}];
// unmatched
{}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I understand correctly, this means removing the first count
messages with a particular rule-id
from each result, where count
is the suppression count. So, in the above example a
and b
would be suppressed but not c
, because in the input result, a
and b
are sorted first in order of location.
I think this behavior would be very surprising to users. This means that some rule violations would be reported only for (the bottom) part of a file.
I think it would be better to report all messages unfiltered if the suppression count is less than the number of messages, because in that situation we can't make a meaningful selection. So for each file and rule-id
we should always report either all messages, or none of them.
@eslint/eslint-tsc looking for another approval on this. |
|
||
### Suppressing violations of a specific rule | ||
|
||
A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values. | |
A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this option can accept an array of string values. |
* The developer runs `eslint --supress-all ./src` to create the suppressions file. | ||
* Running `eslint ./src` reports no violations and exits with status 0. | ||
* After fixing a violation, the suppressions file still contains the now-resolved violation. | ||
* Running `eslint ./src` again reports no violations but exits with a non-zero status code, indicating the suppressions file needs updating. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case, will there be an error message, and what would it look like? Technically, will it be a lint message passed to the formatter along with other lint messages for the file, or a separate output?
That's a good question. I think it makes sense to insert lint messages suppressed by this feature into |
I agree. Makes sense. 👍 |
Makes sense. I think that errors suppressed by the suppressions file will have no |
@softius I think all that's left is to update the RFC to mention |
@softius Any way we can help you to finish the work on this RFC? |
Co-authored-by: Milos Djermanovic <[email protected]>
RFC updated to indicate that matched messages will be moved from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM. Would like @mdjermanovic and @fasttime to verify before merging.
Thanks so much for all of your work and patience.
I think the following review comments haven't been addressed: |
2. **Match Errors Against Suppressions** | ||
* For each error, check if the error and the file are in the suppressions file. | ||
* If yes, decrease count, in memory, by 1 and move the message to `LintResult#suppressedMessages` unless count is zero. | ||
* If no, keep the error. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(related to the discussion under #119 (comment))
For one file and one rule, I think either all or none of the messages should be moved to suppressedMessages. By this algorithm, when the number of error messages is greater than the number stored in the suppressions file, some would be moved to suppressedMessages while some would stay in messages.
|
||
The suggested solution always compares against the existing suppressions file, typically `.eslint-suppressions.json`, unless `--suppressions-location` is specified. This makes it easier for existing and new projects to adopt this feature without the need to adjust scripts in `package.json` and CI/CD workflows. | ||
|
||
To perform the comparison, we will go through each result and message from `ESLint.lintFiles`, checking each error `(severity == 2)` against the suppressions file. By design, we ignore warnings since they don't cause eslint to exit with an error code and serve a different purpose. If the file and rule are listed in the suppressions file, we can move the message to `LintResult#suppressedMessages` and ignore the result message. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SuppressedLintMessage
should have an additional property suppressions
. We should specify what value it will have in this case. I think it can be:
suppressions: [{ kind: "file", justification: "" }]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mdjermanovic can you please clarify what would be the purpose of this new property?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It describes the reason why the lint message is suppressed. It's a mandatory property in SuppressedLintMessage
type (objects in LintResult#suppressedMessages
). Without it, integrations and formatters that use LintResult#suppressedMessages
could break.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @mdjermanovic . I was under the impression that you were referring to a new property for some reason. That is clear now.
@mdjermanovic as it seems like we are getting down to fine-tuning some details, can we move this to Final Commenting? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm fine with moving to Final Commenting with a note that when the number of errors exceeds the allowed number, all errors should be reported, as described in #119 (comment).
* If `--prune-suppressions` is passed, take the updated suppressions from memory to check which suppressions are left. | ||
* For each suppression left, update the suppressions file by either reducing the count or removing the suppression. | ||
4. **Report and exit** | ||
* Exit with a non-zero status if there are unmatched suppressions, optionally listing them in verbose mode. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By verbose mode we probably mean when the --debug
flag is passed on the command line.
/** | ||
* Removes old suppressions that do not occur anymore. | ||
* @returns {void} | ||
*/ | ||
prune() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this method needs to accept an argument with the updated count of suppressed errors per rule and per file.
Here is a high-level overview of the execution flow: | ||
|
||
1. **Check for Options** | ||
* If both `--suppress-all` and `--suppress-rule` are passed, exit with an error (these options are mutually exclusive). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems that --prune-suppressions
will have no effect when --suppress-all
or --suppress-rule
is passed. Should it be also disallowed when one of the other options is passed?
rfcs/designs/2024-baseline-support/README.md
Lines 224 to 230 in f134daa
if (options.suppressAll) { | |
suppressionsManager.suppressAll(results); | |
} else if (options.suppressRule) { | |
suppressionsManager.suppressByRule(results, options.suppressRule); | |
} else if (options.pruneSuppressions) { | |
suppressionsManager.prune(); | |
} |
log.error("There are left suppressions that do not occur anymore. Consider re-running the command with `--prune-suppressions`."); | ||
return 2; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question: do we really want to exit without printing the lint results here?
Summary
Suppress existing violations, so that they are not being reported in subsequent runs. It allows developers to enable one or more lint rules and be notified only when new violations show up.
Related Issues