Configurable linter commands

CHANGELOG.md

@@ -11,6 +11,7 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
 - Core
   - PHP Linters use now the `bartlett/sarif-php-converters` first official release 1.0.0 to generate SARIF reports
   - [Upgrade PHP engine from 8.3 to 8.4](https://github.com/oxsecurity/megalinter/issues/4351) and allow Psalm 5.26 to run on this context (by @llaville)
+  - Linters can specify in the pre/post commands with a `run_before_linters` parameter whether the command is to be executed before the execution of the linters themselves (by @bdovaz in [#4482](https://github.com/oxsecurity/megalinter/pull/4482))
 
 - New linters
   - Reactivate clj-style (Clojure formatter) since its bug is fixed

README.md

@@ -77,7 +77,7 @@ Before going below, see [**Online Documentation Web Site which has a much easier
   - [Which version to use ?](#which-version-to-use-)
   - [GitHub Action](#github-action)
   - [GitLab CI](#gitlab-ci)
-    - [Manual Setup](#manual-setup)
+  - [Manual Setup](#manual-setup)
     - [Using R2Devops](#using-r2devops)
   - [Azure Pipelines](#azure-pipelines)
     - [Single Repository](#single-repository)
@@ -1182,6 +1182,7 @@ PRE_COMMANDS:
   - command: npm install eslint-plugin-whatever
     cwd: root        # Will be run at the root of MegaLinter docker image
     secured_env: true  # True by default, but if defined to false, no global variable will be hidden (for example if you need GITHUB_TOKEN)
+    run_before_linters: True # Will be run before the execution of the linters themselves, required for npm/pip commands that cannot be run in parallel
   - command: echo "pre-test command has been called"
     cwd: workspace   # Will be run at the root of the workspace (usually your repository root)
     continue_if_failed: False  # Will stop the process if command is failed (return code > 0)
@@ -1195,15 +1196,16 @@ PRE_COMMANDS:
     tag: before_plugins # Tag indicating that the command will be run before loading plugins
 ```
 
-| Property               | Description                                                                                                                              | Default value |
-|------------------------|------------------------------------------------------------------------------------------------------------------------------------------|---------------|
-| **command**            | Command line to run                                                                                                                      | Mandatory     |
-| **cwd**                | Directory where to run the command (`workspace` or `root`)                                                                               | `workspace`   |
-| **secured_env**        | Apply filtering of secured env variables before calling the command (default true)<br/>Be careful if you disable it !                    | `true`        |
-| **continue_if_failed** | If set to `false`, stop MegaLinter process in case of command failure                                                                    | `true`        |
-| **venv**               | If set, runs the command into the related python venv                                                                                    | <!-- -->      |
-| **output_variables**   | ENV variables to get from output after running the commands, and store in MegaLinter ENV context, so they can be reused in next commands | `[]`          |
-| **tag**                | Tag defining at which commands entry point the command will be run   (available tags: `before_plugins`)                                  | <!-- -->      |
+| Property               | Description                                                                                                                                     | Default value |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| **command**            | Command line to run                                                                                                                             | Mandatory     |
+| **cwd**                | Directory where to run the command (`workspace` or `root`)                                                                                      | `workspace`   |
+| **run_before_linters** | If set to `true`, runs the command before the execution of the linters themselves, required for npm/pip commands that cannot be run in parallel | `false`       |
+| **secured_env**        | Apply filtering of secured env variables before calling the command (default true)<br/>Be careful if you disable it !                           | `true`        |
+| **continue_if_failed** | If set to `false`, stop MegaLinter process in case of command failure                                                                           | `true`        |
+| **venv**               | If set, runs the command into the related python venv                                                                                           | <!-- -->      |
+| **output_variables**   | ENV variables to get from output after running the commands, and store in MegaLinter ENV context, so they can be reused in next commands        | `[]`          |
+| **tag**                | Tag defining at which commands entry point the command will be run   (available tags: `before_plugins`)                                         | <!-- -->      |
 
 <!-- config-precommands-section-end -->
 <!-- config-postcommands-section-start -->

megalinter/Linter.py

@@ -805,7 +805,7 @@ def update_active_if_file_found(self):
             self.active_only_if_file_found.append(self.config_file_name)
 
     # Processes the linter
-    def run(self):
+    def run(self, run_before_linters=None):
         self.start_perf = perf_counter()
 
         # Initialize linter reports
@@ -816,7 +816,7 @@ def run(self):
         self.before_lint_files()
 
         # Run commands defined in descriptor, or overridden by user in configuration
-        pre_post_factory.run_linter_pre_commands(self.master, self)
+        pre_post_factory.run_linter_pre_commands(self.master, self, run_before_linters)
 
         # Lint each file one by one
         if self.cli_lint_mode == "file":
@@ -871,7 +871,7 @@ def run(self):
             os.remove(self.remote_ignore_file_to_delete)
 
         # Run commands defined in descriptor, or overridden by user in configuration
-        pre_post_factory.run_linter_post_commands(self.master, self)
+        pre_post_factory.run_linter_post_commands(self.master, self, run_before_linters)
 
         # Generate linter reports
         self.elapsed_time_s = perf_counter() - self.start_perf

megalinter/MegaLinter.py

@@ -48,7 +48,7 @@ def run_linters(linters, request_id):
     global REQUEST_CONFIG
     config.set_config(request_id, REQUEST_CONFIG)
     for linter in linters:
-        linter.run()
+        linter.run(run_before_linters=False)
     return linters
 
 
@@ -244,7 +244,17 @@ def run(self):
             config.get(self.request_id, "PARALLEL", "true") == "true"
             and len(active_linters) > 1
         ):
+            for active_linter in active_linters:
+                pre_post_factory.run_linter_pre_commands(
+                    active_linter.master, active_linter, run_before_linters=True
+                )
+
             self.process_linters_parallel(active_linters, linters_do_fixes)
+
+            for active_linter in active_linters:
+                pre_post_factory.run_linter_post_commands(
+                    active_linter.master, active_linter, run_before_linters=True
+                )
         else:
             self.process_linters_serial(active_linters)
 

megalinter/descriptors/schemas/megalinter-descriptor.jsonschema.json

@@ -37,6 +37,20 @@
       "title": "Command",
       "type": "object"
     },
+    "linter_command_info": {
+      "description": "Command information",
+      "$ref": "#/definitions/command_info",
+      "properties": {
+        "run_before_linters": {
+          "default": false,
+          "title": "Process command before running linters",
+          "type": "boolean"
+        }
+      },
+      "required": [],
+      "title": "Command",
+      "type": "object"
+    },
     "enum_flavors": {
       "enum": [
         "all_flavors",
@@ -1232,7 +1246,7 @@
               ]
             ],
             "items": {
-              "$ref": "#/definitions/command_info"
+              "$ref": "#/definitions/linter_command_info"
             },
             "title": "Linter Pre-run commands",
             "type": "array"
@@ -1255,7 +1269,7 @@
               ]
             ],
             "items": {
-              "$ref": "#/definitions/command_info"
+              "$ref": "#/definitions/linter_command_info"
             },
             "title": "Linter Pre-run commands",
             "type": "array"

megalinter/pre_post_factory.py

@@ -34,19 +34,37 @@ def run_descriptor_post_commands(mega_linter, descriptor_id):
 
 
 # Commands to run before a linter (defined in descriptors)
-def run_linter_pre_commands(mega_linter, linter):
+def run_linter_pre_commands(mega_linter, linter, run_before_linters=None):
     if linter.pre_commands is not None:
+        filtered_commands: list = []
+
+        if run_before_linters is None:
+            filtered_commands = linter.pre_commands
+        else:
+            for command_info in linter.pre_commands:
+                if command_info.get("run_before_linters", False) is run_before_linters:
+                    filtered_commands += command_info
+
         return run_commands(
-            linter.pre_commands, "[Pre][" + linter.name + "]", mega_linter, linter
+            filtered_commands, "[Pre][" + linter.name + "]", mega_linter, linter
         )
     return []
 
 
 # Commands to run before a linter (defined in descriptors)
-def run_linter_post_commands(mega_linter, linter):
+def run_linter_post_commands(mega_linter, linter, run_before_linters=None):
     if linter.post_commands is not None:
+        filtered_commands: list = []
+
+        if run_before_linters is None:
+            filtered_commands = linter.post_commands
+        else:
+            for command_info in linter.post_commands:
+                if command_info.get("run_before_linters", False) is run_before_linters:
+                    filtered_commands += command_info
+
         return run_commands(
-            linter.post_commands, "[Post][" + linter.name + "]", mega_linter, linter
+            filtered_commands, "[Post][" + linter.name + "]", mega_linter, linter
         )
     return []