[Draft] Provide mapping from nextflow_schema.json to pipeline parameters format

[mzueva]

Background
Nextflow allows to describe pipeline parameters as nextflow_schema.json file, Cloud Pipeline uses a similar approach with parameters section in config.json file.

Approach

• Describe current supported format of parameters section in config.json file as part of platform documentation
• Describe pipeline parameters GUI as as part of platform documentation
• Provide a mapping between nextflow_schema.json and parameters section in config.json file as a comment to this issue

[tanas80]

Links

Cloud Pipeline Configuration
NextFlow Configuration

Parameters source

NextFlow nf-core pipeline does not receive parameters from environment variables. nf-core pipeline can receive parameters from the command line options or its config file (option -c).
For example:

nextflow run script.nf --parameter_name value -c nf-pipeline.config

or there should be a code for this in the pipeline itself like this:

params.parameter_name= env.parameter_name ?: 'default_value'

nf-core pipelines do not have such a code.

Then, if we want to set pipeline parameters in the platform config, we will need to transfer these parameters either to the command line options, or merge them into the config file in the params section before launching.

Properties/parameters mapping

bold - required to translate

nf-core config	Cloud Pipeline config.json	Translate	Description
$key (property name)	$key (parameter name)		Property/parameter name
default	value		Parameter default value
description	description		Parameter description
title	pretty_name		Parameter title
type	type	See Type mapping	Parameter value's type
format	type	Used with "type": "string", to determine target type to be input, output, path e.t.c. See Type mapping	Additional attribute to determine parameter type
hidden	visible	"hidden": true -> "visible": "false"
^.required	required	Specified at group level parameters
^.title	section	Specified at group level parameters
enum	enum	If the enum attribute is defined, then: - set the attribute "type": "enum" - copy the enum attribute as is	Options for selection
pattern	validation	Build validation from pattern. See validation attributes	Regular expression matching validation
minimum	validation	Build validation from minimum. See validation attributes	Validation for minimum acceptable value
maximum	validation	Build validation from maximum. See validation attributes	Validation for maximum acceptable value
oneOf	validation	Build validation using inner oneOf attributes See validation attributes	Object with conditions One of the conditions must be met
anyOf	validation	Build validation using inner anyOf attributes See validation attributes	Object with conditions Any of the conditions (or several) must be met
errorMessage	validation	Build validation using errorMessage on the top level.	Validation error message
exists, exist	not supported	May be useful for validation	Checks input file existence
minLength	not supported	Shall be implemented	Validation for minimum acceptable length of the value
maxLength	not supported	Shall be implemented	Validation for maximum acceptable length of the value
schema	not supported	May be useful for validation	Schema for input samplesheet columns
mimetype	not supported	May be ignored	Expected format of data in the input file, e.g.: "text/csv", "text/plain"
help_text, help	not supported	May be useful to display in tooltip	Detailed help how to use parameter, large text
fa_icon	not supported	May be ignored	Defines the icon to display near the property

Type mapping

Possible values of the nf-core pipeline attribute type: string, boolean, integer, number.
Translation to config.json attributes:

nf-core config	Cloud Pipeline config.json
"type":"boolean"	"type": "boolean"
"type":"integer"	"type": "string"
"type":"number"	"type": "string"
"type":"string"	Depends on other conditions: - if ($key (property name) contains "outdir" and "format":"directory-path") -> set "type":"output" - else if (any format is specified) -> set "type":"input" - else -> set "type":"string"

Please note, in nf-core config, type value can be an enumeration. For example: "type": ["string", "boolean"]. It is possible for cases with oneOf and/or anyOf constructions (see details below).

Group/section level parameters

Example of nf-core config group-level parameters:

"input_output_options": {
    "title": "Input/output options",                // <- transform to `section` attribute in config.json
    "description": "Define the pipeline",
    "required": [ "input", "outdir" ],               // <- transform to `required` attribute in config.json
    "properties": {
        "input": {
            "type": "string",
            "format": "file-path",
            "exists": true,
            ...
        },
        ...
    }
}

During the convertion:

• for all properties from one group from nf-core config, title value shall be used as section value in config.json
• for only properties specified in required enumeration attribute from nf-core config, "required":"true" shall be specified in config.json

Validation attributes

Possible validation attributes of the nf-core config: pattern, minimum, maximum, minLength, maxLength, oneOf.
Translation to Cloud Pipeline config.json attributes:

nf-core config	Cloud Pipeline config.json
"minimum": <MIN>	"validation": [{ "throw": "<CURRENT_PARAMETER> < <MIN>", "message": "<CURRENT_PARAMETER> shall be greater or equal than <MIN>" }]
"maximum": <MAX>	"validation": [{ "throw": "<CURRENT_PARAMETER> > <MAX>", "message": "<CURRENT_PARAMETER> shall be less or equal than <MAX>" }]
if minimum and maximum are both set for one property	"validation": [{ "throw": "<CURRENT_PARAMETER> > <MAX> || <CURRENT_PARAMETER> < <MIN>", "message": "<CURRENT_PARAMETER> shall be no less than <MIN> and no greater than <MAX>" }]
"pattern": "<PATTERN>"	"validation": [{ "throw": "/DOESN'T MATCH <PATTERN>/.test(<CURRENT_PARAMETER>)", "message": "Incorrect parameter value" }] * Note: need to clarify how to perform DOESN'T MATCH operation with regex
"minLength": <MINL>	Not fully supported yet. Shall be implemented
"maxLength": <MAXL>	Not fully supported yet. Shall be implemented

• If errorMessage is set for a property, its value shall be used as message value in the validation container.

`oneOf` attribute

oneOf attribute - container with conditions. Only one of the conditions must be met.
Examples of oneOf attribute from nf-core config:

1. Several conditions with the same property type, e.g.:

"oneOf": [
    {
        "type": "integer",
        "minimum": 250
    },
    {
        "type": "integer",
        "minimum": 0,
        "maximum": 0
    }
]

This attribute shall be transformed to validation attribute with a single nested container {"throw", "message"}:

• inside validation container in the throw attribute:
  • each condition from oneOf shall be placed in round brackets
  • all conditions from oneOf shall be linked by logical OR (||). Note: currently, it is not supported for regex validation. It shall be implemented
• inside validation container in the message attribute:
  • messages shall be linked by . OR extracted from errorMessage if it exists. Note errorMessage can be set for the whole property, not inside oneOf container
• each container from oneOf attribute shall be transformed by the rules described in the table above
• parameter type shall be selected according to the rules described in the type mapping section.

So, specified example will be transformed in config.json to:

"type": "string",
"validation": [
{
    "throw": "(<CURRENT_PARAMETER> < 250) || (<CURRENT_PARAMETER> < 0 || <CURRENT_PARAMETER> > 0)",
    "message": "<CURRENT_PARAMETER> shall be greater or equal than 250. <CURRENT_PARAMETER> shall be no less than 0 and no greater than 0"
}

2. Several conditions with the different property type, e.g.:

"oneOf": [
    {
        "type": "string",
        "enum": ["auto"]
    },
    {
        "type": "number",
        "minimum": 0
    }
]

This attribute shall be transformed to validation attribute with a single nested container {"throw", "message"}:

• inside validation container in the throw attribute:
  • each condition from oneOf shall be placed in round brackets
  • all conditions from oneOf shall be linked by logical OR (||). Note: currently, it is not supported for regex validation. It shall be implemented
  • if oneOf container includes enum attribute, then in the throw it shall be translated to "<CURRENT_PARAMETER> != <enum_value_1> || <CURRENT_PARAMETER> != <enum_value_2> || ... "
• inside validation container in the message attribute:
  • messages shall be linked by . OR extracted from errorMessage if it exists. Note errorMessage can be set for the whole property, not inside oneOf container
  • if oneOf container includes enum attribute, then in the message it shall be translated to "<CURRENT_PARAMETER> may be <enum_value_1> or <enum_value_2> or ... "
• each container from oneOf attribute shall be transformed by the rules described above
• shall be selected a single parameter type by the following rules:
  • firstly, type of each container inside oneOf attribute shall be defined according to the rules described in the type mapping section
  • secondly, type shall be "merged" by the following conditions
    • if in any container inside oneOf, type is input/output then "merged" type shall be corresponding input/output
    • in other cases, "merged" type shall be string

So, specified example will be transformed in config.json to:

"type": "string",
"validation": [
{
    "throw": "(<CURRENT_PARAMETER> != 'auto') || (<CURRENT_PARAMETER> < 0)",
    "message": "<CURRENT_PARAMETER> shall be no less than 0. <CURRENT_PARAMETER> may be 'auto'"
}