Ensure that parent dir of host_outdir_tgt exists before copying.

[johanneskoester]

This fixes the following bug:

[job job-1] completed success
[step job-1] completed success
[workflow ] starting step job-0
[step job-0] start
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set()
Got workflow error
Traceback (most recent call last):
  File "/home/johannes/.local/opt/miniconda3/envs/snakemake/lib/python3.6/site-packages/cwltool/executors.py", line 162, in run_jobs
    job.run(runtime_context)
  File "/home/johannes/.local/opt/miniconda3/envs/snakemake/lib/python3.6/site-packages/cwltool/job.py", line 484, in run
    runtime = self.create_runtime(env, runtimeContext)
  File "/home/johannes/.local/opt/miniconda3/envs/snakemake/lib/python3.6/site-packages/cwltool/docker.py", line 299, in create_runtime
    self.add_volumes(self.generatemapper, runtime, secret_store=runtimeContext.secret_store)
  File "/home/johannes/.local/opt/miniconda3/envs/snakemake/lib/python3.6/site-packages/cwltool/docker.py", line 235, in add_volumes
    shutil.copy(vol.resolved, host_outdir_tgt)
  File "/home/johannes/.local/opt/miniconda3/envs/snakemake/lib/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/johannes/.local/opt/miniconda3/envs/snakemake/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzsy30x78/dir/test.inter'
Workflow error, try again with --debug for more information:
[Errno 2] No such file or directory: '/tmp/tmpzsy30x78/dir/test.inter'

Which occurs when running a workflow exported from Snakemake that looks like this:

{
    "cwlVersion": "v1.0",
    "$graph": [
        {
            "class": "CommandLineTool",
            "id": "#snakemake-job",
            "label": "Snakemake job executor",
            "hints": [
                {
                    "dockerPull": "quay.io/snakemake/snakemake:v5.2.4",
                    "class": "DockerRequirement"
                }
            ],
            "baseCommand": "snakemake",
            "requirements": {
                "ResourceRequirement": {
                    "coresMin": "$(inputs.cores)"
                }
            },
            "arguments": [
                "--force",
                "--keep-target-files",
                "--keep-remote",
                "--force-use-threads",
                "--wrapper-prefix",
                "https://bitbucket.org/snakemake/snakemake-wrappers/raw/",
                "--notemp",
                "--quiet",
                "--no-hooks",
                "--nolock",
                "--mode",
                "1"
            ],
            "inputs": {
                "snakefile": {
                    "type": "File",
                    "default": {
                        "class": "File",
                        "location": "Snakefile"
                    },
                    "inputBinding": {
                        "prefix": "--snakefile"
                    }
                },
                "sources": {
                    "type": "File[]",
                    "default": [
                        {
                            "class": "File",
                            "location": "Snakefile"
                        }
                    ]
                },
                "cores": {
                    "type": "int",
                    "default": 1,
                    "inputBinding": {
                        "prefix": "--cores"
                    }
                },
                "rules": {
                    "type": "string[]?",
                    "inputBinding": {
                        "prefix": "--allowed-rules"
                    }
                },
                "input_files": {
                    "type": "File[]",
                    "default": []
                },
                "target_files": {
                    "type": "string[]?",
                    "inputBinding": {
                        "position": 0
                    }
                }
            },
            "outputs": {
                "output_files": {
                    "type": {
                        "type": "array",
                        "items": "File"
                    },
                    "outputBinding": {
                        "glob": "$(inputs.target_files)"
                    }
                }
            }
        },
        {
            "class": "Workflow",
            "requirements": {
                "MultipleInputFeatureRequirement": {}
            },
            "steps": [
                {
                    "run": "#snakemake-job",
                    "requirements": {
                        "InitialWorkDirRequirement": {
                            "listing": [
                                {
                                    "entry": "$(inputs.input_files[0])",
                                    "entryname": "dir/test.inter",
                                    "writable": true
                                }
                            ]
                        }
                    },
                    "in": {
                        "cores": {
                            "default": 1
                        },
                        "target_files": {
                            "default": [
                                "dir/test.out"
                            ]
                        },
                        "rules": {
                            "default": [
                                "rule1"
                            ]
                        },
                        "input_files": {
                            "source": [
                                "#main/job-1/output_files"
                            ],
                            "linkMerge": "merge_flattened"
                        }
                    },
                    "out": [
                        "output_files"
                    ],
                    "id": "#main/job-0"
                },
                {
                    "run": "#snakemake-job",
                    "requirements": {
                        "InitialWorkDirRequirement": {
                            "listing": [
                                {
                                    "entry": "$(inputs.input_files[0])",
                                    "entryname": "test.in",
                                    "writable": true
                                }
                            ]
                        }
                    },
                    "in": {
                        "cores": {
                            "default": 1
                        },
                        "target_files": {
                            "default": [
                                "dir/test.inter"
                            ]
                        },
                        "rules": {
                            "default": [
                                "rule2"
                            ]
                        },
                        "input_files": {
                            "source": [
                                "#main/input/job-1"
                            ],
                            "linkMerge": "merge_flattened"
                        }
                    },
                    "out": [
                        "output_files"
                    ],
                    "id": "#main/job-1"
                }
            ],
            "inputs": [
                {
                    "type": {
                        "type": "array",
                        "items": "File"
                    },
                    "default": [
                        {
                            "class": "File",
                            "location": "test.in"
                        }
                    ],
                    "id": "#main/input/job-1"
                }
            ],
            "outputs": [
                {
                    "type": {
                        "type": "array",
                        "items": "File"
                    },
                    "outputSource": "#main/job-0/output_files",
                    "id": "#main/output/job-0"
                },
                {
                    "type": {
                        "type": "array",
                        "items": "File"
                    },
                    "outputSource": "#main/job-1/output_files",
                    "id": "#main/output/job-1"
                }
            ],
            "id": "#main"
        }
    ]
}

[cwl-bot]

Can one of the admins verify this patch?

[codecov]

Codecov Report

Merging #902 into master will increase coverage by 0.01%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master     #902      +/-   ##
==========================================
+ Coverage   72.71%   72.72%   +0.01%     
==========================================
  Files          29       29              
  Lines        5808     5811       +3     
  Branches     1454     1455       +1     
==========================================
+ Hits         4223     4226       +3     
  Misses       1133     1133              
  Partials      452      452

Impacted Files	Coverage Δ
cwltool/docker.py	43.62% <33.33%> (-0.13%)	⬇️
cwltool/executors.py	75.69% <0%> (+1.1%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50d86c5...bf2411f. Read the comment docs.

[mr-c]

Jenkins, add to test list

[mr-c]

This is interesting, why doesn't this bug occur with https://github.com/EBI-Metagenomics/ebi-metagenomics-cwl/blob/master/tools/FragGeneScan1_20.cwl#L27 ?

Looking into it further, I'm not convinced that what I did in https://github.com/EBI-Metagenomics/ebi-metagenomics-cwl/blob/master/tools/FragGeneScan1_20.cwl#L27 is allowed by the standard.

[mr-c]

https://www.commonwl.org/v1.0/Workflow.html#Dirent

entryname The name of the file or subdirectory to create in the output directory.

Nothing here about a relative path name, like we are both doing.

[johanneskoester]

Ok, but it is very useful, right? I think it should be allowed.

[mr-c]

Yes, it is useful. Should be allowed and should the standard be changed are two different things, though!

[johanneskoester]

Well, as long as this is not merged or the standard is not changed, there cannot be Snakemake export to CWL *creatingsomepressure* ;-). So maybe there can be a minor update of the standard?

[johanneskoester]

Just one word ;-)...

[johanneskoester]

Do it :-).

[mr-c]

Must you have a subdirectory for inputs?

[tetron]

Would something like this work?

                        "InitialWorkDirRequirement": {
                            "listing": [
                                {
                                    "entry": "$(
{
  'class': 'Directory',
  'basename': 'dir',
  'listing': [{
    'class': 'File',
    'basename': 'test.inter',
    'location': inputs.input_files[0].location
  }]
})",
                                    "writable": true
                                }
                            ]
                        }

[johanneskoester]

Thanks @tetron! Looks like that might work for now. I will test it.

[johanneskoester]

Does that also work if dir has a subfolder containing the file?

[tetron]

You can nest a Directory object in the listing of a parent Directory object, if that's what you're asking. Or do you mean, what should you do if inputs.input_files[0] is a Directory and not a File?

[johanneskoester]

The former. I will try that.

[johanneskoester]

Ok, works, thank you! I suggest to either keep this PR open or add a note to the source that this bug needs to be fixed once the standard has been updated to allow paths in Dirent.