Staging input files in a non-writable sub-directory fails on Linux #1042

bogdang989 · 2019-01-23T14:40:18Z

Expected Behavior

Completed run.

Actual Behavior

Executing on Mac this completes, but on Linux it fails due to trying to delete and not having write permissions for the staged file (discovered by Jenkins, confirmed on a local Ubuntu machine).

Setting the folder to be staged as writable: true gets past this error as it avoids calling os.remove(host_outdir_tgt)
``

Workflow Code

Workflow code can be found here:
common-workflow-language/common-workflow-language@09a8975

class: CommandLineTool
cwlVersion: v1.0
baseCommand:
  - ls
  - staged
inputs:
  - id: infiles
    type: File[]
outputs:
  - id: outfile
    type: File
    outputBinding:
      glob: staged/whale.txt
requirements:
  - class: InitialWorkDirRequirement
    listing:
      - entry: >-
          ${ return { 'class': 'Directory', 'listing': inputs.infiles,
          'basename': 'staged'} }
  - class: InlineJavascriptRequirement
hints:
  - class: DockerRequirement
    dockerPull: 'debian:stretch-slim'

Full Traceback

Exception while running job
Traceback (most recent call last):
  File "/home/bogdan/Documents/cwltool/cwltool/job.py", line 329, in _execute
    inplace_update=self.inplace_update)
  File "/home/bogdan/Documents/cwltool/cwltool/job.py", line 148, in relink_initialworkdir
    os.remove(host_outdir_tgt)
PermissionError: [Errno 13] Permission denied: '/tmp/uue6d20d/staged/whale.txt'
[job stage_array_in_dir.cwl] completed permanentFail
{}
Final process status is permanentFail

Your Environment

cwltool version: 1.0.20181217162649
Check using cwltool --version

The text was updated successfully, but these errors were encountered:

mr-c · 2019-01-25T20:08:47Z

Why doesn't it have write permissions?

bogdang989 · 2019-01-25T21:45:18Z

I am not sure why tbh. It is quite strange, running the same tool on Mac works fine but on Linux it fails (both on the Jenkins run and local Ubuntu).
In debug mode, when I stop at the line os.remove(host_outdir_tgt)- https://github.com/common-workflow-language/cwltool/blob/master/cwltool/job.py#L148 and evaluate os.access(host_outdir_tgt, os.W_OK), I get True on Mac and False on Ubuntu.
If I am not mistaken, this is docker user's permission for a file on a mounted folder.

alexiswl · 2022-01-12T22:07:37Z

I think I've figured out the issue here (although I am not sure of the best solution).

tl;dr it's basically docker being silly with nested mount paths. Why it's fine on mac and not linux I don't know.

cwltool version: 3.1.20211107152837

If I run the cwltool below through docker I get the following logs:

INFO /usr/local/bin/cwltool 3.1.20211107152837
INFO Resolved 'whale-tool.cwl' to 'file:///home/ec2-user/whale-tool.cwl'
DEBUG [job whale-tool.cwl] initializing from file:///home/ec2-user/whale-tool.cwl
DEBUG [job whale-tool.cwl] {
    "infiles": [
        {
            "class": "File",
            "location": "file:///home/ec2-user/whale.txt",
            "size": 11,
            "basename": "whale.txt",
            "nameroot": "whale",
            "nameext": ".txt"
        }
    ]
}
DEBUG [job whale-tool.cwl] path mappings is {
    "file:///home/ec2-user/whale.txt": [
        "/home/ec2-user/whale.txt",
        "/eVOEcE/staged/whale.txt",
        "File",
        false
    ]
}
DEBUG [job whale-tool.cwl] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "ls"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "staged"
    }
]
DEBUG [job whale-tool.cwl] initial work dir {
    "_:3d2fc630-2d3c-4b8d-b572-60d2834c4347": [
        "_:3d2fc630-2d3c-4b8d-b572-60d2834c4347",
        "/eVOEcE/staged",
        "Directory",
        true
    ],
    "file:///home/ec2-user/whale.txt": [
        "/home/ec2-user/whale.txt",
        "/eVOEcE/staged/whale.txt",
        "File",
        true
    ]
}
INFO [job whale-tool.cwl] /tmp/s003k18w$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/s003k18w,target=/eVOEcE \
    --mount=type=bind,source=/tmp/c_tvtgb9,target=/tmp \
    --mount=type=bind,source=/home/ec2-user/whale.txt,target=/eVOEcE/staged/whale.txt,readonly \
    --workdir=/eVOEcE \
    --read-only=true \
    --user=1000:1000 \
    --rm \
    --cidfile=/tmp/hb7_imwb/20220112220125-982650.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/eVOEcE \
    debian:stretch-slim \
    ls \
    staged
whale.txt
INFO [job whale-tool.cwl] Max memory used: 0MiB
INFO [job whale-tool.cwl] completed success
DEBUG [job whale-tool.cwl] outputs {
    "outfile": {
        "location": "/home/ec2-user/whale.txt",
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$c75fe8a7be45b1120f8da36d8c1ba7c3ac895185",
        "size": 11,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG [job whale-tool.cwl] Removing input staging directory /tmp/8e6z44ce
DEBUG [job whale-tool.cwl] Removing temporary directory /tmp/c_tvtgb9
DEBUG Removing intermediate output directory /tmp/s003k18w
INFO Final process status is success

Note that /tmp/442nvwe9 is mounted to /SMiCml inside the container but /home/ec2-user/whale.txt is mounted to //eVOEcE/staged/whale.txt as read-only.

After running the workflow we see that some stagnant empty files are left over.

$ find /tmp/s003k18w/

/tmp/s003k18w/
/tmp/s003k18w/staged
/tmp/s003k18w/staged/whale.txt

If we have a look at the whale.txt file there we see that it's empty and it's owned by root.

$ ls -lsrth /tmp/s003k18w/staged/whale.txt
0 -rwxr-xr-x 1 root root 0 Jan 12 22:01 /tmp/s003k18w/staged/whale.txt

This file cannot be deleted by the user

$ rm  /tmp/s003k18w/staged/whale.txt
rm: remove write-protected regular empty file ‘/tmp/s003k18w/staged/whale.txt’? y
rm: cannot remove ‘/tmp/s003k18w/staged/whale.txt’: Permission denied

Nor does it have the same ionode as the original file

$ stat whale.txt
  File: ‘whale.txt’
  Size: 11              Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d   Inode: 1262513     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ec2-user)   Gid: ( 1000/ec2-user)
Access: 2022-01-12 22:00:32.657353343 +0000
Modify: 2022-01-12 22:00:32.657353343 +0000
Change: 2022-01-12 22:00:32.657353343 +0000
 Birth: -

$ stat /tmp/s003k18w/staged/whale.txt
  File: ‘/tmp/s003k18w/staged/whale.txt’
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 10301h/66305d   Inode: 4943920     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-01-12 22:01:26.169507599 +0000
Modify: 2022-01-12 22:01:26.169507599 +0000
Change: 2022-01-12 22:01:26.169507599 +0000
 Birth: -

Reproduce with:

CWLTOOL:

class: CommandLineTool
cwlVersion: v1.0
baseCommand:
  - ls
  - staged
inputs:
  - id: infiles
    type: File[]
outputs:
  - id: outfile
    type: File
    outputBinding:
      glob: staged/whale.txt
requirements:
  - class: InitialWorkDirRequirement
    listing:
      - entry: >-
          ${ return { 'class': 'Directory', 'listing': inputs.infiles,
          'basename': 'staged'} }
  - class: InlineJavascriptRequirement
hints:
  - class: DockerRequirement
    dockerPull: 'debian:stretch-slim'

Input:

infiles:
  - class: File
    location: whale.txt

whale.txt

$ cat whale.txt
Free willy

bogdang989 linked a pull request Jan 24, 2019 that will close this issue

Avoid relink if not having permissions #1045

Open

mr-c added bug help wanted labels Apr 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Staging input files in a non-writable sub-directory fails on Linux #1042

Staging input files in a non-writable sub-directory fails on Linux #1042

bogdang989 commented Jan 23, 2019 •

edited

Loading

mr-c commented Jan 25, 2019

bogdang989 commented Jan 25, 2019

alexiswl commented Jan 12, 2022 •

edited

Loading

Staging input files in a non-writable sub-directory fails on Linux #1042

Staging input files in a non-writable sub-directory fails on Linux #1042

Comments

bogdang989 commented Jan 23, 2019 • edited Loading

Expected Behavior

Actual Behavior

Workflow Code

Full Traceback

Your Environment

mr-c commented Jan 25, 2019

bogdang989 commented Jan 25, 2019

alexiswl commented Jan 12, 2022 • edited Loading

bogdang989 commented Jan 23, 2019 •

edited

Loading

alexiswl commented Jan 12, 2022 •

edited

Loading