Unable to Windows container's env:PATH variable (Or any system wide env vars)

[Togtja]

Environment

Running a Windows 2022 Core EC2 Instance with Containers enabled

> buildctl --version
buildctl github.com/moby/buildkit v0.15.1 979542e90f2cb38077c808e0867d8d2c16ed10b8
> docker buildx version
github.com/docker/buildx v0.16.2 99dea6dacacc3d604788953088560b9880550570
n> docker buildx inspect
Name:          buildkit-exp
Driver:        remote
Last Activity: 2024-08-08 17:50:13 +0000 UTC

Nodes:
Name:             buildkit-exp0
Endpoint:         npipe:////./pipe/buildkitd
Status:           running
BuildKit version: v0.15.1
Platforms:        windows/amd64
Labels:
 org.mobyproject.buildkit.worker.containerd.namespace: buildkit
 org.mobyproject.buildkit.worker.containerd.uuid:      608b0545-44cb-435e-9276-41a3d3f431cb
 org.mobyproject.buildkit.worker.executor:             containerd
 org.mobyproject.buildkit.worker.hostname:             EC2AMAZ-KNKRDFH
 org.mobyproject.buildkit.worker.network:              cni
 org.mobyproject.buildkit.worker.selinux.enabled:      false
 org.mobyproject.buildkit.worker.snapshotter:          windows
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
 Keep Bytes:    488.3MiB
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    18.63GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 18.63GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 18.63GiB
 

(I am uncertain if this issues lies with buildx or buildkit, but gut feeling says buildkit)

Setup

To test setting path behaviour I use a few small powershell script to test

check_path.ps1

Write-Host "Checking env:PATH: $env:PATH"
$regedit_path = (Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path
Write-Host "Checking Regedit: $regedit_path"

regedit.ps1

Write-Host "Original $env:PATH"
$env:PATH = (Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path
Write-Host "From Registry: $env:PATH" 

$set_reg_test = "set\reg\test"

Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -Value "$env:PATH;$set_reg_test"
Write-Host "From Registry After setting it:"
(Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path
Write-Host "After : $env:PATH"

set_env_var.ps1

Write-Host "Original $env:PATH"

$set_env_var_test = "set\env\var\test"

[Environment]::SetEnvironmentVariable('PATH',  $env:PATH + ";$set_env_var_test", [EnvironmentVariableTarget]::Machine); 

Write-Host "After : $env:PATH"

set_x.ps1

Write-Host "Original $env:PATH"

$set_x_test = "set\x\test"

setx /M PATH   "$env:PATH;$set_x_test"

Write-Host "After : $env:PATH"

Finally the docker file

FROM --platform=windows/amd64 mcr.microsoft.com/windows/servercore:ltsc2022

ARG SCRIPT
ARG SCRIPT_NAME

SHELL ["C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "-Command"]
COPY scripts .
RUN ./check_path.ps1
RUN ./set_x.ps1
RUN ./check_path.ps1
RUN ./set_env_var.ps1
RUN ./check_path.ps1
RUN ./regedit.ps1
RUN ./check_path.ps1

Behaviour

Buildx

When running this as ``docker buildx build . --progress=plain --no-cache`
The relevant output:

#7 [3/9] RUN ./check_path.ps1
#7 5.344 Checking env:PATH: c:\Windows\System32;c:\Windows
#7 5.573 Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\

#8 [4/9] RUN ./set_x.ps1
#8 6.215 Original c:\Windows\System32;c:\Windows
#8 6.270 
#8 6.270 SUCCESS: Specified value was saved.
#8 6.274 After : c:\Windows\System32;c:\Windows

#9 [5/9] RUN ./check_path.ps1
#9 6.251 Checking env:PATH: c:\Windows\System32;c:\Windows
#9 6.463 Checking Regedit: c:\Windows\System32;c:\Windows;set\x\test

#10 [6/9] RUN ./set_env_var.ps1
#10 6.584 Original c:\Windows\System32;c:\Windows
#10 6.636 After : c:\Windows\System32;c:\Windows

#11 [7/9] RUN ./check_path.ps1
#11 6.776 Checking env:PATH: c:\Windows\System32;c:\Windows
#11 6.987 Checking Regedit: c:\Windows\System32;c:\Windows;set\env\var\test

#12 [8/9] RUN ./regedit.ps1
#12 6.792 Original c:\Windows\System32;c:\Windows
#12 7.021 From Registry: c:\Windows\System32;c:\Windows;set\env\var\test
#12 7.056 From Registry After setting it:
#12 7.103 c:\Windows\System32;c:\Windows;set\env\var\test;set\reg\test
#12 7.104 After : c:\Windows\System32;c:\Windows;set\env\var\test

#13 [9/9] RUN ./check_path.ps1
#13 6.949 Checking env:PATH: c:\Windows\System32;c:\Windows
#13 7.147 Checking Regedit: c:\Windows\System32;c:\Windows;set\env\var\test;set\reg\test

Normal docker build

With the command docker build . --no-cache
Relevant output:

Step 6/12 : RUN ./check_path.ps1
Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps
Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\

Step 7/12 : RUN ./set_x.ps1
Original C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps

SUCCESS: Specified value was saved.
After : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps

Step 8/12 : RUN ./check_path.ps1
Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;
Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test

Step 9/12 : RUN ./set_env_var.ps1
Original C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;
After : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;

Step 10/12 : RUN ./check_path.ps1
Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;
Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test

Step 11/12 : RUN ./regedit.ps1
Original C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;
From Registry: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test
From Registry After setting it:
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;set\reg\test
After : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test

Step 12/12 : RUN ./check_path.ps1
Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;set\reg\test;
Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;set\reg\test

Successfully built 208f0b07cbca

From what I can tell there are 2 things of concernt.

1. During the normal docker build all methods of setting the path works. Where none of the methods in the buildx works to persistently set the path, thought you are able to set the Regedit key, but it is not read as path.
2. Following the last point, when trying to set the path in the buildx version it does set the regedit key, but it seems like the regedit key is not been read from. This is very clear in the first ./check_path.ps1 step where the regedit key are the same, but the PATH env is different for the buildx build.

(While I only really tested env:Path I suspect this goes for all system env vars)

I would expect the behavour to be the same, or at least a way to persistently set the environment variable so I can use it in further scripts.
Any help would be appreciated, and I am willing to test things if needed

[TBBle]

This is actually a known issue, although we didn't produce a separate bug report for it yet. (And that's on me, I was supposed to do that a long time ago, the unresolved discussion is buried in comments on #3158)

The problem is that BuildKit currently sets a default PATH in the image manifest environment block when one isn't set, which overrides what's set inside the image's registry. The workaround is to use the ENV command in your Dockerfile to replicate any PATH changes you make in the relevant file.

[Togtja]

Thanks you! I don't know if you want me to leave the issue open until the PR is merged, or close it?

As for a workaround, after your comment, I thought of a better? workaround, at least better for my usecase.
I am actually installing software in the scripts, and those will set the PATH, and that can be a pain to replicate, but since the PATH registry get set you can actually extract that, set that in to a temporary variable and then in the DockerFile set ENV PATH to that temporary variable.

So basically this

FROM --platform=windows/amd64 mcr.microsoft.com/windows/servercore:ltsc2022

ARG SCRIPT
ARG SCRIPT_NAME

SHELL ["C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "-Command"]

Copy scripts .
RUN dir; .\install_git.ps1

RUN $env:PATH = (Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path; setx /M TMP_PATH $env:PATH

RUN echo "PATH: $env:PATH and TEMP path $env:TMP_PATH"

ENV PATH "$TMP_PATH"

RUN echo "AFTER_PATH: $env:PATH"

Relevant output:

#7 [3/6] .install_git.ps1
#7 98.23 Git install Error Code output:  0
#7 98.24 
#7 98.24
#7 DONE 101.8s

#8 [4/6] RUN $env:PATH = (Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path; setx /M TMP_PATH $env:PATH
#8 55.49 
#8 55.49 SUCCESS: Specified value was saved.
#8 DONE 58.0s

#9 [5/6] RUN echo "PATH: $env:PATH and TEMP path $env:TMP_PATH"
#9 7.447 PATH:
#9 7.447 c:\Windows\System32;c:\Windows
#9 7.507 and
#9 7.507 TEMP
#9 7.507 path
#9 7.507 C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Git\cmd
#9 DONE 8.8s

#10 [6/6] RUN echo "AFTER_PATH: $env:PATH"
#10 6.704 
#10 6.704 AFTER_PATH:
#10 6.704 C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps
#10 DONE 7.9s

The .\install_git.ps1 for completness

#Install git
Invoke-WebRequest -Uri "https://github.com/git-for-windows/git/releases/download/v2.45.2.windows.1/Git-2.45.2-64-bit.exe" -OutFile "git.exe"
$process = Start-Process -FilePath ".\git.exe" -ArgumentList "/VERYSILENT /NORESTART /NOCANCEL /SP- /CLOSEAPPLICATIONS /RESTARTAPPLICATIONS
/COMPONENTS=icons,ext\reg\shellhere,assoc,assoc_sh" -Wait -PassThru

Write-Host "Git install Error Code output:  $($process.ExitCode)"
if ($process.ExitCode -ne 0) {
    Write-Host "Error: Git installation failed. Exiting script."
    Write-Host "Error code: $($process.ExitCode)"
    exit 1
}

Not how I am not the one setting the PATH directly, but C:\Program Files\Git\cmd still shows up in the regedit and thus gets extracted when setting the PATH enviorment

[Togtja]

After a little bit of further tinkering
If you just set the ENV PATH at the beginning of the Dockerfile`, it seems to fix it.
Same test as the issue report

FROM --platform=windows/amd64 mcr.microsoft.com/windows/servercore:ltsc2022

ARG SCRIPT
ARG SCRIPT_NAME

SHELL ["C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "-Command"]

RUN $env:PATH = (Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path; setx /M TMP_PATH $env:PATH
ENV PATH "$TMP_PATH"

COPY scripts .
RUN ./check_path.ps1
RUN ./set_x.ps1
RUN ./check_path.ps1
RUN ./set_env_var.ps1
RUN ./check_path.ps1
RUN ./regedit.ps1
RUN ./check_path.ps1

RUN echo "AFTER_PATH: $env:PATH"

Output

#10 [ 6/13] RUN ./check_path.ps1
#10 5.546 Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps
#10 5.784 Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\
#10 DONE 7.0s

#11 [ 7/13] RUN ./set_x.ps1
#11 7.388 Original C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps
#11 7.498 
#11 7.498 SUCCESS: Specified value was saved.
#11 7.501 After: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps
#11 DONE 8.8s

#12 [ 8/13] RUN ./check_path.ps1
#12 7.682 Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;
#12 8.183 Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test
#12 DONE 9.6s

#13 [ 9/13] RUN ./set_env_var.ps1
#13 7.505 Original C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;
#13 7.562 After : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;
#13 DONE 8.8s

#14 [10/13] RUN ./check_path.ps1
#14 8.107 Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;
#14 8.321 Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test
#14 DONE 9.6s

#15 [11/13] RUN ./regedit.ps1
#15 7.225 Original C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;
#15 7.438 From Registry: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test
#15 7.477 From Registry After setting it:
#15 7.522 C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;set\reg\test
#15 7.523 After : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test
#15 DONE 8.7s

#16 [12/13] RUN ./check_path.ps1
#16 7.922 t;Checking env:PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;set\reg\tes
#16 8.116 Checking Regedit: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;set\reg\test
#16 DONE 9.4s

#17 [13/13] RUN echo "AFTER_PATH: $env:PATH"
#17 7.989 AFTER_PATH:
#17 7.990 C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;set\x\test;;set\env\var\test;set\reg\test;
#17 DONE 9.2s

All the different ways seems to now actually set the environment variables, which makes it even a better workaround. No need to replicate any path changes. Just set it in the beginning to something, and then it seems to work as expected