Fails to upload cleanup.sh after reboot

[Swampen]

Description

My build is failing at random builds and they are all caused by this error:

Error uploading script: Error copying input data into local temporary file. Check that TEMPDIR has enough space. Please see https://www.packer.io/docs/other/environment-variables#tmpdirfor more info. Error: connection reset

While inspecting the debug logs, it seems like a 1 minute delay is somewhat to short short. Is this a known problem?

2022/08/14 09:49:19 ui: [1;32m==> azure-arm: Pausing 1m0s before the next provisioner...[0m
2022/08/14 09:50:19 ui: [1;32m==> azure-arm: Provisioning with shell script: /home/vsts/work/1/s/runner-images/images/linux/scripts/installers/cleanup.sh[0m
2022/08/14 09:50:19 packer-provisioner-shell plugin: Opening /home/vsts/work/1/s/runner-images/images/linux/scripts/installers/cleanup.sh for reading
2022/08/14 09:50:19 packer-provisioner-shell plugin: [INFO] 833 bytes written for 'uploadData'
2022/08/14 09:50:19 [INFO] 833 bytes written for 'uploadData'
2022/08/14 09:50:19 packer-builder-azure-arm plugin: [DEBUG] Opening new ssh session
2022/08/14 10:06:05 packer-builder-azure-arm plugin: [ERROR] ssh session open error: 'ssh: unexpected packet in response to channel open: <nil>', attempting reconnect
2022/08/14 10:06:05 packer-builder-azure-arm plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/08/14 10:06:05 packer-builder-azure-arm plugin: [DEBUG] handshaking with SSH
2022/08/14 10:06:06 packer-builder-azure-arm plugin: [DEBUG] handshake complete!
2022/08/14 10:06:06 packer-builder-azure-arm plugin: [INFO] no local agent socket, will not connect agent
2022/08/14 10:06:06 packer-builder-azure-arm plugin: [DEBUG] Starting remote scp process:  scp -vt /tmp
2022/08/14 10:06:06 packer-builder-azure-arm plugin: [DEBUG] Started SCP session, beginning transfers...
2022/08/14 10:06:06 packer-builder-azure-arm plugin: [DEBUG] Copying input data into temporary file so we can read the length
2022/08/14 10:06:06 packer-provisioner-shell plugin: Retryable error: Error uploading script: Error copying input data into local temporary file. Check that TEMPDIR has enough space. Please see https://www.packer.io/docs/other/environment-variables#tmpdirfor more info. Error: connection reset
2022/08/14 10:06:06 [INFO] (telemetry) ending shell
2022/08/14 10:06:06 ui: [1;32m==> azure-arm: Provisioning step had errors: Running the cleanup provisioner, if present...[0m

Platforms affected

• Azure DevOps
• GitHub Actions

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

Private project in Azure DevOps. Building the current code on main

Is it regression?

No

Expected behavior

Uploading the cleanup script succeeds after reboot

Actual behavior

Uploading the cleanup script fails after reboot

Repro steps

1. Initiate packer with packer build commands
2. Wait until the cleanup step and see if the build fails

[Swampen]

No one else with this problem?

[strat-alex]

I do have that issue but not consistently.
It's very weird as here it fails every other time:

==> azure-arm.build_image: Provisioning with shell script: /agent/_work/1/s/images/linux/scripts/base/snap.sh
==> azure-arm.build_image: Provisioning with shell script: /agent/_work/1/s/images/linux/scripts/base/reboot.sh
    azure-arm.build_image: Reboot VM
==> azure-arm.build_image: Pausing 1m0s before the next provisioner...
==> azure-arm.build_image: Provisioning with shell script: /agent/_work/1/s/images/linux/scripts/installers/cleanup.sh
==> azure-arm.build_image: Error uploading script: Error copying input data into local temporary file. Check that TEMPDIR has enough space. Please see https://developer.hashicorp.com/packer/docs/configure#tmpdirfor more info. Error: connection reset
==> azure-arm.build_image: Step "StepProvision" failed
==> azure-arm.build_image: Provisioning step had errors: Running the cleanup provisioner, if present...
==> azure-arm.build_image: 
==> azure-arm.build_image: Deleting individual resources ...

Running privately on the current git version of the ado server.

I'm going to test with the 2m delay today, as advised here: #7478 .

[strat-alex]

The 2min wait does solve the issue consistently for me, with the 1 min delay, I had a 50% success rate.

Maybe this gives a little context: the VMSS containing the agent running the pipeline is located in West Europe, behind an Azure VWAN, the VWAN is connected to an AFW, but the VNET in question is not secured by an Azure Firewall yet (that's planned somewhere next month).

[nadavshaham]

@Swampen created a PR to fix #7478

[samus-aran]

We're locking this discussion because it has not had recent activity and/or other members have asked for more information to assist you but received no response. Thank you for helping us maintain a productive and tidy community for all our members.