Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NodeUnpublish stuck when the filesystem is shutdown #615

Open
tiagolobocastro opened this issue Jan 21, 2025 · 0 comments
Open

NodeUnpublish stuck when the filesystem is shutdown #615

tiagolobocastro opened this issue Jan 21, 2025 · 0 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug

Comments

@tiagolobocastro
Copy link
Contributor

What steps did you take and what happened:
When some goes awry there's a chance a filesystem may shutdown, example: #614
In this case, deleting the Pod is stuck because NodeUnpublish is stuck.

What did you expect to happen:
Expect to be able to delete the pod.

The output of the following commands will help us better understand what's going on:

  • kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin
I0121 15:24:24.353104       1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount","volume_id":"pvc-f5e204ba-3797-47f9-bfe7-236d078292f7"}
E0121 15:24:24.356452       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = unable to umount the volume pvc-f5e204ba-3797-47f9-bfe7-236d078292f7 err : Error checking if path exists: stat /var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount: input/output error
I0121 15:24:24.956720       1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount","volume_id":"pvc-f5e204ba-3797-47f9-bfe7-236d078292f7"}
E0121 15:24:24.961777       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = unable to umount the volume pvc-f5e204ba-3797-47f9-bfe7-236d078292f7 err : Error checking if path exists: stat /var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount: input/output error
I0121 15:24:25.964596       1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount","volume_id":"pvc-f5e204ba-3797-47f9-bfe7-236d078292f7"}
E0121 15:24:25.968437       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = unable to umount the volume pvc-f5e204ba-3797-47f9-bfe7-236d078292f7 err : Error checking if path exists: stat /var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount: input/output error

Environment:

  • LocalPV-ZFS version: 2.7.0-develop
  • Cloud provider or hardware configuration: qemu vm x86_64
  • OS (e.g. from /etc/os-release): NixOS 24.11 (Vicuna)
  • Kernel version: Linux nixos 6.6.69
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug
Projects
None yet
Development

No branches or pull requests

2 participants