Issue#

After a rolling update, or after a node goes down, the new pod stays in ContainerCreating forever. kubectl describe pod shows:

Warning FailedAttachVolume 2m attachdetach-controller Multi-Attach error for volume "pvc-3f1c2a9e-8b7d-4e21-9c55-1a2b3c4d5e6f" Volume is already used by pod(s) web-6d8f7c9b5-x2k4p

or:

Multi-Attach error for volume "pvc-3f1c2a9e-8b7d-4e21-9c55-1a2b3c4d5e6f" Volume is already exclusively attached to one node and can't be attached to another

Cause#

The PVC uses the ReadWriteOnce access mode, so it can only be attached to one node at a time. This usually happens in one of two ways:

  1. RollingUpdate strategy: The Deployment starts the new pod before it stops the old one. If the new pod is scheduled on a different node, the volume is still attached to the old node.
  2. Dead or unreachable node: The old pod is stuck in Terminating or Unknown on a node that can't confirm the detach, so the VolumeAttachment is never released.

Solution#

  1. Find the pod and node that still hold the volume:
kubectl get pods -o wide | grep web kubectl get volumeattachment | grep pvc-3f1c2a9e
  1. If the old node is healthy: Change the Deployment to Recreate so the old pod is stopped before the new one starts:
kubectl patch deployment web \ -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'

You can also set it in the manifest:

spec: strategy: type: Recreate

Tip: If you need one replica with stable storage, a StatefulSet is usually a better fit than a Deployment.

  1. If the old node is dead: Mark it as out of service so Kubernetes force-detaches its volumes (GA since v1.28):
kubectl taint nodes worker-2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

If the old pod is still stuck, force-delete it:

kubectl delete pod web-6d8f7c9b5-x2k4p --grace-period=0 --force
  1. Use this only as a last resort, when the node is gone and the attachment is still there. Delete the stale VolumeAttachment:
kubectl delete volumeattachment csi-0a1b2c3d4e5f...

Warning: Do this only if you are sure the old node is powered off. Attaching an RWO disk to two nodes can corrupt data.

  1. After the node is repaired and back in the cluster, remove the taint:
kubectl taint nodes worker-2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-

Verify#

kubectl get volumeattachment | grep pvc-3f1c2a9e kubectl get pods -o wide | grep web kubectl describe pod <new-pod> | grep -A5 Events

The VolumeAttachment should now point to the new pod's node and show ATTACHED as true. The pod should be Running, and no new FailedAttachVolume events should appear.