Resize a PersistentVolumeClaim

Kubernetes supports online expansion of PVCs backed by XFS, ext3, or ext4. Editing the PVC’s requested storage triggers the storage backend to grow the underlying disk, and the file system is expanded in place with no pod restart if the CSI driver supports it. See Expanding Persistent Volumes Claims.

PVCs can only grow — the Kubernetes API rejects shrinks.

Example: resize a PVC used by a Deployment from 10 GiB to 20 GiB.

• Confirm the storage class supports expansion:

  kubectl get storageclass -o custom-columns=NAME:.metadata.name,EXPAND:.allowVolumeExpansion

• Prepare the size change in Helm / ArgoCD / whatever manages the PVC (do not merge yet).

• Have cluster access — see k8s-oncall-setup.md.

kubectl -n thanos describe Deployment/receive-gitlab-thanos-compactor
kubectl -n thanos get pvc receive-gitlab-thanos-compactor

Merge your Helm / ArgoCD MR. The controller patches the PVC’s spec.resources.requests.storage, the backend grows the disk, and the file system expands online.

Verify:

kubectl -n thanos get pvc -l app.kubernetes.io/component=compactor -o wide

StatefulSet controllers do not propagate storage changes from volumeClaimTemplates to existing PVCs — see Kubernetes enhancement #661 (closed, no KEP merged). To grow only the PVC data without touching the template, patch each PVC directly.

The GCE PD CSI driver used by GKE supports online volume expansion, so no pod bounce is required — the file system grows in place while the pod continues running.

1. Find the PVCs:

   kubectl -n mimir get pvc | grep compact

2. Patch each PVC’s spec.resources.requests.storage to the new value:

   kubectl -n mimir patch pvc <pvc-name> -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

3. Confirm the resize completed:

   kubectl -n mimir get pvc | grep compact

   Rarely, if the CSI driver cannot grow the volume online, the PVC gains a FileSystemResizePending condition and completes on the next pod restart:

   - message: Waiting for user to (re-)start a pod to finish file system resize of volume on node.
     status: "True"
     type: FileSystemResizePending

   Bounce the pod to unblock it — for a StatefulSet, scale to 0 and back:

   kubectl -n mimir scale statefulsets/mimir-compactor --replicas=0
   kubectl -n mimir scale statefulsets/mimir-compactor --replicas=1

Kubernetes forbids mutating a StatefulSet’s volumeClaimTemplates. Attempting to sync an ArgoCD change that modifies it is rejected as long as the StatefulSet exists. The safe path is orphan-delete: remove the StatefulSet controller but leave pods and PVCs running.

Open the ArgoCD / Helm MR with the new volumeClaimTemplates size. Do not merge yet.

kubectl delete statefulset <sts> -n <namespace> --cascade=orphan

Repeat per StatefulSet if the workload uses one StatefulSet per zone.

Pods and PVCs keep running.

NAMESPACE=<namespace>
SIZE=<new-size>            # e.g. 250Gi
SELECTOR=<label-selector>  # from the StatefulSet's spec.selector.matchLabels

for pvc in $(kubectl get pvc -n "$NAMESPACE" -l "$SELECTOR" -o jsonpath='{.items[*].metadata.name}'); do
  kubectl patch pvc "$pvc" -n "$NAMESPACE" -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"$SIZE\"}}}}"
done

To find the right selector:

kubectl get statefulset <sts> -n <namespace> -o jsonpath='{.spec.selector.matchLabels}'

ArgoCD recreates the StatefulSet with the updated volumeClaimTemplates; pods reschedule against the already-resized PVCs.

kubectl get statefulset <sts> -n <namespace>
kubectl get pods         -n <namespace> -l "$SELECTOR"
kubectl get pvc          -n <namespace> -l "$SELECTOR"

PVCs cannot be shrunk. If you have orphan-deleted the StatefulSet but not yet merged the config MR, recreate the StatefulSet manually at the original size to restore the controller. The PVC resize itself is irreversible.

• Kubernetes: Expanding Persistent Volumes Claims
• GKE Storage Classes
• k8s-operations.md — general workload operations.
• alerts/kube_persistent_volume_claim_disk_space.md — PVC saturation alert.