WalReceiverDiskFilling

This alert fires when the WAL receiver has less than 15% free space on /var/opt/gitlab, where it keeps its WAL copy. If the disk fills, the receiver stops copying WAL.

The receiver keeps about 1 day of WAL: its cleanup runs every 8 hours and deletes files older than 1 day only when GCS has exactly the same file. So the disk fills when:

• the cleanup is not deleting (dry run, failing, or not running; see the cleanup alerts);
• the primary’s upload to GCS has stopped, so GCS doesn’t have the files: then the receiver is doing its job, and these files may be the only copy;
• files were kept after a split brain, because they differ from GCS;
• the cluster writes much more WAL than usual.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

• Patroni Service

• node_filesystem_avail_bytes / node_filesystem_size_bytes for /var/opt/gitlab on machines that publish walrecv_heartbeat_timestamp_seconds (that is, WAL receivers).

• Fires after 10 minutes below 15% free. Clears when space is freed or the disk is grown.
• Severity s3: no page. In gprd it goes to incident.io (every gprd alert does; it pages only alerts with a pager label, and this one has none), #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

On the receiver:

sudo systemctl list-timers walrecv-cleanup.timer --no-pager      # when it last ran and runs next
sudo journalctl -u walrecv-cleanup --since "-1d" --no-pager | grep -E "ERROR|deleted or would delete"
cat /var/opt/gitlab/walrecv-metrics/walrecv_cleanup.prom          # the numbers behind these alerts
df -h /var/opt/gitlab
sudo du -xsh /var/opt/gitlab/* | sort -h | tail   # what uses the disk: the WAL folders, or something else

Also check whether the primary’s upload to GCS works (the walgBackupDelayed alert, and sudo bash -c 'ls /var/opt/gitlab/walrecv/tl*/' | tail against the newest file in GCS).

• Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
• Check recent production change requests for failovers or Patroni changes on the cluster.

• The cleanup fails or doesn’t run: see WalReceiverCleanupNotSucceeding and WalReceiverCleanupKeepingFiles.
• The primary’s upload to GCS stopped: do not delete the receiver’s files; they may be the only copy. Fix the upload, or follow WAL receiver: hand over to a new primary and WAL receiver: restore with the receiver’s files. If needed, grow the receiver’s data disk (data_disk_size of the receiver’s module in config-mgmt).
• Files kept after a split brain: remove them only after the incident is closed and those writes are recovered or written off (WAL receiver: save the old primary’s files).

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

• Main issue: production#23059 (WAL archiving resilience, option C)
• Alerts plan: production#23274
• Runbooks and hand-over procedure: production#23275
• Where the numbers come from: gitlab-patroni!211
• Other WAL receiver alerts: WalReceiverNotConnected, WalReceiverSlotRetainingTooMuchWal, WalReceiverGapDetected, WalReceiverFencedButStillPrimary, WalReceiverCleanupKeepingFiles, WalReceiverCleanupNotSucceeding, WalReceiverMetricsMissing