Skip to content
Runbooks

WalReceiverGapDetected

Overview

Section titled “Overview”

This alert fires when the WAL receiver has found a gap: WAL is missing between its own files and the point where the primary will send WAL from.

The receiver checks this each time it starts copying a new timeline (a new folder tl<number>). If the primary’s slot starts later than the last file the receiver has, for example because the slot was lost while the receiver was off too long, the WAL in between is missing. The receiver then logs GAP ALARM and writes a marker file GAP-<time> in /var/opt/gitlab/walrecv/.

Impact: the receiver’s copy has a hole. A restore that relied on the receiver’s files would stop at the gap without an error. The missing files are normally in GCS.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • walrecv_gap_markers: the number of GAP-* marker files on the receiver.

Alert Behavior

Section titled “Alert Behavior”
  • Fires after 2 minutes. Clears only when the GAP-* marker is removed, after you checked that GCS has the missing files.
  • Severity s3: no page. In gprd it goes to incident.io (every gprd alert does; it pages only alerts with a pager label, and this one has none), #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl status walrecv.service                 # is the receiver running?
sudo journalctl -u walrecv --since "-1h" --no-pager | tail -30   # what it did last
sudo cat /var/opt/gitlab/walrecv/node                  # the IP of the primary it follows
sudo ls /var/opt/gitlab/walrecv/fenced/                # fenced nodes (normally empty)
cat /var/opt/gitlab/walrecv-metrics/walrecv.prom       # the numbers behind these alerts

On the primary:

Terminal window
sudo gitlab-patronictl list
sudo gitlab-psql -c "SELECT slot_name, active, wal_status, pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS kept FROM pg_replication_slots WHERE slot_name LIKE 'walrecv%';"

The GAP ALARM line in the receiver’s log says which file the receiver has up to, and where the slot starts:

Terminal window
sudo journalctl -u walrecv --no-pager | grep "GAP ALARM" | tail -3
sudo bash -c 'ls /var/opt/gitlab/walrecv/GAP-*'

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • Check that GCS has every missing WAL file, and then remove the GAP-* marker. The receiver doesn’t need the files back: a restore reads from GCS any file the receiver doesn’t have. If GCS is missing a file too, that WAL exists nowhere else: raise it in the incident. Full procedure: WAL receiver: the slot is lost.
  • Find out why the slot was lost (the receiver was off too long, or WalReceiverSlotRetainingTooMuchWal was ignored).

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”