WalReceiverMetricsMissing

This alert fires when the WAL receiver is not publishing its metrics: walrecv_heartbeat_timestamp_seconds is missing on a walrecv-* node, or it is more than 10 minutes old, or the receiver’s node_exporter can’t be scraped (up is 0). The receiver rewrites the heartbeat every 10 seconds.

While this fires, the other receiver alerts that use the receiver’s own numbers (WalReceiverDiskFilling, WalReceiverGapDetected, WalReceiverFencedButStillPrimary, WalReceiverCleanupKeepingFiles, WalReceiverCleanupNotSucceeding) may not fire, because they need the same metrics. The slot alerts still work, because they use the primary’s metrics.

Common causes:

• node_exporter’s text-file folder (collector.textfile.directory) isn’t set, so chef didn’t link the receiver’s metric files (chef logs a warning: node_exporter has no collector.textfile.directory);
• the links in node_exporter’s folder to /var/opt/gitlab/walrecv-metrics/ are missing or broken;
• the receiver’s script is stuck, so it stops rewriting the file;
• node_exporter itself is down.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

• Patroni Service

• walrecv_heartbeat_timestamp_seconds: written by the receiver every 10 seconds, in /var/opt/gitlab/walrecv-metrics/walrecv.prom.
• node_uname_info{fqdn=~"walrecv-.*"}: shows that node_exporter is running on a receiver, so a missing heartbeat can be told apart from a missing node.
• up{job="scrapeConfig/monitoring/prometheus-agent-node-exporter-gce", fqdn=~"walrecv-.*"}: 0 when node_exporter on the receiver can’t be scraped.

• Fires about 20 minutes after the last heartbeat (the query waits until it is 10 minutes old, then for: 10m adds 10 more), or after 10 minutes if the heartbeat is missing altogether, or after 10 minutes of up being 0. Clears as soon as the heartbeat is fresh again and node_exporter can be scraped.
• Severity s3: no page. In gprd it goes to incident.io (every gprd alert does; it pages only alerts with a pager label, and this one has none), #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

On the receiver:

grep heartbeat /var/opt/gitlab/walrecv-metrics/walrecv.prom; date +%s       # the heartbeat should be within 10 s of now
ls -l /opt/prometheus/node_exporter/metrics/ | grep walrecv                  # both links should point to /var/opt/gitlab/walrecv-metrics/
systemctl is-active walrecv
sudo sv status node_exporter                                                  # node_exporter runs under runit, not systemd
sudo journalctl -u walrecv --since "-15min" --no-pager | tail -5

• Check recent chef-repo MRs for the receiver’s role, and for node_exporter changes.
• Check recent production change requests for changes on the receiver.

• If the links are missing, run chef on the receiver: sudo chef-client. If chef warns that node_exporter has no text-file folder, fix the node_exporter settings in the receiver’s role.
• If the heartbeat in the file is old but the service is active, the script is stuck: sudo systemctl restart walrecv, then check that the heartbeat moves.
• If the walrecv service is stopped: sudo systemctl start walrecv. Chef also starts it again on its next run.
• If node_exporter is down (sv status shows down, and up for the receiver is 0): sudo sv restart node_exporter. It runs under runit: systemctl shows only an old, unused wrapper, which always says inactive.

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

• Main issue: production#23059 (WAL archiving resilience, option C)
• Alerts plan: production#23274
• Runbooks and hand-over procedure: production#23275
• Where the numbers come from: gitlab-patroni!211
• Other WAL receiver alerts: WalReceiverNotConnected, WalReceiverSlotRetainingTooMuchWal, WalReceiverDiskFilling, WalReceiverGapDetected, WalReceiverFencedButStillPrimary, WalReceiverCleanupKeepingFiles, WalReceiverCleanupNotSucceeding