Siphon Service

• Alerts: https://alerts.gitlab.net/#/alerts?filter=%7Btype%3D%22siphon%22%2C%20tier%3D%22inf%22%7D
• Label: gitlab-com/gl-infra/production~“Service::Siphon”

Siphon is an in-house, high-throughput data replication tool. It captures row-level changes from PostgreSQL via logical replication (CDC, change data capture) and replicates them to other data stores. The main target is ClickHouse, the analytical database. Siphon also replicates data to a GCS bucket (Parquet files) for the data team.

Siphon uses NATS JetStream as the pub-sub layer for moving data between its components.

Siphon has two main components, communicating via NATS JetStream:

• Producer: connects to PostgreSQL, reads the WAL stream via a logical replication slot, buffers events in memory, and publishes them to a NATS stream.
• Consumer: subscribes to the NATS stream and writes events to the target store (ClickHouse, or Parquet files on GCS).

Initial snapshots (when a new table is added to Siphon) are handled separately: existing rows are extracted in a REPEATABLE READ transaction on an archive node (if available) and sent to a snapshot NATS stream, which is then merged into the main stream before CDC resumes.

There are a few smaller companion applications too, see Infrastructure setup.

Note: the Argo CD applications and clusters are named orbit-* because they live under the Orbit umbrella, but this is the Siphon deployment that moves data from PostgreSQL to the analytics ClickHouse (other name: datalake).

Primary contact: #g_analytics_platform_insights Slack channel (team handbook).

Individuals with the most context: @ahegyi, @arun.sori and @ankitbhatnagar

For immediate emergencies or if there is no response from the primary contacts, escalate to #database_operations.

Siphon is not user-facing. A Siphon outage delays analytics data, it doesn’t break GitLab.com.

There are two signals that matter, and every Siphon incident starts from one of them:

1. PostgreSQL replication slot lag (producer side). The slot is a tap that flows constantly, and it must be consumed all the time. An unconsumed logical replication slot makes PostgreSQL retain WAL, and unbounded WAL retention eventually fills the disk on a GitLab.com Patroni primary.
2. Data lag / freshness in ClickHouse (consumer side). Delayed data means outdated or incorrect reports in the product. This can be tolerated for a while. Data is buffered in NATS, so it’s fine if somebody from the Siphon team looks at it within a few hours. It doesn’t need a fix in the middle of the night at the moment.

Once you know which side the problem is on, find the matching failure mode below.

Things are less critical than when the Siphon rollout started:

• NATS is no longer a single point of failure for PostgreSQL. If NATS is out, the producer goes into emergency mode and keeps consuming the slot.
• The primary’s disk is protected by max_slot_wal_keep_size. GitLab.com Patroni clusters have it configured, so PostgreSQL invalidates a slot that retains too much WAL instead of running out of disk. No manual slot drop is needed on GitLab.com. An invalidated slot is still a lost slot for Siphon (a data gap that needs repair), so the goal is to recover the producer well before the limit.
• Backup slots are being rolled out by the database team, so that losing the main slot won’t create a data gap. Not in effect yet, see Replication slot gone or unusable.

On-call priority order:

1. Start monitoring the slot immediately and keep watching it: the PostgreSQL logical replication lag chart on the Siphon Producers dashboard (the series prefixed with PRIMARY), and disk utilisation on the Patroni overview dashboard. The underlying metric is pg_replication_slots_confirmed_flush_lsn_bytes.
2. Recover the producer (NATS health, pod restarts, see First 15 minutes). A producer that is running again advances the slot and releases WAL with no data loss.
3. Escalate early if lag keeps growing while you work on the producer: bring in #g_analytics_platform_insights and #database_operations. If primary disk utilisation is above 70% and trending upward, raise the severity.
4. Data freshness in ClickHouse comes last.

The configuration comes from several layers. It’s useful to know where each one lives before changing something.

A few things worth knowing:

• The table definitions are tested against the current PostgreSQL schema on every monolith CI run. On every GitLab release they’re packaged as an extra deliverable, tied to that GitLab version. This is the only dependency Siphon has on the monolith, and it matters on Dedicated and self-managed: the table definitions must come from the same GitLab version that’s running, because tables and columns drift between versions.
• In the apps repo, the table_mapping sections are generated from the monolith table YAML files (currently a semi-manual process when a new table is added). Generated content is marked with # siphon:managed:begin ... comments. Don’t edit those sections by hand unless there’s an urgent fix.
• Production runs several ClickHouse consumers, each with its own set of tables. The generator distributes the tables across the consumers. To find out which consumer handles a given table (for example labels), search for the table in services/siphon/env/orbit-prd/values.yaml.

• P0, as fast as possible: edit the config map (or the deployment) directly in Kubernetes. It takes effect immediately. Keep in mind that the next Argo CD sync overrides it, so it’s only a temporary fix. Follow up with an MR to the apps repo, otherwise the change gets reverted on the next rollout.
• P1, urgent: make the change in the apps repo (for example bump the memory in services/siphon/env/orbit-prd/values.yaml), merge it, and roll it out manually via Argo CD.

Auto-sync is not enabled for Siphon on GitLab.com, so merged changes in the apps repo don’t deploy on their own. To roll out:

1. Open the Argo CD UI and search for siphon.
2. Several applications show up. Look at the destination: orbit-stg or orbit-prd.
3. Hit Sync on the relevant application. All Siphon components are restarted.

It’s generally safe to sync at any time: Siphon stops and all components start up again, resuming from where they left off.

Staging and production have the same shape. Production simply runs more consumer instances to handle the data volume.

flowchart LR
  subgraph PG[PostgreSQL]
    main[(main)]
    ci[(ci)]
    sec[(sec)]
  end
  main --> pm[main producer]
  ci --> pc[ci producer]
  sec --> ps[sec producer]
  pm & pc & ps --> NATS[[NATS JetStream]]
  pm & pc & ps -. NATS down .-> FS[(GCS failover store)]
  NATS --> CC[ClickHouse consumers]
  NATS --> GC[GCS Parquet consumers]
  CC --> CH[(ClickHouse)]
  GC --> GCS[(GCS)]
  R[Reconcilers] --> CH
  R --> NATS
  RM[Retention manager] --> NATS
  CS[Consistency src/tgt] --> NATS

In general, the components are built so that a noisy component doesn’t affect the others.

The initial data snapshot on production reads from the DB archive node (patroni-dr-archive), which is not part of the Patroni cluster. On staging the snapshot runs from a replica node.

On GitLab.com there’s always traffic flowing, so no traffic means something is wrong.

There’s one more relevant signal outside of Siphon: the replication slot itself. If the slot is dropped or invalidated on PostgreSQL by something external, Siphon doesn’t really know about it. The producer will crash-loop and the data lag climbs. See Replication slot gone or unusable.

Dashboards live in the Siphon Grafana folder (playground/sandbox dashboards, they’re kept but not managed in this repo). Select the stg or prd environment.

Kibana logs:

• Staging: nonprod-log.gitlab.net
• Production: log.gprd.gitlab.net
• Useful filter: json.labels.k8s-pod/app : "postgres-main-producer" AND json.jsonPayload.level : "error"

Kubernetes logs only show the last few minutes of the current pod, which is good for quick debugging. To line up events over time (or after a pod restart), Kibana is better.

Siphon exposes more metrics than what’s on the dashboards. When looking for something specific, pointing an agent at the Siphon repo is a quick way to find out whether a metric exists.

Siphon Producers: select the environment and the application (DB): main, ci or sec.

• PostgreSQL logical replication lag. Always select the first series, prefixed with PRIMARY: that’s the slot on the primary. The other series are copies kept in sync on the replicas by the slot sync feature (sync_replication_slots), or unrelated slots (for example from a snapshot database created during an incident).
  • Occasional spikes are totally fine.
  • What shouldn’t happen is a constant increase. That means the producer is down, stuck, or not processing fast enough, and WAL retention on the primary is growing.
  • Intermittently high lag on the replica series can happen when a long vacuum or transaction runs on the primary. Nothing to do, it resolves itself.
• Logical replication operations. Inserts, updates and deletes per second, with a per-table breakdown. Higher lag usually correlates with higher traffic here, and it shows who the main contributor is.
• Snapshot operations total. Only renders when a snapshot is running (a new table was added, or a correction snapshot). Snapshots are usually short: a big increase, then back to zero. A snapshot is a heavy operation that can affect CDC processing, so if this chart shows something during an incident, the incident might be related.
• Emergency mode activated. A 0/1 flag. When NATS is gone, it jumps to 1 and the chart shows how many packages are written to the emergency object store.
• Drained messages after emergency mode. When NATS is back, the producer drains the object store automatically and this chart shows the progress. Less than 1-2h of buffering is restored pretty quickly. Note: WAL is not read while the drain runs.

ClickHouse Consumers: select the environment and the consumer application (production has 5).

• Number of events by stream. Inserts, updates and deletes per table. Large traffic may translate to higher data lag.
• Siphon data lag by stream. How long one package takes from the producer until it’s inserted into ClickHouse. Under normal circumstances it’s under 30 seconds, and less than a minute is considered healthy. The lag usually moves the same way for every table.
  • A flat line: OK, the consumer keeps up.
  • Line goes up: packages for that NATS subject (ClickHouse table) can’t be processed fast enough. Look for recent snapshots (snapshot data and CDC data end up on the same stream, so a snapshot delays processing), and look at the query timings and event counts for the table.
• Query timings. Insert and delete timings per table. Are inserts to ClickHouse getting slower?
• Duration and NATS timing metrics. How long processing a package or table takes, and how long NATS calls take. These can spike in specific cases, which helps debugging.

Consumers with a -refresh suffix process delayed refresh work (for example reconciler fixes). An accumulation there is expected after the reconciler finds many inconsistencies.

Example: a report comes in that merge request statistics don’t update after a merge request status change. Find the consumer that owns merge_requests, check that events are ingested for it at all, then check its data lag. If the lag keeps climbing, that explains the outdated reporting. Next questions: Is ClickHouse slow? Is there a background migration in the monolith writing an insane amount of data to the table?

In general: look at the metrics and look for spikes. Under normal circumstances things produce a nice straight line.

ClickHouse Reconcilers: select the environment (and the reconciler application, there are several on production).

The reconcilers scan tables on a schedule and fix outdated traversal_path values. Worth a look when a customer reports incorrect counts. Two run modes:

• Empty traversal_path: catches the chicken-egg problem (for example an issue ingested before its project). Runs every few minutes.
• Full: scans the whole table. Runs every few hours.

The top charts show how long a scan took, the bottom charts show how many inconsistencies were found. There’s no good or bad number, it depends on the traffic. These are global metrics, so they can’t be filtered by group.

A gauge caveat that can mislead: siphon_data_lag_ms is a gauge that is only written after a message is successfully processed, so a frozen value means “nothing processed recently,” not “no lag.”

Both orbit-stg and orbit-prd are available via the glsh wrapper in runbooks. Follow the k8s oncall setup first.

glsh kube use-cluster orbit-prd --no-proxy

Important deployments (in the siphon namespace, the pod app label matches the deployment name):

The producer always connects to the primary for logical replication.

Common commands:

# Overview, look for CrashLoopBackOff
kubectl get pods -n siphon -o wide

# Logs
kubectl logs -n siphon -l app=postgres-main-producer --tail=200
kubectl logs -n siphon -f <pod-name>

# Restart by scaling down and up
kubectl scale deployment postgres-main-producer --replicas=0 -n siphon
kubectl scale deployment postgres-main-producer --replicas=1 -n siphon

# Or restart in place
kubectl rollout restart deployment/postgres-main-producer -n siphon

Restarting is always safe, for producers and consumers. Remember that while a producer is down, WAL accumulates on the PostgreSQL side, so bring it back up.

The NATS CLI is available in the nats-box deployment (the default context is already set up with the client certificates):

kubectl exec -it -n nats deploy/nats-box -- sh
nats stream ls

Tip: letting an agent (e.g. Claude Code) run read-only kubectl commands helps a lot with debugging: “my producer is down, can you check the logs?” saves reading through the logs manually. Tell it explicitly not to change anything.

This section aims to give enough context to triage a page without knowing the codebase.

PostgreSQL ──WAL──> Producer ──> NATS JetStream ──> Consumer ──> ClickHouse
                       │                              │
                    (buffers in memory;          (distributed lock:
                     LSN acked to PG only         one active consumer
                     after the batch is           per stream and
                     flushed to NATS)             application id)

Three NATS streams per producer, and knowing which is which explains most confusing dashboards:

When a snapshot finishes, the producer pauses replication, merges Snapshot + Temporary into Main in LSN order, then resumes. In rare cases a stuck merge can stall CDC entirely for that producer, and it looks like “the producer stopped” rather than “a snapshot is slow.” While a merge runs, the producer pod reports NotReady (/health/ready returns 503 merge in progress; replication is paused).

Delivery is at-least-once. ClickHouse tables use ReplacingMergeTree, so duplicated delivery is fine: re-delivered events are deduplicated by primary key (versioned on _siphon_replicated_at).

1. Which side? Replication slot lag (producer) or data lag (consumer)? See How urgent is it?.

2. Which producer? The slot name encodes it: prd_main_siphon_slot_1 is main on production. Producers are per database (main, ci, sec).

3. Is NATS healthy? If NATS is down, everything downstream is down and the producer is likely in emergency mode. See the NATS runbook.

4. Is the producer running and progressing?

   glsh kube use-cluster orbit-prd --no-proxy
   kubectl get pods -n siphon -o wide
   kubectl logs -n siphon -l app=postgres-<db>-producer --tail=200

5. Is a snapshot running? Check the “Snapshot operations total” chart, or the admin API on the producer pod. This answers “is it paused or snapshotting?” in one call:

   kubectl port-forward -n siphon deploy/postgres-<db>-producer 8082:8082
   curl -s localhost:8082/v1/status | jq
   # => {"app_id": "...", "snapshot_paused": false, "snapshot_running": false}

   In emergency mode this endpoint returns 503 (no producer is currently registered to accept snapshots), which is itself a useful signal.

6. Bouncing the producer is safe and often sufficient. Siphon always resumes from the PostgreSQL replication slot (confirmed_flush_lsn), never from a cached position, so a restart can’t skip WAL.

   kubectl rollout restart deployment/postgres-<db>-producer -n siphon

Producers and consumers behave differently on persistent failure, which changes what a crash-looping pod means:

• Producers first restart in-process (siphon producer failed, restarting...) with exponential backoff. After roughly 15 minutes of consecutive failures the process exits and Kubernetes restarts the pod. The budget resets after any run lasting more than 3 minutes, so a CrashLoopBackOff producer means a persistent error that fails fast (config, credentials, grants, unusable failover storage, a missing slot).
• Consumers retry indefinitely with a 3-second backoff and log consumer failed - restarting. Only startup errors (bad config, handler build failure) exit non-zero. Once running, a permanently broken consumer looks like a running pod with flat throughput, not a crash loop.

On the GitLab.com deployments (Helm chart v2), each Siphon pod serves two ports:

• 8080 (http): /metrics and the /health* probe endpoints.
• 8082 (api): the /v1 admin routes. They’re only meaningful on producers.

kubectl port-forward -n siphon deploy/postgres-<db>-producer 8080:8080 8082:8082

When several producers share a pod, an app_id query parameter selects one; omitting it with multiple producers registered returns 409 listing the valid ids. While a producer is in emergency mode it doesn’t register for the /v1 routes, so they return 503.

There’s no endpoint to abort a running snapshot or merge, and no way to force a failover drain. The only control over in-flight work is a process restart (and, for a queued snapshot request, removing it from NATS).

Each failure mode lists the detection signal, the risk, and the steps. Producer-side failures can affect the PostgreSQL primary and are the urgent ones. Consumer-side failures only cause data lag.

Detection: SiphonLogicalReplicationSlotLagHigh or SiphonServiceSiphonProducersTrafficCessation. WAL retention alerts on PostgreSQL might fire too. The slot name encodes the affected DB and environment, e.g. stg_main_siphon_slot_1, prd_ci_siphon_slot_1.

Risk: An inactive replication slot causes PostgreSQL to retain WAL. On GitLab.com, max_slot_wal_keep_size caps this: once the slot retains more than the limit, PostgreSQL invalidates it instead of letting WAL fill the disk. That protects the primary, but an invalidated slot turns this into the Replication slot gone or unusable failure mode, a data gap in every table of that database. So the producer needs to recover well before the limit.

Source: the gprd-base-db-patroni-*-v17 / gstg-base-db-patroni-*-v17 roles in chef-repo.

Steps:

Work through these in order. Keep watching slot lag and primary disk utilisation the whole time.

1. Determine which DB is affected: the slot name (or the application name on the producer dashboard) contains the DB name: main, ci, or sec.

2. Start monitoring slot lag and primary disk now. Keep the Siphon Producers and the Patroni overview dashboards open. Note the growth rate: compared to the max_slot_wal_keep_size limit, it tells you how much time is left.

3. Check if NATS is up and running. If NATS is down and emergency mode doesn’t kick in, Siphon is down. See the NATS runbook.

4. Check whether a snapshot or merge is running. See Snapshot or stream merge stuck.

5. Check the logs for the cause. Common ones: crash loop due to config or connectivity, OOM, a missing slot.

6. Try to recover the producer. Restart it. This is safe: the producer always resumes from the slot, so no data is lost.

   kubectl get pods -n siphon -o wide
   kubectl logs -n siphon -l app=postgres-<db>-producer --tail=200
   kubectl rollout restart deployment/postgres-<db>-producer -n siphon

   A recovering producer shows siphon_operations_total rising and slot lag falling.

7. If the producer is running but can’t keep up, look at the logical replication operations chart: is there unusually high write traffic (e.g. a background migration)? Bring in the Siphon team.

8. Escalate if lag keeps growing. Post in #g_analytics_platform_insights and #database_operations. If primary disk utilisation is above 70% and trending upward, raise the incident severity.

Detection: SiphonEmergencyModeActive (fires after siphon_failover_mode_active{app_id} == 1 for 1h). Log: entering failover mode: NATS unreachable, buffering CDC to durable storage. siphon_failover_writes_total climbing. Note: emergency mode is called failover mode in the code, config and logs.

Description: Emergency mode removes NATS as a single point of failure. Data normally flows through NATS. If NATS is out, the producer can’t send data, and without a fallback WAL would start accumulating on the PostgreSQL side. With emergency mode, Siphon detects that NATS is down and buffers only the CDC data to object storage (GCS) instead. The WAL is still consumed and the slot keeps advancing, so the PostgreSQL side is safe.

What happens, step by step (as seen in the staging demo):

1. NATS goes down. The producer can’t publish, retries a few times (intermittent errors can happen), then gives up and restarts itself in-process.
2. On startup, a probe checks that the failover storage is usable and whether NATS is available. NATS is unreachable, so the producer enters emergency mode and starts writing CDC data to GCS.
3. A watcher polls NATS in the background (recovery_poll_seconds). When NATS is back, it logs NATS recovered; exiting failover mode and restarting siphon core and restarts the process.
4. The next startup sees NATS is up and there’s data in the emergency store, so it drains it into NATS (subject drain complete per subject when done), and normal replication continues.

Things to keep in mind:

• The object storage buffer is not a pub-sub system, it’s just a buffer. Consumers (ClickHouse, GCS Parquet) don’t receive any new data while emergency mode is active. Downstream applications see outdated data until the buffer fully drains.
• WAL is not read during the drain, so the slot doesn’t advance while it runs. Less than 1-2h of buffering drains pretty quickly.
• There’s no operator toggle. Emergency mode can’t be switched on while NATS is healthy, and can’t be switched off while NATS is unhealthy (short of removing queueing.failover_storage from the config).
• Snapshotting, partition monitoring and the /v1 admin API are disabled while in emergency mode. New partitions are picked up after returning to normal mode.
• A short NATS outage (for example a NATS upgrade) triggering emergency mode for a few minutes is totally legit. It’s only an incident when it lasts an hour or longer.
• Emergency mode has been tested for about an hour, not yet for several hours. It should work, but recovering NATS quickly is still the goal.

Risk: Moderate and mostly deferred. Data is safe but not in NATS, so consumers see no new data and the data lag grows. A drain that keeps failing (drain failover store: ... in the logs) counts towards the producer’s retry budget and eventually crash-loops the pod, at which point the slot stops advancing. That escalates to the replication lag failure mode.

Steps:

1. Fix NATS first. This probably needs a quick fix. See the NATS runbook.
2. Watch the automatic recovery and drain on the producer dashboard (“Drained messages after emergency mode”). A restart isn’t required; it only skips the recovery poll. The in-flight backlog is siphon_failover_writes_total - siphon_failover_drain_published_total, and it should fall to zero. siphon_failover_drain_errors_total rising means drain runs are aborting and will retry.
3. If the producer won’t start at all, check for failover storage not usable: in the logs. On every start the producer round-trips a test object through the failover store and refuses to continue if that fails. Check bucket access and workload identity.

Detection: The producer crash-loops with slot-related errors, SiphonServiceSiphonProducersTrafficCessation fires, and the data lag climbs. The slot is missing from pg_replication_slots, or it shows as invalidated (e.g. because of max_slot_wal_keep_size). Siphon itself can’t really see this, the cause is external: the slot was dropped or invalidated on PostgreSQL.

SELECT slot_name, active, wal_status, invalidation_reason
FROM pg_replication_slots
WHERE slot_type = 'logical' AND slot_name ILIKE '%siphon%';

wal_status = 'lost' (or a non-null invalidation_reason) means the slot is unusable.

Description: The logical replication slot is the consistent point where Siphon takes data from PostgreSQL. If the slot is gone or unusable, replication can’t resume from that point anymore. No data is lost in PostgreSQL, but Siphon can’t consistently continue the operation as before: there will be a data gap in all tables of that database. If the slot was gone for a few seconds, it might be a few thousand rows; if it’s minutes, it can be gigabytes of data.

Risk: Serious for data correctness, not for PostgreSQL. No panic, there are ways to recover.

Steps:

1. Escalate to #g_analytics_platform_insights. This needs the Siphon team.
2. Once Siphon is running again (with a recreated slot), consistency checks with corrections enabled repair the rows that were skipped. Run them for the tables of the affected database. It takes a while, it’s not immediate.
3. A full re-snapshot of the affected database’s tables (1-2 weeks on production) is only the worst-case fallback.

Backup slots (not in effect yet, rollout in progress): the database team maintains a backup slot: a logical replication slot that just sits there without a consumer. A cron script drops and recreates it periodically (about every hour), so on the lag chart it goes up and then drops to zero. Once the rollout is done, Siphon will be configured (replication.slot_priority_list) so that when the main slot is lost, the producer crashes, restarts, and picks up the backup slot. In that case no data is lost. Instead, messages are re-delivered because the backup slot is behind the active slot: in the worst case about one hour of extra traffic to replay. That’s fine, ClickHouse is set up for at-least-once delivery. Until then, losing the slot means a data gap as described above. The runbook steps will be added once it’s in place.

Detection: Producer pod restarts with OOMKilled, crash loop.

Description: Under normal circumstances, standard CDC traffic doesn’t OOM. There’s also a protection against very large column values: Siphon won’t read more than max_column_size_in_bytes from a given column (10 MiB on production); larger values are replicated as an empty string.

What can happen is a snapshot request, for example snapshotting a very wide table (like issues with large descriptions). Siphon reads too much data from the database at once and runs out of memory, then it’s a crash loop.

Steps: these three levers usually work:

1. Bump the memory. This is the magic fix in most cases, until it succeeds. Memory limits and GOMEMLIMIT are set per deployment profile in the apps repo (services/siphon/env/orbit-prd/values.yaml, under deploymentProfiles).
2. Reduce the batch sizes.
3. Reduce the memory buffer sizes (replication.memory_buffer_size_in_bytes, replication.snapshot_memory_buffer_size_bytes, or the per-table table_mapping[].snapshot_memory_buffer_size_bytes).

Also check GOMEMLIMIT, which affects how much memory the Go binary can consume before the GC gets aggressive. Generally it should be around 80% of the memory limit. The Helm chart applies it on all deployments, but when bumping the memory limit, adjust GOMEMLIMIT too. A GOMEMLIMIT that’s too low relative to the live heap makes the GC burn CPU and slows everything down.

For a P0, edit the deployment directly (see Changing configuration during an incident), but don’t forget the apps repo MR: the next rollout reverts live edits.

In the very unlikely case that more memory doesn’t help, cancel the snapshot request and retry it later with adjusted settings. See Cancelling a snapshot request.

Why it loops: snapshot requests sit in a NATS stream and the producer holds the in-flight one while it snapshots. When the pod OOMs, the message is redelivered after the ack wait (MaxDeliver: -1), so the producer picks up the same request again and crashes again. The request queue is strictly ordered, so a stuck request also blocks every snapshot queued behind it.

Where the requests live (example for siphon_main_1 on orbit-prd):

For ci and sec, replace siphon_main_1 with siphon_ci_1 or siphon_sec_1.

Steps (the nats commands run in the nats-box pod, see Kubernetes access):

1. Find the stuck message. The producer logs received snapshot event for table with LSN with the schema and table fields, which tells which table it is. The in-flight message’s sequence number is delivered.stream_seq in the consumer info (with num_ack_pending >= 1):

   nats consumer info prd_siphon_snapshot_requests prd_siphon_event_stream-siphon_main_1-snap-consumer

   The payloads are zstd-compressed JSON, so grepping for the table name doesn’t work. Requests from the consistency corrector carry a readable Nats-Msg-Id header (<run_id>.<table>.<target>.correction.<hash>), visible with nats stream get prd_siphon_snapshot_requests <seq> --json (.hdrs is base64). Manual POST /v1/snapshot requests have no Nats-Msg-Id.

2. Stop the producer, delete the message by sequence, start the producer:

   kubectl scale deployment postgres-main-producer --replicas=0 -n siphon
   nats stream rmm prd_siphon_snapshot_requests <seq> -f   # in nats-box
   kubectl scale deployment postgres-main-producer --replicas=1 -n siphon

   • The producer must be stopped first. While it runs, it holds the message and keeps extending it. POST /v1/snapshot/pause doesn’t help here: the pause is in-memory only and is lost on the next OOM restart.
   • Delete by sequence (rmm) instead of acking with nats consumer next ... --ack. consumer next acks whatever the consumer hands out next. If the ack wait hasn’t expired yet, that’s the next queued request (a different table), and the wrong request gets silently dropped.

3. No manual cleanup needed. The next snapshot for that table purges the partial snapshot stream data and re-merges any leftover temp stream data. The snapshot status for the table stays in_progress until a new run overwrites it. AFAIK that’s only cosmetic.

4. Re-trigger later with adjusted settings, via the admin API on the producer:

   kubectl port-forward -n siphon deploy/postgres-main-producer 8082:8082
   curl -sX POST localhost:8082/v1/snapshot -H 'Content-Type: application/json' \
     -d '{"app_id":"siphon_main_1","tables":[{"schema":"public","table":"issues"}]}' | jq

   To make the request smaller, split the table into key ranges with filters:

   {"app_id":"siphon_main_1","tables":[{"schema":"public","table":"merge_requests","filters":[{"type":"key_ranges","columns":["id"],"start":["1000000"],"end":["2000000"]}]}]}

   The producer-side levers are config changes (they need a restart): replication.snapshot_memory_buffer_size_bytes (or per table table_mapping[].snapshot_memory_buffer_size_bytes) and replication.initial_data_snapshot_threads_per_table.

Detection: siphon_snapshot_operations_total flat while a snapshot is in progress, or CDC throughput (siphon_operations_total) has stopped entirely with Merging table, pausing replication in the logs and no subsequent All pending merges finished, resuming replication.

Risk: High. A merge pauses CDC replication for the whole producer, so the replication slot stops advancing and WAL accumulates on the primary. A stuck merge escalates into the replication lag failure mode. Merges have a hard 30-minute deadline after which they fail and the producer core restarts in-process (not a pod restart). Every in-process restart re-runs the snapshot/merge cycle, so a merge that keeps timing out shows up as a repeating log pattern.

Steps:

1. Confirm what the producer thinks it’s doing (with the port-forward to 8082 from First 15 minutes):

   curl -s localhost:8082/v1/status | jq
   curl -s "localhost:8082/v1/snapshot-status/public/<table>" | jq

2. Look for the phase in the logs. Useful strings, in order of the lifecycle:

   kubectl logs -n siphon -l app=postgres-<db>-producer --tail=500 \
     | grep -E "snapshot slice progress|snapshot complete|Merging table|merge in progress|merge complete|snapshot failed"

   merge in progress with a rising messages_merged means it’s working, let it finish. A repeated snapshot failed means the snapshot is being retried in a loop.

3. Common benign cause on replica snapshots: waiting for the snapshot database to catch up means the snapshot source replica is behind the publication LSN. This is transient and resolves on its own.

4. If the merge is genuinely wedged, restart the producer. Recovery is idempotent: the merge consumer is always recreated from the start of the stream, orphaned merge consumers are cleaned up on every (re)start, and re-merging is safe because ClickHouse deduplicates by primary key.

   kubectl rollout restart deployment/postgres-<db>-producer -n siphon

   There’s no way to abort a merge without restarting the process.

Detection: Repeated new table or partition found, restarting to pick it up / restarting Siphon producer due to partition change in the producer logs. These are in-process restarts, so the pod restart count doesn’t move.

Description: The partition monitor restarts the producer core whenever it detects a new table or partition, so the new relation is picked up with a correct table mapping. A few restarts around partition rollover are expected, this is by design, not a fault. Adding the table without a restart would make PostgreSQL stream a relation the producer can’t route, silently dropping events while the LSN advances past them.

Risk: Low if it converges within a few cycles. If it does not converge, the producer isn’t replicating and the slot isn’t advancing, escalating to the replication lag failure mode.

Steps:

1. Check whether it’s converging. If restarts stop after a few cycles, no action.

2. If it loops indefinitely, the publication ADD TABLE is failing. Look for:

   kubectl logs -n siphon -l app=postgres-<db>-producer --tail=300 \
     | grep -i "failed to add table to publication"

   ... due to lock timeout, retrying means ALTER PUBLICATION couldn’t get its lock, typically because of long-running transactions on the table. Otherwise it’s usually a grants problem on the primary: the Siphon role needs ownership/permission to alter the publication. Escalate to #database_operations with the table name.

3. Note the interaction with pause: partition monitoring is skipped entirely while snapshots are paused. A forgotten POST /v1/snapshot/pause silently disables partition handling, check siphon_snapshot_paused.

Detection: siphon_snapshot_paused{app_id} == 1 for an extended period.

Risk: Snapshots don’t run and partition monitoring is disabled, so new partitions aren’t picked up. CDC for already-mapped tables continues normally, which is what makes this easy to miss.

Nothing auto-resumes a paused producer. The pause is in-process only, so any restart clears it. This means the gauge can also be cleared accidentally by an unrelated restart.

Steps:

kubectl port-forward -n siphon deploy/postgres-<db>-producer 8082:8082
curl -s localhost:8082/v1/status | jq          # confirm snapshot_paused
curl -sX POST localhost:8082/v1/snapshot/resume | jq

Pausing doesn’t stop an in-flight snapshot; it only prevents new ones from starting.

Detection: Producer restart loop with put object to store or writing GCS object in the logs. Consumers logging failed to fetch referenced event with data lag climbing.

Description: Events larger than the NATS payload limit and all snapshot events, regardless of size, are written to object storage, with only a reference published to NATS. siphon_oversized_events_total counts all of these, snapshot events included.

siphon_object_storage_put_failures_total and the put/get duration metrics are reliable for the gcs, s3 and file backends. With the nats object-store backend, producer Puts are not instrumented (consumer Gets are), so detect producer-side failures from logs.

Risk: On the producer, a failed Put aborts the flush and restarts the producer, so the slot stops advancing. This escalates to the replication lag failure mode. On the consumer, a missing object is a permanent poison message (NAKed and redelivered forever, see ClickHouse write failures).

Steps:

1. Check the bucket is reachable and the workload identity / credentials are valid. GCS and S3 buckets are managed externally, Siphon doesn’t create them. Only the nats object-store backend is auto-created.
2. For consumers that can’t fetch a referenced object, check whether it expired. GCS/S3 log object ... not found in bucket; the NATS backend logs getting object .... If the bucket TTL (NATS object-store TTL, or the GCS/S3 lifecycle policy) is shorter than the stream’s MaxAge, a consumer that falls behind will hit permanently missing objects. This is a configuration bug worth escalating, not a transient fault.
3. Multiple producers sharing a bucket must have distinct application_identifier values; objects are namespaced by it.

Consumer-side failures don’t threaten PostgreSQL. The only bad thing that can happen here is delayed data in ClickHouse, which can cause outdated or incorrect reporting in the product. Depending on how long it lasts, customer communication might be needed.

Detection: Consumer pods crash-looping, SiphonServiceSiphonConsumersTrafficCessation, data lag climbing for all tables.

Description: GitLab.com uses ClickHouse Cloud. It can happen that ClickHouse Cloud is down or inaccessible (network issue, doesn’t matter). The consumers crash-loop because they can’t access the ClickHouse database. Nothing goes wrong on the producer side: data still ends up in NATS, the number of undelivered messages simply increases.

Steps:

1. Check ClickHouse health, see the ClickHouse runbook and the ClickHouse Cloud console.
2. Once connectivity is restored (without data loss), the consumers recover on their own and burn off the accumulated data. This can take minutes to hours depending on the length of the outage. No data loss.

Detection: siphon_data_lag_ms{source="data"} climbing steadily on the ClickHouse Consumers dashboard while the producer is healthy. Data is processed, but pileups start to show. PostgreSQL replication lag is normal.

Description: Data can’t be ingested fast enough. Note that a stalled consumer is still a running pod, it retries forever rather than crash-looping.

Usual culprits:

• A recent code change, for example a change to the ClickHouse queries used for ingestion. The graphs are usually telling here.
• A ClickHouse service issue, or another service or user overloading ClickHouse.
• Unusually high write traffic on a table, for example a background migration in the monolith, or a heavily written table like project_authorizations. Cross-check with the logical replication operations chart on the producer dashboard.
• A recent snapshot. Snapshot data and CDC data end up on the same stream, so a large snapshot delays CDC processing for that table.

Risk: ClickHouse data goes stale, and downstream systems like Orbit operate on stale data. A consumer that stays stalled long enough (more than the ~10 days NATS retention) can be outrun by retention.

Steps:

1. Find the lagging tables and their consumer, check the event counts and query timings.
2. Check ClickHouse resource usage and slow queries. ClickHouse Cloud’s advanced dashboards show CPU utilisation, and the monitoring section is a good place to look around. Request ClickHouse Cloud access if you don’t have it yet.
3. A quick mitigation might be temporarily scaling up the ClickHouse Cloud service (a few clicks). Not 100% sure it helps every time: it might just be spiky traffic for a few hours that goes back to normal on its own.
4. If the consumer is stuck on one message, see ClickHouse write failures.

This isn’t something to fix immediately in the middle of the night. If the data is buffering fine in NATS, it’s OK for the Siphon team to look at it within a day.

Detection: siphon_clickhouse_consumer_batch_retry_total rising. siphon_clickhouse_consumer_adaptive_batch_size (a histogram of successful batch sizes) trending down. Logs: Batch failed, retrying with smaller size.

Description: It’s self-healing. On size-related ClickHouse errors (memory limits, timeouts, “too many parts”, query or AST too large, read limits) the consumer halves the batch size and retries, down to a single row, and caches the reduced size for 5 minutes. Steady low-level retries with data still flowing are normal backpressure, not an incident.

Escalation: Batch failed with non-retriable error (also logged when a size error persists at batch size 1). Connection drops and schema mismatches are deliberately not retried by halving, because halving can’t fix them. These become handler failures, and the message is NAKed and redelivered forever (MaxDeliver: -1, no dead-letter queue): a poison message that stalls the consumer. The signature is repeated group NAKed, error handling the package at the same stream_sequence, followed by consumer failed - restarting.

Steps:

1. Check ClickHouse health first, see the ClickHouse runbook.
2. Too many parts usually means ClickHouse merges are behind, not a Siphon fault.
3. A schema mismatch (a column present in PostgreSQL but missing in the ClickHouse target table) stalls permanently. Escalate to the Siphon team; it needs a target table migration.

Detection: Data quality reports: missing rows, missing deletes/updates, incorrect values for a time window.

Description: This can happen in many ways: ClickHouse crashed and lost inserts, an older backup was restored, somebody accidentally deleted or updated rows, or a data migration in the monolith deleted records from ClickHouse.

Risk: Major for data correctness, but not a reason for panic.

Steps: Run a consistency check for the affected tables. Checksumming finds the key ranges that differ, and with corrections enabled those ranges are re-read from PostgreSQL and re-inserted into ClickHouse. Depending on the number of tables, their size and the loss window, this takes from a few minutes to a few hours.

The consistency checker compares a PostgreSQL table with its ClickHouse table range by range. On GitLab.com, corrections are enabled: every mismatching range is submitted to the producer as a correction snapshot, which re-reads the rows and fixes the ClickHouse side.

A check is triggered by publishing a request to NATS (from the nats-box pod, see Kubernetes access). The subject is <stream>.<application_identifier>.request:

On staging, check the postgres-consistency-source deployment in services/siphon/env/orbit-stg/values.yaml for the stream and application identifiers.

Example for the notes table (missing or extra rows):

nats pub 'prd_siphon_consistency.prd_main_consistency.request' '{
  "verification_mode": "primary_keys",
  "source": {"table_schema": "public", "table_name": "notes"},
  "targets": [{"identifier": "main_ch", "table_name": "siphon_notes"}]
}'

• primary_keys catches missing and extra rows. To catch wrong values (a badly replicated UPDATE), use "verification_mode": "custom" with a columns list that includes the primary key, for example "columns": [{"name": "id"}, {"name": "note"}, {"name": "updated_at"}]. It’s considerably more expensive. numeric, double precision, bytea, time and array columns aren’t supported in custom mode.
• For a partitioned table, address the child partition, e.g. "source": {"table_schema": "gitlab_partitions_dynamic", "table_name": "ci_builds_100"} with "table_name": "siphon_p_ci_builds" as the target.

Follow the results:

nats sub 'prd_siphon_consistency.prd_main_consistency.verdict.>'    # final result per table
nats sub 'prd_siphon_consistency.prd_main_consistency.mismatch.>'   # ranges that differ
nats sub 'prd_siphon_consistency.prd_main_consistency.rejected'     # refused requests

Things to keep in mind:

• Corrections are serial. The producer applies one correction at a time, and each one is a snapshot plus a merge, which pauses replication briefly. A wide repair takes a while, and it shows up on the “Snapshot operations total” chart.
• Concurrent writes show up as mismatches. Rows written while the pass runs can be reported as mismatches. A rerun usually clears them.
• checked lower than expected in the verdict means the target restarted during the pass, not that the table is clean.
• More details: consistency check and correction docs.

Description: Something happens to the ClickHouse instance (for example somebody deletes it). A new ClickHouse Cloud instance has to be created and repopulated. Two options:

1. Full re-snapshot from all 3 databases (main, ci, sec). Terabytes of data need to be moved, so the runtime is 1-2 weeks. Ideally this is avoided, because everything depending on ClickHouse is stalled, unavailable or incorrect during that time.
2. Restore from a ClickHouse backup and re-apply the NATS traffic. NATS keeps data for about 10 days, so the consumers can be pointed to an older stream sequence number to replay the traffic and eventually become consistent. Then consistency checks can fix the rest.

The second option is the more likely scenario, but it hasn’t been tested. There’s no runbook procedure for this option. Escalate to the Siphon team.

Detection: Consumers crash-looping, no new data arriving to ClickHouse.

Description: Same story as above: nothing is ingested and the data lag grows, which may affect downstream services. On the producer side, emergency mode keeps PostgreSQL safe. Fix NATS, see the NATS runbook. Consumers recover on their own.

Detection: A customer reports that analytics counts are off, typically after moving a group or a project to another namespace (the old counts are still included in their report).

Description: traversal_path is a ClickHouse-only, denormalised column (the leading primary key in most Siphon-replicated tables) that encodes the organization, group and project hierarchy. It’s set on insert via a dictionary lookup, so it can go stale:

• When a group or project is moved, or an organization is assigned.
• Because of the chicken-egg problem: a record (issue, merge request) is ingested before its project or group, so its traversal_path becomes 0/. This usually happens with GitLab QA tests.

The reconcilers scan the tables, detect outdated values, and fix them by submitting a special package that’s processed by the consumers (the -refresh consumers).

Steps:

1. Check the ClickHouse Reconcilers dashboard: when was the last reconciliation for the given table, and how many rows were reconciled.
2. After a group or project move, it can take from minutes to several hours to be fully reconciled. That’s expected today (there are ideas to improve this to a few minutes).

These processes are pretty stable. Even if they crash, it’s not an emergency and doesn’t warrant an S1 or S2. The Siphon team can look at it and fix it.

Detection: Two opposite failures, with different signals.

Aggressive retention, data loss that has already happened: siphon_retention_floor_below_first_seq_total > 0, with the log line slowest consumer ack floor is below stream first sequence with pending data. This means stream retention evicted messages a live consumer hadn’t acked yet. Any non-zero value warrants investigation of the named consumer, it has a permanent gap. min_days (10 on production) is the floor the retention window can never go below; a consumer lagging by more than that is at risk.

Stalled, unbounded disk growth: NATS stream storage growing while siphon_retention_maxage_seconds doesn’t change. Streams are created with unlimited MaxBytes/MaxMsgs, so MaxAge is the only limit; a retention loop that never updates lets the NATS disk grow without bound.

The trap: siphon_retention_maxage_updates_total staying flat is normal in steady state, because updates within 24h of the current value are skipped. The skip reasons (no consumers on stream, lowest ack floor is 0 (nothing acked), and MaxAge within no-op threshold) emit no error metric at all, so an alert built only on siphon_retention_claim_errors_total will miss a permanently stalled loop. Read the skip reason from the logs.

Steps:

1. Check which case applies:

   kubectl logs -n siphon -l app=retention-manager --tail=200 \
     | grep -E "retention cycle failed|retention manager failed|skipping( retention)? update|clamping to first sequence"

2. For hard failures, siphon_retention_claim_errors_total carries a phase label (connect, stream_info, list_consumers, message_time, update_maxage) that localises the problem. phase="update_maxage" means NATS rejected the write.

3. siphon_retention_maxage_seconds reports the current window; 0 means unlimited.

4. Retention is configured one manager per NATS stream, not per producer. Duplicate entries for the same stream are a misconfiguration.

Everything above (dashboards, Kibana, kubectl, Argo CD) applies to GitLab.com staging and production, where the Siphon team has full access. On Dedicated it’s more involved:

• Access isn’t guaranteed, and the dashboards aren’t fully ready yet. There are rudimentary dashboards for NATS and Siphon; more detailed ones are coming.
• Metrics are available, so Prometheus queries work. Logs are shipped to OpenSearch.
• Most actions (restarting a service, changing the config map) are done by the Dedicated SREs. They follow a break glass workflow and have to inform the customer, so any change requires more sign-offs.
• There are no extra safeguards for the replication slot on Dedicated today; monitoring the slot is up to the SREs.
• The table definitions must match the GitLab version (tables and columns drift between versions), see How Siphon is configured.

The Siphon team can still help by guiding the SREs through the steps.