DatabasePartitionSyncStalled

This alert fires when 5 or more partitioned tables have been missing at least one partition for 12 hours, in one environment.

GitLab keeps partitions ahead of time for its partitioned tables: daily tables 28 days ahead, monthly tables 6 months ahead, and CI tables one partition ahead. The dynamic partition sync (Gitlab::Database::Partitioning.sync_partitions) creates them. A healthy sync creates whatever is missing within one run, so a growing list of tables with missing partitions means the sync has stopped. That happens in two ways:

• The sync is disabled. partition_manager_sync_partitions (or disallow_database_ddl_feature_flags) was turned off, for example during an incident or maintenance, and has not been turned back on.
• The sync keeps failing. Each run tries and fails, for example because the ATTACH PARTITION for the next CI partition cannot get its lock on p_ci_builds (INC-15918).

Nothing breaks straight away, which is why this alert does not page. But every day the sync stays stalled uses up headroom: once a daily table runs out of its 28 days, inserts into it fail. Old partitions also stop being detached, so disk use grows.

When it fires, find out which of the two cases it is, and make sure there is an owner and a date for getting the sync running again.

• Patroni Service
• CI Postgres (Patroni) Service
• Team: Database Operations (database_automation), #s_database_excellence_alerts

• db_partitions_missing: a Rails gauge, one series per partitioned table (table label). It is the number of partitions the table’s partitioning strategy expects but cannot find.
• Database::PartitionManagementWorker (every 6 hours) and Database::DropDetachedPartitionsWorker set it after each run (partition_monitoring.rb). They report even when the sync is disabled.
• The gauge lives on the Sidekiq pod that ran the worker. Every pod keeps exporting the value from its own last run until it is replaced, so when pods live long (over a weekend, with no deploys) several pods export different values for the same table, and only the newest is current. Do not take the max across pods: on 2026-11-09 one pod kept a stalled report for a day after the sync had recovered.
• gitlab:db_partitions_missing:latest_report: a recording rule that keeps only the newest report, from the pod that started exporting the gauge most recently (gitlab:db_partitions_report:first_seen_timestamp_seconds). A pod that runs the worker a second time is the exception: its newer values only count once another pod reports. A pod’s report keeps counting for 7 hours after the pod is gone, longer than one worker interval. That bridges the gap after a deploy replaces every pod, and it stops an older pod’s stale report from taking over when only the newest pod dies.
• Threshold. Over the 30 days before INC-15918, gprd never had more than 3 tables missing a partition at once. Those are usually tables that rotate quickly, such as p_ci_finished_pipeline_ch_sync_events, and they catch up on the next run. Within hours of disabling the sync in INC-15918 the count was 12, and 25 a day later. gstg had one spike to 22 tables in the same period, which cleared after about 9 hours. The 12-hour for keeps spikes like that from firing.
• For a daily table, the missing count is roughly the number of days the sync has been stalled. When it gets close to 28, inserts into that table are about to fail.

• Expected to be rare. It fires when the sync is disabled for longer than about half a day, or when a partition keeps failing to attach.
• If the sync was disabled on purpose and there is a plan to re-enable it, you can silence the alert until the planned date. Do not silence it with no end date: the alert is the reminder that the flag is still off.

• Default s4. There is no customer impact while headroom remains.
• Raise it if any daily table is close to running out (missing count above about 20), or if the next CI partition is needed soon (see CI partition sync lock contention).

List the tables that are missing partitions in the newest report, and how many each:

gitlab:db_partitions_missing:latest_report{env="gprd"} > 0

To see what each pod reports, newest pod first, and spot stale pods:

sort_desc(gitlab:db_partitions_report:first_seen_timestamp_seconds{env="gprd"})
count by (pod) (db_partitions_missing{env="gprd", type="sidekiq"} > 0)

Check the flags:

/chatops gitlab run feature get partition_manager_sync_partitions
/chatops gitlab run feature get disallow_database_ddl_feature_flags

In Kibana (Rails logs), search for:

• Skipping sync of dynamic postgres partitions: the sync is disabled, and the log names the flag.
• Failed to create / detach partition(s): the sync runs but fails. The log names the table and the exception, for example AttemptsExhaustedError.
• Skipping partition sync: another DDL action holds the lease of the <database> database: the sync is skipped, because reindexing, an async index or an async constraint holds the database’s async DDL lease (gitlab/database/asyncddl/actions/<database>).

• Flag changes made with chatops are recorded in the feature flag log. The most recent issue for partition_manager_sync_partitions shows who changed it and when.
• To roll back a deliberate disable, re-enable the flag as described in CI partition sync lock contention.

1. The sync is disabled. Find who disabled it and why, usually in an incident or a change request, and agree a re-enable date. For the CI tables, follow the re-enable steps in CI partition sync lock contention.

2. The sync runs but fails for the CI tables (p_ci_builds and the tables with a foreign key to it): follow CI partition sync lock contention. Check the patroni-ci rails_primary_sql Apdex as well. Repeated attach attempts can queue CI writes.

3. The sync runs but fails for other tables. The failure log names the table and the exception. A lock timeout points to contention on that table (PostgreSQL locking). Raise anything else with the team that owns the table.

4. The sync is skipped because the async DDL lease is taken. Reindexing can hold the lease for up to a day, so a skip during the weekend reindex is expected and clears when it finishes. If no REINDEX, CREATE INDEX CONCURRENTLY or VALIDATE CONSTRAINT is running on that database (check pg_stat_activity on its primary), the holder probably died and left the lease behind. A reindex that was killed leaves its row in the started state:

   SELECT index_identifier, action_start
   FROM postgres_reindex_actions
   WHERE state = 0 AND action_end IS NULL
   ORDER BY action_start DESC
   LIMIT 5;

   That is what happened over the weekend of 2026-11-07: two REINDEX CONCURRENTLY runs outlived their CronJob pod, the next hourly run replaced the pod (concurrencyPolicy: Replace), and the 1-day lease stayed behind on main and CI. The lease expires on its own. Ask the Database Operations team before you release it by hand.

• INC-15918 / production#23392: partition 118 could not attach to p_ci_builds. The sync was disabled to stop the retries, after which the missing count kept growing across every database.

• Sidekiq: the gauge comes from Sidekiq workers, so the alert goes quiet if they stop running.
• The ops feature flags partition_manager_sync_partitions and disallow_database_ddl_feature_flags.

• Database Operations in #g_database_operations.
• For the partitioning framework itself, the Database Frameworks team.

• Alert definition
• Tuning: keep the threshold above the normal 1–3 tables, and keep the for duration longer than one gap between worker runs (6 hours). Build any new query on gitlab:db_partitions_missing:latest_report, not on the raw gauge.
• Edit this playbook
• Update the template used to format this playbook

• Related alerts
• CI partition sync lock contention
• Postgres maintenance and the DDL feature flags