DuoWorkflowSessionCreditsSyncBacklogPersistent

This alert fires when the v2 Duo Agent Platform session credits sync has more work than it can finish. The concurrency limit queue of Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker has not been empty once in the last 24 hours, so groups left over from one tick are still waiting when the next tick starts, for at least 4 ticks in a row.

How the sync works:

• Every 6 hours, Ai::DuoWorkflows::SyncSessionCreditsCronWorker selects the top-level groups that used Duo Agent Platform in the last 3 days and have the duo_workflow_session_credits_sync_v2 feature flag on.
• It schedules one Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker job per group, spread over the first 5.5 hours of the tick.
• At most 20 of those jobs run at once (the worker’s concurrency limit). Jobs that cannot start wait in the concurrency limit queue, and a healthy tick drains it before the next one.

Nothing is lost while the queue is backlogged: each job resumes from its group’s cursor. But the groups at the back of the queue sync later and later, so their credit usage falls behind.

When it fires, find out why the sync cannot keep up (see Troubleshooting), and raise the concurrency limit or fix the slow dependency.

• Sidekiq service overview
• Sidekiq concurrency limit
• Owner: group::optimize, Slack #g_analytics_optimize

The alert reads sidekiq_concurrency_limit_queue_jobs{worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"}: the number of jobs waiting in the worker’s concurrency limit queue, exported by ConcurrencyLimitSampler in Sidekiq. It is a count of jobs.

The SLO is an empty queue at the start of every tick, and the alert is meant to fire after 4 ticks in a row (more than 24 hours). The tick start time is set per instance, so the rule cannot pick out tick starts. It checks instead that the queue has not been empty once in 24 hours. This is a conservative proxy: it can miss a backlog that drains briefly in the middle of a tick and refills before the next one. The exact measurement is namespace_jobs_buffered on the cron’s done line (see Verification).

• max_over_time(...[5m]) bridges the sampler’s lease handoff: every 5 minutes the old holder reports 0 for a moment.
• The count guard needs most of 24 hours of data, so a series that just appeared cannot fire early.

Normally the queue is empty most of the time and grows for a while during a tick, only when more than 20 jobs are due at once.

• The alert is s3: it does not page, and goes to #g_analytics_optimize.
• It is expected to be rare: it means the number of active groups outgrew the concurrency limit, or jobs got much slower.
• It clears as soon as the queue empties once, even if the next tick backlogs again; it then needs another full 24 hours to fire.
• During a CustomersDot outage each job fails on its first request and ends, so the queue still drains and the alert does not fire. The error rate alert covers that case.
• A CustomersDot rate limit makes jobs wait inside the job, up to their 120-second budget, before they stop with stop_reason: rate_limited. If it hits many groups at once, the jobs get slower and the queue can back up.

• Impact: customers of GitLab.com top-level groups that use Duo Agent Platform see session credit usage that is hours behind. Billing data in CustomersDot is not affected.
• Incident severity: usually S4. Raise to S3 if the queue keeps growing from tick to tick, or if customers report stale credit usage.

Queue size, bridged over the sampler handoff:

max by (env, environment) (
  max_over_time(sidekiq_concurrency_limit_queue_jobs{env="gprd", worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"}[5m])
)

Running jobs against the limit:

max by (env) (sidekiq_concurrency_limit_current_concurrent_jobs{env="gprd", worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"})
max by (env) (sidekiq_concurrency_limit_current_limit{env="gprd", worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"})

Dashboard: Sidekiq worker concurrency detail.

Kibana, pubsub-sidekiq-inf-gprd index, cron done lines, one per tick:

json.class: "Ai::DuoWorkflows::SyncSessionCreditsCronWorker" and json.job_status: "done"

Under json.extra.ai_duo_workflows_sync_session_credits_cron_worker.*:

• namespace_jobs_buffered: jobs still in the queue when the tick started. This is the SLO measurement itself; it should be 0.
• namespaces_enqueued: jobs scheduled by this tick.

Namespace job done lines, for the job durations and stop reasons:

json.class: "Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker" and json.job_status: ("done" or "fail")

• Recent changes to the session credits code
• Feature flag rollout steps for duo_workflow_session_credits_sync_v2 add groups at once: search #production or the feature flag log.

1. Check namespace_jobs_buffered on the last few cron done lines. Growing from tick to tick means the sync falls further behind every tick; flat means it is just over capacity.
2. Compare namespaces_enqueued across ticks. A step up usually follows a feature flag rollout step.
3. Check how long the namespace jobs take (json.duration_s on the done lines). Most should finish well under the 120-second budget; many at 120 seconds with stop_reason: over_time mean groups with large backlogs, for example right after a rollout step, when each new group reads 90 days of sessions.
4. Check whether jobs are slow because of a dependency: CustomersDot response times, or ClickHouse insert latency.

The capacity of one tick is about 20 × 360 minutes ÷ average job duration, since the queue keeps draining until the next tick starts. At the 120-second worst case that is about 3,600 jobs.

• Rollout step added more groups than one tick can sync: hold the rollout until the backlog drains. A new group’s first syncs are the slowest, so it often drains on its own within a day or two.
• Steady growth in active groups: raise the worker’s concurrency limit (concurrency_limit in SyncNamespaceSessionCreditsWorker). Agree with the Fulfillment team first, because every job makes CustomersDot calls (see INC-13873).
• A slow dependency: fix or escalate it; the backlog drains once jobs are fast again.

• CustomersDot GraphQL sessions query (job duration).
• ClickHouse (job duration).
• Redis concurrency limit queue (memorystore-redis-concurrency-limit).

• group::optimize in #g_analytics_optimize.
• Raising the concurrency limit or CustomersDot slowness: the Fulfillment team in #g_monetization_platform.

• Alert definition, generator wiring and unit tests.
• The 24-hour window is the SLO’s 4 ticks. Shortening it makes the alert fire on a single slow tick, which drains on its own.
• Update the template used to format this playbook

• Related alerts
• SidekiqServiceDuoWorkflowSessionCreditsSyncErrorSLOViolation
• Add SLIs, SLOs and alerts for v2 session credits ingestion