Sidekiq Concurrency Limit

To protect the primary database against misbehaving/inefficient workers which can lead into incidents like slowdown of jobs processing, web availability, etc, we have developed a circuit breaking mechanism within Sidekiq itself.

When the database usage of a worker violates an indicator, Sidekiq will throttle the worker by decreasing its concurrency limit at an interval of every minute. In the worst case scenario, the worker’s concurrency limit will be suppressed down to 1.

Once the database usage has gone to a healthy level, the concurrency limit will automatically recover towards its default limit, but at a much slower rate than the throttling rate. The definition of the throttling/recovery rate is defined here.

Observability around concurrency limit can be viewed at sidekiq: Worker Concurrency Detail dashboard.

There are 2 indicators on which the application will throttle a worker:

1. DB duration usage (primary DBs only)

   Dashboard:

   • Main DB duration
   • CI DB duration
   • Sec DB duration

   By default, the per-minute DB duration should not exceed a limit of 20,000 DB seconds/minute on non-high urgency worker and 100,000 DB seconds/minute on high-urgency workers (source).

   The limits above can also be overwritten as described below. To check the current limit:

   glsh application_settings get resource_usage_limits -e gprd

2. Number of non-idle DB connections

   Dashboard:

   • pgbouncer connection saturation
   • pgbouncer-ci connection saturation
   • pgbouncer-sec connection saturation

   Sidekiq periodically samples non-idle DB connections from pg_stat_activity to determine which worker classes are consuming the most connections.

   The system determines the predominant worker (the worker consuming the most connections) by:

   1. Summing the number of connections used by a worker over the last 4 samples of pg_stat_activity (approximately 4 minutes of data)
   2. The worker with the most aggregated connections is the “predominant worker”

The table below illustrates what happens when each indicator is violated:

The DB duration usage described above can only be updated by calling the application settings API. It cannot currently be set using the admin web UI.

1. Prepare a JSON file. Here’s an example to update a single worker Chaos::DbSleepWorker to have its own limit on the main DB:

   Click to expand

   ❯ cat rules.json
   {
     "rules": [
       {
         "name": "main_db_duration_limit_per_worker",
         "resource_key": "db_main_duration_s",
         "metadata": {
           "db_config_name": "main"
         },
         "scopes": [
           "worker_name"
         ],
         "rules": [
          {
            "selector": "worker_name=Chaos::DbSleepWorker",
            "threshold": 5,
            "interval": 60
          },
           {
             "selector": "urgency=high",
             "threshold": 100000,
             "interval": 60
           },
           {
             "selector": "*",
             "threshold": 20000,
             "interval": 60
           }
         ]
       },
       {
         "name": "ci_db_duration_limit_per_worker",
         "resource_key": "db_ci_duration_s",
         "metadata": {
           "db_config_name": "ci"
         },
         "scopes": [
           "worker_name"
         ],
         "rules": [
           {
             "selector": "urgency=high",
             "threshold": 100000,
             "interval": 60
           },
           {
             "selector": "*",
             "threshold": 20000,
             "interval": 60
           }
         ]
       },
       {
         "name": "sec_db_duration_limit_per_worker",
         "resource_key": "db_sec_duration_s",
         "metadata": {
           "db_config_name": "sec"
         },
         "scopes": [
           "worker_name"
         ],
         "rules": [
           {
             "selector": "urgency=high",
             "threshold": 100000,
             "interval": 60
           },
           {
             "selector": "*",
             "threshold": 20000,
             "interval": 60
           }
         ]
       }
     ]
   }

   To prepare a file with the current configuration to edit, run:

   glsh application_settings get resource_usage_limits > rules.json

2. Run a helper script glsh application_settings resource_usage_limits to update the limits with an admin PAT.

   glsh application_settings set resource_usage_limits -f rules.json -e gprd

To disable throttling globally for all workers:

/chatops gitlab run feature set sidekiq_throttling_middleware false

To disable throttling for a worker:

# replace Security::SecretDetection::GitlabTokenVerificationWorker with the worker you want to disable
/chatops gitlab run feature set `disable_sidekiq_throttling_middleware_Security::SecretDetection::GitlabTokenVerificationWorker` true

This alert fires when a Sidekiq worker has accumulated too many jobs in the Concurrency Limit queue (>100,000 jobs for more than 1 hour).

The long backlog is usually caused by higher arrival rate of jobs compared to the rate of resumed jobs by ConcurrencyLimit::ResumeWorker.

• The arrival rate is equal to the worker deferment rate, which can be found in the Worker deferment rate panel.
• The rate of resuming jobs can be found in Kibana.

If the arrival rate is consistently higher than the rate of resuming jobs, the only option is to disable the concurrency limit for the worker class as described in Option 2 below.

These jobs are queued on the dedicated memorystore-redis-concurrency-limit instance. A large backlog can saturate its memory. The instance runs noeviction, so at maxmemory new deferred jobs are rejected. Check its memory on the overview dashboard.

If the worker can safely handle more concurrent jobs:

1. Locate the worker definition in the codebase
2. Check current concurrency limit setting from the dashboard or the worker class definition.
3. Create an MR to increase the limit to an appropriate value based on concurrent jobs

If concurrency_limit attribute is not set in the worker class, consider overriding max_concurrency_limit_percentage attribute to use higher percentage of max total threads in the Sidekiq shard. The default percentage can be found here (based on worker’s urgency).

Alternatively, disable_sidekiq_concurrency_limit_middleware_#{worker_name} feature flag can be enabled to help clear the backlogs instantly without waiting for deployment as in Option 1.

1. Enable the feature flag:

/chatops gitlab run feature set `disable_sidekiq_concurrency_limit_middleware_WebHooks::LogExecutionWorker` true --ignore-feature-flag-consistency-check

2. Monitor the concurrency limit queue size to confirm it’s draining
3. If we decide to increase the concurrency limit, wait until the limit has been increased and disable the feature flag back:

/chatops gitlab run feature delete `disable_sidekiq_concurrency_limit_middleware_WebHooks::LogExecutionWorker`

When the concurrency limit middleware is disabled:

• Jobs will be resumed at a higher pace.
• New jobs will execute immediately.

1. Create an issue to properly address the root cause if Option 2 was used
2. Update monitoring thresholds if needed

This alert fires when a worker has been HardThrottled continuously for more than 5 minutes. It is an s3 symptom alert and does not page.

HardThrottle means the worker failed both database usage indicators in the same minute: it exceeded its DB duration limit, and it was the predominant holder of non-idle DB connections. Each minute in that state halves the worker’s current concurrency limit, down to a floor of 1. See Throttling events.

The alert is driven by sidekiq_throttling_events_total{strategy="HardThrottle"}, which the Throttling Events panel plots per worker and strategy.

1. Open the Current limit panel for the worker. The gap between Current limit and Max limit is how far it has been throttled.
2. Check which indicator tripped. DB duration is in the Kibana links under Database usage indicators; connection pressure is on the pgbouncer saturation panels linked there.
3. Check the Concurrency limit queue sizes panel. A throttled worker defers jobs, so a sustained throttle turns into a SidekiqConcurrencyLimitQueueBacklogged alert.

Throttling is the system working as designed. The current limit recovers towards Max limit on its own once the worker’s DB usage drops, at the recovery rate defined here.

Intervene only when the throttle is causing user-facing impact or a backlog.

Do not raise the worker’s DB duration limit as a fix. The limits are set per urgency class (production#20787). A per-worker override skews DB usage across workers and is hard to rebalance later.

Find what increased the worker’s DB usage:

1. Check whether a single user, project or namespace is driving the volume. In the DB duration Kibana views, split by json.meta.user, json.meta.project or json.meta.root_namespace (field reference).
2. Check for fan-out. Split by json.meta.caller_id to see what is enqueuing the jobs.
3. Check for a recent change to the worker’s code path. Compare the throttle start time against recent deployments and the worker’s recent MRs.
4. Check for an unoptimised query. Use the Marginalia sampler dashboard filtered to the worker’s endpoint.

Then:

• Open an issue against the worker’s owning group with the findings.
• If the throttle is causing a backlog, follow SidekiqConcurrencyLimitQueueBacklogged.
• Disabling throttling for the worker (Disabling the Throttling/Circuit Breaker feature entirely) is a last resort. Open a tracking issue to re-enable it.

• sidekiq: Worker Concurrency Detail
• memorystore-redis-concurrency-limit: Overview
• Sidekiq: Overview

• Concurrency limit worker attribute