Job Router

The Job Router is a GitLab Relay (KAS) module. Runners with it enabled request CI jobs from Relay over gRPC instead of polling Rails directly. Relay forwards each request to GitLab over HTTP and hands the response back to the runner. Runner Core owns the module and its SLI. The SLI’s alerts post to #alerts-ci-cd through feature_category: runner_core, and its weekly error budget report goes to #ci-section through team: runner_core. Environments owns the Relay service it runs in. Runners Platform owns the runner shards and the runner-side alerts. Pages from the SLO alerts go to the SRE on-call, as for every S2 SLO alert.

This page is the operational reference for the Job Router. The alert pages link here for anything they do not repeat:

• KasServiceJobRouterErrorSLOViolation
• KasServiceJobRouterTrafficCessation and KasServiceJobRouterTrafficAbsent
• RunnerManagerNotRequestingJobs, the runner-side alert, owned by Runners Platform

runner manager --gRPC GetJob--> HAProxy main (kas frontend) --> Relay --POST /api/v4/internal/ci/job_router/jobs/request--> Workhorse --> Rails

GitLab-operated managers reach the kas frontend as kas-internal.<env>.gitlab.net:8150, an internal load balancer behind a Private Service Connect endpoint in the ci VPC. Self-managed runners reach it as kas.gitlab.com through Cloudflare Spectrum. Same frontend, same Relay. The PSC side is described in the runner networking page.

1. The runner asks Rails where the router is with GET /api/v4/runners/router/discovery. Rails answers with the Relay URL, grpcs://kas.gitlab.com, when the job_router flag is on for the runner’s group (or, for an instance runner, one of the instance flags), and 501 when it is not. The runner caches the answer for an hour. A runner with job_router_url set still runs discovery, and still falls back on 501, but dials the configured address instead of the advertised one. Our managers set it to the PSC address.
2. The runner keeps one gRPC connection to Relay and calls GetJob once per poll, with the same JSON it would send to POST /api/v4/jobs/request.
3. Relay resolves the runner token through GET /api/v4/internal/ci/agents/runner/info, then forwards the job request to the internal endpoint. Workhorse treats it like the direct endpoint: with no job available it holds the request for its CI long-poll window (50 seconds) and answers 204. Relay passes the X-GitLab-Last-Update token back to the runner, which echoes it on the next poll, so Workhorse coalesces empty polls instead of sending them to Rails.
4. Relay maps GitLab’s answer to a gRPC status code and returns the body unchanged.

Relay keeps no state for this. Any Relay pod can serve any request.

A runner routes only when its shard flag and the Rails flag for its group are both on. Everything else polls Rails directly, as before. Flag changes are recorded in feature-flag-log.

The runner is built to survive the router. When Relay is unreachable, its circuit breaker trips after three consecutive failures spanning at least 5 seconds, and the runner polls Rails directly until Relay answers again. When the Rails flag is off, Relay answers Unimplemented and the runner drops the router on its next poll. So a Job Router problem usually costs job pickup time rather than jobs, and the first question in an incident is: are runners falling back cleanly, and is the direct path healthy?

Two switches, one on each side of the connection. Neither needs a restart.

/chatops run feature set job_router false
/chatops run feature set --group=gitlab-org job_router false

Run it in #production and cross-post to #g_runner_core. Check how the flag is enabled first, with /chatops run feature get job_router: while it is enabled globally, disabling it for one group has no effect, and the global flag also covers instance runners, so the rollback is the global disable. While the flag is off, the internal endpoint answers 501, Relay returns Unimplemented, and each runner drops the router on its next poll. Discovery answers 501 as well, so runners stay on the direct path until the flag is back on.

Verify: Unimplemented replaces the other codes on GetJob by Response Code, router_disabled appears on Runner-side Fallbacks/sec, and direct POST /api/v4/jobs/request traffic comes back on the incident dashboard. The traffic cessation alert will fire once the 30-minute rate has been zero for 5 minutes. Silence it with type="kas", component="job_router".

To re-enable, set the flag to true. Runners rediscover the router when their discovery cache expires, within an hour, or on restart.

Set FF_USE_JOB_ROUTER to false in the shard’s runner configuration:

• Kubernetes shards: featureFlags.FF_USE_JOB_ROUTER in services/gitlab-runner/env/gprd-ci/shards/<shard>/clusters/<cluster>/values.yaml in argocd-apps. ArgoCD applies the change.
• Chef shards: cookbook-gitlab-runner.runners.<runner>.feature_flags.FF_USE_JOB_ROUTER in the role in chef-repo. The next chef-client run applies it.

The runner reloads its configuration when the file changes and stops calling the router. Use this when one shard misbehaves, or to keep the Rails flag on for everyone else.

A Relay regression is rolled back like any other auto-deploy component. Restarting or scaling Relay is in Relay basic troubleshooting. Runners on the direct path are unaffected by anything you do to Relay.

Rails limits job requests per runner token (runner_jobs_request_api_limit, 2,000 per minute by default). A shard whose managers share one token can hit it. Through the router a 429 arrives as ResourceExhausted.

GitLab-operated managers are exempt: HAProxy sets X-GitLab-RateLimit-Bypass: 1 for requests whose source is the PSC NAT subnet of the manager traffic. On the Relay path that works because Relay forwards the manager’s address as GitLab-Real-IP and HAProxy adopts it as the source for connections from the KAS pod range before setting the header. Self-managed runners using the router arrive with their own address and the limit applies to them. If a GitLab-operated shard shows ResourceExhausted, check that its job router calls reach Rails from 10.218.11.16/28 (json.remote_ip on the internal endpoint in Kibana); if they come from a KAS pod address instead, the forwarding chain is broken somewhere between Relay and the https frontend. Relay forwards GitLab’s Retry-After and does not retry. The runner logs Checking for jobs... rate limited, releases its request slot, lowers its adaptive request concurrency, and polls again after check_interval.

Signs: ResourceExhausted on the response code panel, gitlab_runner_job_router_get_job_requests_total{code="ResourceExhausted"} rising for a shard, gitlab_labkit_rate_limiter_rule_evaluations_total{rule="limit_runner_job_requests_by_runner_token", result="block"} rising on Rails, and slow pickup on that shard with nothing else wrong. For a GitLab-operated shard, repair the forwarding chain above. Raising the limit is a stopgap only, and more tokens are not an option: a shard keeps one token and relies on the bypass. The alerts for it, and the steps to take, are in RunnerTokenRateLimit. The issue that made this visible is production-engineering#29774.

There is no latency alert. One is tracked in gitlab-runner#39391. Use the panels:

• GetJob Duration - GitLab returned a job, on both rows. Only result="job" tells you anything, because an empty poll waits out the 50 second long-poll window by design. The Relay-side panel is what Relay controls; the runner-side panel adds the network.
• Runner Controller Lookup Duration: the GET /api/v4/internal/ci/job_router/runner_controllers/job_admission call Relay makes on every job before handing it over. It runs inline in GetJob, so it is part of every job’s latency even though admission control is off on GitLab.com.
• Rails API latency and Workhorse queueing on the API dashboard. Relay CPU, memory, and in-flight requests on the Relay overview.

Slow GetJob for result="job" with a healthy Rails points at Relay. Check saturation and recent Relay deploys, and raise gitlab.kas.minReplicas in k8s-workloads if the pods are full. If pickup is unacceptably slow and the cause is on the router path, roll back to direct polling.

• Job Router: Relay-side GetJob, the runner-side view with fallbacks and circuit breakers, admission control (off on GitLab.com), and overall Relay health. Every panel description explains its codes and labels.
• Relay overview: the job_router SLI against its SLO, saturation, and the WebSocket versus gRPC transport row.
• CI Runners: request concurrency and CI Runners: incident, GitLab application for the direct path and Workhorse polling.

• Relay, Kibana index pubsub-kas-inf-gprd-*: failed job requests (Failed to request a CI job from GitLab, with json.grpc_code) and rate-limited job requests. json.correlation_id follows one request; json.agent_key identifies the runner. Unimplemented and cancelled polls are not logged.
• Runner managers, pubsub-runner-inf-gprd-*: router state changes (discovery, breaker trips, dial failures, disabled, recovered), failed GetJob calls, and rate-limited polls. Filter on json.shard.
• Rails, pubsub-rails-inf-gprd-*: the internal endpoint, with the same json.correlation_id. See Correlation IDs.
• Workhorse, pubsub-workhorse-inf-gprd-*: the internal endpoint.

• The runner creates the correlation ID. It is 32 lowercase hex characters. On the direct path the runner sends it as the X-Request-Id header. On the Job Router path it sends it as x-request-id gRPC metadata. Relay copies it into the header it sends to GitLab, and returns GitLab’s ID to the runner.
• On the internal paths, through the internal gateway or through Relay, the ID reaches Rails unchanged. Workhorse keeps IDs that come from trusted internal addresses, and nginx and HAProxy pass the header through. The same correlation_id therefore appears in the runner, Relay, Workhorse, and Rails logs.
• Runners that reach gitlab.com through Cloudflare get a different ID. This applies to customer runners and to the shards that are not on Private Service Connect: delivery-build, qa, windows and the macOS VM managers. The Cells HTTP router Worker replaces the header with <cf-ray>-<colo>. The runner logs the ID it receives in the response, so runner and Rails logs still match. kas.gitlab.com is a Spectrum TCP application, so Cloudflare never rewrites Job Router requests.
• The shape of the ID shows where it was created. 32 hex characters means the runner created it. 16 hex characters followed by a colo code means Cloudflare did. gitlab-runner#39944 has the counts and the Kibana queries.

# GetJob by gRPC code on Relay
sum by (grpc_code) (rate(grpc_server_handled_total{env="gprd", type="kas", grpc_service="gitlab.agent.job_router.rpc.JobRouter"}[5m]))

# GetJob by outcome, and the p95 for jobs actually returned
sum by (result) (rate(job_router_get_job_duration_seconds_count{env="gprd", type="kas"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(job_router_get_job_duration_seconds_bucket{env="gprd", type="kas", result="job"}[5m])))

# Why runner requests bypassed the router, per shard
sum by (shard, reason) (rate(gitlab_runner_job_router_fallbacks_total{environment="gprd"}[5m]))

# Runner managers whose breaker is open or half-open, per shard
count by (shard) (gitlab_runner_job_router_circuit_breaker_state{environment="gprd"} > 0)

# The internal endpoint as Workhorse sees it
sum by (code) (rate(gitlab_workhorse_http_requests_total{env="gprd", route_id="api_job_router_jobs_request"}[5m]))

# Job requests blocked by the per-token rate limit
sum by (result) (rate(gitlab_labkit_rate_limiter_rule_evaluations_total{env="gprd", rule="limit_runner_job_requests_by_runner_token"}[5m]))

• SLI: job_router in metrics-catalog/services/kas.jsonnet
• Dashboard source: dashboards/kas/job-router.dashboard.jsonnet
• Runner client: router/ in gitlab-runner. Relay module: internal/module/job_router/ in gitlab-agent
• Readiness assessment
• Runner documentation: feature flags and rate limiting