GATE - iam-data-access — Runbook

iam-data-access is the composite Lookup + Update gRPC service in the GATE/IAM monorepo (gitlab-org/auth/iam): it serves identity/access relationship reads (Lookup API) and writes (Update API) on one shared gRPC port, backed by a Yugabyte data store (see the YugabyteDB Aeon runbook for the DB side). It is deployed via Runway under the service ID iam-data-gke-grpc.

Use this when something is wrong or a consumer reports errors calling into IAM. Sections are ordered for an incident: triage first, then observability, alerts, troubleshooting, and escalation. The Reference section at the bottom covers environments, Vault paths, and config file locations.

• Scope: staging or production? (Runway service ID either way: iam-data-gke-grpc)
• Health: kubelet already probes Health continuously (liveness/readiness), so a one-off manual call mostly just re-checks what Kubernetes already knows. Instead, watch the probe heartbeat itself: same production/staging A steady stream of Health calls means probes are succeeding; gaps or errors mean they’re not. (A manual external call is still useful for testing the full client→gateway→pod path specifically — see 4.1.)
• Deployment status: check the Runway deployment pipeline for the latest run. A green “Upload Resource Manifests” job is not a successful deploy — it only publishes a Flux OCI artifact; Flux reconciles it async on the GKE cluster. To confirm the actual pod state — see 4.1: kubectl get pod -n iam-data-gke-grpc.
• Client vs. service: is only one caller failing (suspect auth, not this service) or is everyone failing (suspect deployment or the backing DB)?
• Logs: production · staging (Grafana Explore, health checks excluded; see Observability to adjust the query).
• Recent change: check the security-mirror main pipeline for a recent Runway trigger, and recent MRs on gitlab-org/auth/iam — did a deploy or config change trigger this? If so, assess a rollback (4.5).
• Capture: exact error text (gRPC status + message), affected RPC, timestamps, env — you’ll need these for escalation.

Then jump to the matching section: Observability · Alerts · Troubleshooting · Incident response · Reference.

Start here: production logs · staging logs — Grafana Explore, health checks excluded. These are also the service catalog’s logging links for iam-data-gke-grpc.

To adjust the query: datasource ClickHouse - Runway Production → table observability → query type logs.

• Primary filter: ServiceName = 'iam-data-gke-grpc'
• Other useful fields: SeverityText (info/warn/error), Attributes['environment'] (staging/production), Attributes['cloud.runtime'] (gke), k8s.pod.name, k8s.deployment.name, k8s.namespace.name, k8s.cluster.name.
• Example filter: WHERE ServiceName = 'iam-data-gke-grpc' AND SeverityText = 'error' AND Attributes['environment'] = 'production'.

iam-data-gke-grpc overview.

• Shows the default runway-k8s-archetype SLIs: apdex/error-rate/RPS from runway_lb_* (frontend/backend load-balancer metrics), plus saturation (CPU/memory/HPA).
• DB-layer (“storage”) metrics — iam_data_access_db_pool_connections (Yugabyte pool connections by pool/state, plotted against iam_data_access_db_pool_limit as dashed red per-pool ceilings — sum by (pool), which sums across replicas (each opens its own pool, so a ceiling scales with the replica count) while keeping lookup and update separate, so each limit stays comparable to its own usage series) and iam_data_access_repository_op_duration_seconds (repository-layer operation duration).
• DB pool contention — iam_data_access_db_pool_acquires_total (acquire rate), iam_data_access_db_pool_acquire_wait_seconds_total / acquires (mean time blocked waiting for a connection), iam_data_access_db_pool_acquires_empty_total / acquires (fraction of acquires that found the pool empty — the saturation signal), and iam_data_access_db_pool_acquires_canceled_total (callers that gave up waiting; should be flat zero). All labelled by pool. Read these before the raw connection counts: they show whether a full pool is actually costing latency. See docs/metrics.md in the iam repo for the full metric list.
• gRPC latency/request/error rate — grpc_server_handling_seconds (p50/p99 by method) and grpc_server_handled_total (request rate, error rate by grpc_code), split into Lookup/Update panels. Excludes each service’s own Health probe RPC.
• gRPC auth vs. other errors — both auth layers (4.2) reject with grpc_code=Unauthenticated, so a dedicated panel splits error rate into auth rejections vs. everything else, to see which dominates.
• None of the app-level metrics above are wired into the SLI/alerting framework yet — they’re only visualized as raw panels so far.

gRPC Health RPC per sub-service (gitlab.iam.lookup.v1.LookupService/Health, gitlab.iam.update.v1.UpdateService/Health), served on the same port as regular gRPC traffic (:5005) — kubelet’s liveness/readiness probes and the Health RPC share that port. App metrics (see Metrics above) are served separately on :9090.

The Runway metrics-catalog entry for iam-data-gke-grpc provides the default runway-k8s-archetype alerts via Alertmanager/Slack/incident.io:

• SLO violation alerts: apdex, error rate, traffic cessation — based on the generic runway_lb_* SLI (no RPC-method-level breakdown yet, see Metrics).
• Saturation alerts: CPU, memory, HPA utilization.
• Default severity is S4 (Slack #feed_alerts-general); S1/S2 pages on-call. See alert routing conventions.

These alert on service-level symptoms only — they won’t catch an iam-update-only or iam-read-only regression while the other half of the composite service looks healthy. For that, also watch consumer-side symptoms: GitLab Rails-side errors on calls into IAM, or reports from teams integrating with the Lookup/Update APIs.

Symptoms: connection refused/timeouts on all RPCs (including Health), or a deploy that “succeeded” in CI but nothing changed.

• Check the Runway deployment pipeline — a green “Upload Resource Manifests” job only publishes a Flux OCI artifact; Flux reconciles it asynchronously on the target GKE cluster.

• Check deployment health; two options:

  1. Via kubectl:
     • Use kubectl via the KAS proxy on ops.gitlab.net — see Connecting with kubectl for setup (glab cluster agent update-kubeconfig, pointed at the right agent — find it on the kas-clusters agents page).
     • Then kubectl get pod -n iam-data-gke-grpc to see actual pod status/restarts/events.
  2. Via GCP console: Go to deployment details

• Check logs for crash/startup errors (bad CONFIG_TOML, missing Vault secret — see 4.4).

• To test the full external path (client → gateway → pod) rather than just pod-level health, call the Health RPC directly — exempt from both auth layers, so no token needed:

  # production: iam-data-gke-grpc.svc.gitlab.net
  # staging:    staging.iam-data-gke-grpc.svc.gitlab.net
  buf build -o - | grpcurl -protoset /dev/stdin \
    -d '{}' <host>:443 gitlab.iam.lookup.v1.LookupService/Health

  Useful when the health-check heartbeat in logs looks fine (pod is healthy) but callers still can’t reach the service — points at routing/gateway rather than the pod itself.

Symptoms: calls fail with opaque Unauthenticated (gRPC) or 401 (HTTP); Health still works (it’s exempt from both auth layers).

Two independent layers, both required on every protected RPC:

• Confirm the caller is using the gitlab-iam-data-access-token header specifically (per-service header names are intentional — sending it under another service’s header name gets the same opaque rejection).
• Confirm the JWT’s aud claim matches this service’s configured expected_audiences.
• Service token value: Vault-managed (see Vault paths).
• Full details: docs/service-access.md in the iam repo.

iam-data-access is backed by a YugabyteDB Aeon cluster. Connection failures, latency, connection-pool exhaustion, node/cluster health — all covered in the YugabyteDB Aeon runbook, not duplicated here.

Symptoms: service fails to start, or starts with wrong config (wrong DB host, CORS rejecting a valid origin, wrong OAuth issuer).

• Runway can’t mount configMap volumes, so non-secret config is passed inline as a CONFIG_TOML env var (.runway/values-{staging,production}.yaml) — the container entrypoint writes it to disk and points CONFIG_FILE_PATH at it.
• Secrets are injected as individual env vars from Vault via Runway’s secretEnvFrom (workloadSecrets: [yugabyte, service-token] in .runway/deployment.yaml). A missing secret shows as its config-file placeholder value (VAULT_MANAGED) leaking through unchanged — check the Vault paths below.
• Confirm expected values against Vault paths and the environment’s CONFIG_TOML block in .runway/values-<env>.yaml.

• Production deploys are manual-gated (runway_manual_production_deploy: true) — staging auto-deploys on every release tag.
• Chart version is pinned per deploy (oci://registry.gitlab.com/gitlab-org/auth/iam/chart/iam-data-access:<ChartVersion> in .runway/deployment.yaml) — a rollback is re-triggering the Runway deployment pipeline against the previous known-good tag.
• See docs/release.md in the iam repo for the full release/deploy flow.

Flux may auto-rollback before you start: runwayctl generates every HelmRelease with upgrade.remediation.strategy: rollback and upgrade.remediation.retries: 3, with spec.timeout: 5m per attempt — up to 4 failed attempts total (initial plus 3 retries), so worst case is roughly 20 minutes to exhaustion. Rollback happens between attempts and the release is left rolled back once retries are exhausted, so a flapping deployment is expected behaviour for up to ~20 minutes; observing one rollback does not mean Flux has finished. This catches releases whose readiness probe never succeeds (DB unreachable, JWKS fails to load, pod hangs on startup). Check whether a rollback is already in flight before starting a manual one. These values come from runwayctl’s source defaults rather than the live HelmRelease, so a per-service override would take precedence — read the live HelmRelease to confirm.

Flux will not catch a release that comes up healthy but is functionally wrong — bad authorization logic, wrong data, etc. Pods are Ready, the upgrade succeeds, remediation never fires.

Smoke tests won’t revert a bad deploy: they run in CI after the deployment is already published to Flux. A failure turns the pipeline red and leaves the deployment in place; there’s no revert job wired in.

Check the schema before rolling back (the critical one): migrations are forward-only and never rolled back per ADR-DATA-02. Rolling the image back does not roll the schema back. Rolling back across an expand-phase migration is safe. Rolling back across a contract phase (DROP TABLE, DROP COLUMN, ALTER COLUMN TYPE, column renames) or a NOT NULL column added without a default is not safe — the older binary expects structure the database no longer has, or doesn’t write a column the database now requires. Then the rollback breaks the service instead of fixing it.

Find schema changes between two releases with:

git diff v<old>..v<new> -- db/migrations/ \
  | grep -iE '^\+.*(DROP (TABLE|COLUMN|INDEX)|ALTER COLUMN .*TYPE|RENAME (TO|COLUMN)|NOT NULL)'

Empty output means the schema is safe to roll back across. Any output means read it and judge — it’s a prompt, not a gate. A NOT NULL add is harmless on an empty table (migrations document this) and dangerous on a populated one.

The re-trigger route has an expiry: re-running “Upload Resource Manifests” on an older pipeline works by re-fetching its upstream artifacts, which expire after a window. Check they still exist before depending on it under pressure; use Option B otherwise.

Option A: redeploy the previous release (only if schema check came back empty)

1. Find the deployment project for this service: deployments/iam-data-gke-grpc.
2. Navigate to the pipeline list and find the last known-good release version (check the tag).
3. In that pipeline, locate the 📤 Upload Resource Manifests: [production] job. Re-run it if it already ran, or play it if it has not.
4. That job publishes a Flux OCI artifact; Flux reconciles asynchronously on the cluster (see 4.1 for confirmation mechanisms). Monitor pod status while Flux works.

Option B: forward fix (when schema check found a contract-phase or NOT NULL change, or if Option A’s artifacts have expired)

1. Revert the offending commit on canonical main.
2. Merge it with a fix: prefix — semantic-release cuts a new patch release and the security-mirror main pipeline triggers the deployment pipeline in the deployments project.
3. A fresh deployment pipeline is created in the deployments project — find it and play 📤 Upload Resource Manifests: [production].

This path costs a full release cycle — slower than Option A, but always available.

One release rolls three deployments: this repo drives iam-data-gke-grpc (data-access), iam-auth-ext-gke (auth HTTP), and iam-auth-gke-grpc (auth gRPC). Each has its own deployment project and manual production gate. If the release touched more than data-access, repeat your chosen option per affected deployment.

Verify the rollback: the authoritative signal is the commit SHA returned by the service’s Health RPC. Use the grpcurl invocation and kubectl get pod path described in 4.1 to check which release is actually serving. If Flux auto-remediated while you were working, the pods may already be on the previous release — the Health SHA settles which one is live.

Staging issues are normally ≤ S3 unless they block a release.

See Quick triage. Mitigate what’s in our control (rollback, config/secret fix); escalate deployment/cluster-level issues to Runway.

• On-call escalation: https://app.incident.io/gitlab/on-call/escalation-paths/01KCQ3V11Z5MXPYSWEG4F368CV
• Owning team: group::authentication
• Incident channel: #g_authentication

• Ask in #f_runway for deployment/cluster-level issues (Flux reconciliation, pod crash diagnosis, PSC/network issues) — the Runway team owns the platform.
• Read-only kubectl access for non-Infra engineers is now available for both staging and production — see Connecting with kubectl.

There are two distinct entrypoints into this service — which one a caller uses depends on whether it runs in the same GKE cluster as this service:

Public entrypoint (Cloudflare zone): .runway/deployment.yaml sets loadBalancing.externalLoadBalancer.enabled and .cloudflareSetting.enabled to true, which provisions a dedicated Cloudflare zone per environment — production at iam-data-gke-grpc.svc.gitlab.net, staging at staging.iam-data-gke-grpc.svc.gitlab.net. These front the same GKE external load balancer Runway provisions from the GCP origin (<workload-name>[.staging].runway.gitlab.net). This path is for callers outside this service’s GKE cluster — e.g. the main GitLab.com GKE clusters, which run separately from Runway’s. No auth header is required to reach the host itself — the service’s own two-layer auth still applies once connected.

Internal entrypoint (direct pod IPs via Headless Service): the intent is that other services co-located in the same GKE cluster — e.g. Artifact Registry, which does run in the same staging cluster (runway-gke-stg-us-east1) — skip the public internet entirely and reach this service’s pods directly via in-cluster Service discovery (plain Kubernetes DNS, not a Private Service Connect setup), rather than the external load balancer/Cloudflare path above.

Secrets are injected via Runway’s secretEnvFrom; deep links to the output secrets:

• Yugabyte creds: staging · production
• Service token: same Runway KV mount, env/<env>/service/iam-data-gke-grpc/service-token (see workloadSecrets in .runway/deployment.yaml).

See iam/.runway/*

• Release flow, mirrors, and manual-production-gate details: docs/release.md.
• Actual Runway staging/production deploy jobs run in the separate Infra-owned project — deployments pipeline list, not in iam’s own CI. Staging auto-deploys on release tag; production requires manual approval.

• gitlab-org/auth/iam — repo root
• docs/service-access.md — service-token + JWT auth model
• docs/relationships-api.md — Lookup/Update API & data model
• YugabyteDB Aeon runbook — backing database