PMDB distribution service

• Alerts: https://alerts.gitlab.net/#/alerts?filter=%7Btype%3D%22pmdb-dist-svc%22%2C%20tier%3D%22sv%22%7D
• Label: gitlab-com/gl-infra/production~“Service::pmdb-dist-svc”

• pmdb-dist-svc asia-northeast1
• pmdb-dist-svc us-east1
• pmdb-dist-svc europe-west1

pmdb-dist-svc, aka PDS, is responsible for providing non public PMDB data to GitLab instances. PMDB data like malware advisories live in a private bucket. GitLab instances need to be authenticated before PDS provides them with signed URL to download the data.

You can find the overall architecture in the dedicated ADD document.

The service exposes custom application metrics scraped by Runway into the mimir-runway datasource (use it in Grafana Explore for all queries below). Custom alerts are defined in mimir-rules/runway/pmdb-dist-svc.yml, are scoped to production (env="gprd"), and route to #g_ast-composition-analysis-alerts (team composition_analysis).

The /all and /delta endpoints and the manifest currently serve malware data only; when the licenses endpoints are added, the malware-named alerts and their selectors need revisiting.

Alerts: MalwareManifestRefreshErrorRateHigh, MalwareManifestStale2h/4h.

Each instance refreshes the malware manifest from GCS every ~5 minutes. The error-rate alert means refresh attempts are actively failing; the staleness alerts mean an instance hasn’t completed a successful refresh in hours (this also catches a silently stalled refresh loop that produces no errors). Either way, affected instances serve an increasingly outdated manifest.

• Check Cloud Run logs for refresh errors (GCS access, timeouts).

• Check GCS availability and the service account’s permissions on the manifest bucket. GCS-side errors by operation:

  sum by (operation) (rate(gitlab_object_storage_operations_total{env="gprd", result="error"}[5m]))

• Worst-instance staleness (healthy: ~300s):

  max((time() - gitlab_manifest_cache_last_refresh_success_timestamp_seconds{env="gprd"})
    and (gitlab_manifest_cache_last_refresh_success_timestamp_seconds{env="gprd"} > 0))

Alerts: MalwareManifestNotUpdated12h/24h.

The service is healthy — refresh checks succeed — but the manifest content never changes. This means the upstream publishing pipeline (PMDB) has stopped producing new manifests; investigate there, not in this service. Publish count over the last 12h (each bump above 0 is a publish):

max(increase(gitlab_manifest_cache_refresh_total{env="gprd", result="changed"}[12h]))

Alerts: MalwareAllCacheMissRateHigh, MalwareDeltaCacheMissRateHigh.

A background loop pre-signs object-storage URLs so requests are served from cache; a miss signs on demand inside the request, adding latency. Sustained misses mean the pre-signing loop isn’t keeping up or isn’t running.

• Check logs for signing errors and IAM SignBlob failures.

• Signing quota exhaustion (should be 0) and p95 signing latency:

  sum(rate(gitlab_signed_url_generation_duration_seconds_count{env="gprd", result="quota_exhausted"}[5m]))
  
  histogram_quantile(0.95, sum by (le) (rate(gitlab_signed_url_generation_duration_seconds_bucket{env="gprd", result="success"}[5m])))

• Note: during very-low-traffic hours a handful of lookups can inflate the miss ratio; sanity-check volume before digging deeper:

  sum by (endpoint) (increase(gitlab_signed_url_cache_operations_total{env="gprd"}[1h]))

Alert: MalwareDeltaUnsupportedPurlTypesRequests.

Some client is persistently calling /delta with purl types that aren’t in the malware manifest; the service skips them and reports them in the response’s not_supported array. The offending registry names are deliberately not a metric label (unbounded cardinality) — find them in the request logs under skipping unsupported purl_types in /delta request, then track down and fix the client.

Alert: PmdbDistSvcJwksRefreshErrors.

The service can’t fetch OIDC signing keys. Not yet user-visible: cached keys keep validating tokens for up to 7 days, after which all clients get 401s indistinguishable from bad tokens — fix before the cache expires. One fetch attempt covers every issuer in PMDB_OIDC_PROVIDERS; any single issuer failing counts as an error for the whole attempt.

• Check service logs for which issuer is failing.
• Verify the issuer’s .well-known/openid-configuration and JWKS endpoints are reachable from the service.

Alert: PmdbDistSvcRegionInstancesAtMax.

The region has run at the Cloud Run max-instances cap (10; baseline 5) for 30+ minutes — autoscaling is saturated and additional load can’t scale out. Investigate whether the traffic is legitimate; if sustained, raise max-instances in the Runway config and update the alert threshold to match (it’s hardcoded in the alert expression).