Pulp SLIs
Pulp uses Service Level Indicators (SLIs) to monitor the health and performance of its components. These SLIs are defined in the metrics catalog and are used to measure availability and latency for the service.
Overview
Section titled “Overview”The Pulp service has the following SLIs:
| SLI | Description | Type |
|---|---|---|
pulp_app_api | Pulp application API service requests | Apdex + Error Rate |
pulp_app_content | Pulp application content API requests | Apdex + Error Rate |
pulp_nginx | Nginx ingress controller load balancer requests | Apdex + Error Rate |
pulp_cloudsql | GCP CloudSQL PostgreSQL database operations | Apdex + Request Rate |
pulp_gcs | GCS bucket storage operations | Request Rate |
pulp_redis | GCP Redis Memorystore caching operations | Request Rate |
Observability
Section titled “Observability”The SLIs appear in the Pulp Overview dashboard.
CloudSQL and Memorystore
Section titled “CloudSQL and Memorystore”The Pulp Overview dashboard includes two collapsed rows: CloudSQL and
Memorystore. The CloudSQL row shows performance, connection, disk,
network, and tuple panels. The Memorystore row shows client, workload,
and data panels.
To find out if CloudSQL is slow, open the CloudSQL row. The first panels
show the query latency, the query, IO, and lock time from Query Insights,
CPU and memory utilization, the data cache hit ratio, temporary bytes,
deadlocks, disk throughput, and transactions.
The Stackdriver exporters label these metrics type="pulp".
gitlab-com/gl-infra/argocd/apps!4039 and gitlab-com/gl-infra/argocd/apps!4053
add the relabel rules. gitlab-com/gl-infra/argocd/apps!4052 adds the ops
metric allowlists.
The saturation rules track six components: cloudsql_cpu, cloudsql_disk,
cloudsql_memory, memorystore_redis_cpu, memorystore_redis_memory, and
memorystore_system_memory.
Application SLIs
Section titled “Application SLIs”pulp_app_api
Section titled “pulp_app_api”The pulp_app_api SLI monitors the Pulp API service, which handles administrative operations such as repository management, content synchronization, and user management.
Metrics:
api_request_duration_milliseconds_bucket- Request latency histogramapi_request_duration_milliseconds_count- Total request count
Significant Labels:
http_method- HTTP request method (GET, POST, PUT, DELETE, etc.)http_target- API endpoint pathhttp_status_code- HTTP response status code
Error Rate: Tracks 5xx HTTP status codes as errors.
pulp_app_content
Section titled “pulp_app_content”The pulp_app_content SLI monitors the Pulp Content API service, which handles package downloads and content delivery to clients (e.g., yum/dnf clients fetching packages).
Metrics:
content_request_duration_milliseconds_bucket- Request latency histogramcontent_request_duration_milliseconds_count- Total request count
Apdex Thresholds:
Significant Labels:
http_method- HTTP request methodhttp_route- Content API routehttp_status_code- HTTP response status code
Error Rate: Tracks 5xx HTTP status codes as errors.
Other Metrics
Section titled “Other Metrics”In addition to the SLIs above, the following metrics are available in the Pulp Overview dashboard.
Task Queue Metrics
Section titled “Task Queue Metrics”Longest Unblocked Task Wait Time
Section titled “Longest Unblocked Task Wait Time”Tracks how long the oldest unblocked task has been waiting in the queue. Lower values are better.
Metric:
Unblocked Task Queue Length
Section titled “Unblocked Task Queue Length”Tracks the number of unblocked tasks waiting to be processed. Lower values are better.
Metric:
Infrastructure SLIs
Section titled “Infrastructure SLIs”pulp_nginx
Section titled “pulp_nginx”Monitors the nginx ingress controller that load balances traffic to Pulp services.
Metrics:
nginx_ingress_controller_request_duration_seconds_bucket- Request latency histogramnginx_ingress_controller_requests- Total request count
Apdex Threshold:
Significant Labels:
method- HTTP methodpath- Request pathstatus- HTTP status code
pulp_cloudsql
Section titled “pulp_cloudsql”Monitors the GCP CloudSQL PostgreSQL instance used by Pulp.
Metrics:
stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_statements_executed_count(request rate)stackdriver_cloudsql_instance_database_cloudsql_googleapis_com_database_postgresql_insights_aggregate_latencies_bucket(apdex, from Query Insights, in microseconds)
Apdex: a query is satisfied if it takes less than 1 s
(the 1048576 microsecond bucket, stored as le="1.048576e+06"). A
database query that takes longer than 1 s is slow. The SLO alert fires
when more than 0.1% of the queries are slow, for example because of CPU,
IO, or lock problems. It does not fire for one slow query.
Significant Labels:
database_id- Cloud SQL database identifierdatabase- Database nameoperation_type- Type of SQL operation
pulp_gcs
Section titled “pulp_gcs”Monitors the GCS bucket used for package storage.
Metrics:
Significant Labels:
bucket_name- GCS bucket namemethod- API method
pulp_redis
Section titled “pulp_redis”Monitors the GCP Redis Memorystore instance used for caching and session management.
Metrics:
Significant Labels:
instance_id- Redis instance identifier