ActiveContext

ActiveContext enables AI-powered features that understand your codebase by indexing code files into searchable embeddings. The system chunks code files into logical segments, generates vector embeddings for each chunk, and stores them in a vector database (Elasticsearch, OpenSearch, PostgreSQL with pgvector). This enables Retrieval Augmented Generation (RAG) features like Codebase as Chat Context in GitLab Duo, where the AI can semantically search your code to provide contextually relevant responses to your questions.

The ActiveContext is an abstraction layer over different vector stores used for embeddings indexing, search, and other operations. See the Design Document and How It Works for an overview.

There are 2 ActiveContext workers:

• Ai::ActiveContext::MigrationWorker: runs every 5 minutes and picks up new migrations
• Ai::ActiveContext::BulkProcessWorker: runs every minute and processes documents queued for embeddings generation

The Code Embeddings Pipeline is an embeddings indexing pipeline for project code files that makes use of the ActiveContext abstraction layer. See the Design Document for an architecture overview.

The workers in the Code Embeddings Pipeline are scheduled through the Ai::ActiveContext::Code::SchedulingWorker. The SchedulingWorker runs every minute and checks which workers defined in the Ai::ActiveContext::Code::SchedulingService are due to run. For further details, please refer to the Index State Management section in the Design Document.

ActiveContext pipelines send embeddings generation requests to the AI Gateway.

The Code Embeddings Pipeline uses Vertex AI’s text-embedding-005 model and sends embeddings generation requests through the AI Gateway Vertex AI proxy endpoint: /v1/proxy/vertex-ai/.

• Advanced Search runbook

To start running the ActiveContext indexing pipelines, you need to create an Ai::ActiveContext::Connection record then activate it. This ensures that the pipeline workers (for example, the ones defined in Ai::ActiveContext::Code::SchedulingService) will proceed to run.

Use the gitlab:semantic_search:code:info rake task to display the current status of Semantic Code Search, including indexing status, vector store connection details, repository statistics, and embedding queue sizes.

sudo gitlab-rake gitlab:semantic_search:code:info

To monitor status continuously, provide a watch interval in seconds:

sudo gitlab-rake "gitlab:semantic_search:code:info[5]"

Press Ctrl+C to stop.

Example output:

Semantic Code Search
Indexing enabled:               yes
Indexing active:                yes

Connection:
  Name:                         elasticsearch
  Adapter:                      ActiveContext::Databases::Elasticsearch::Adapter
  Active:                       yes

Repositories
Total:                          42
Ready:                          40 (95%)
Pending:                        1 (2%)
Code indexing in progress:      1 (2%)
Embedding indexing in progress: 0 (0%)
Failed:                         0 (0%)

Embedding Queues
Code queue:                     5
  Shard count:                  1
  Shard limit:                  1000
Backfill queue:                 0
  Shard count:                  1
  Shard limit:                  1000
Retry queue:                    0
Dead queue:                     0

See also: Semantic code search documentation

To pause the MigrationWorker, you can use [Sidekiq Chatops command to drop jobs]:

/chatops gitlab run feature set drop_sidekiq_jobs_Ai::ActiveContext::MigrationWorker true --ignore-feature-flag-consistency-check

To un-pause/restart, simply delete the drop-job Feature Flag:

/chatops gitlab run feature delete drop_sidekiq_jobs_Ai::ActiveContext::MigrationWorker --ignore-feature-flag-consistency-check

This pauses workers that have operations that access the vector store, ensuring that we avoid data loss during maintenance tasks in the vector store (like upgrades).

ActiveContext indexing is paused when either of the following settings is true:

• active_context_pause_indexing — dedicated ActiveContext pause setting (added in gitlab-org/gitlab!229832)
• elasticsearch_pause_indexing — global legacy setting (also pauses Advanced Search)

To pause only ActiveContext indexing without affecting Advanced Search, use the Admin API:

curl --request PUT \
  --header "PRIVATE-TOKEN: <your_token>" \
  --data "active_context_pause_indexing=true" \
  "https://gitlab.com/api/v4/application/settings"

Or via Rails console:

ApplicationSetting.current.update!(active_context_pause_indexing: true)

To resume:

ApplicationSetting.current.update!(active_context_pause_indexing: false)

You can verify the current state with:

Gitlab::CurrentSettings.active_context_indexing_paused?

Note: if elasticsearch_pause_indexing is also true, ActiveContext indexing will remain paused even if active_context_pause_indexing is false. To pause both ActiveContext and Advanced Search indexing at once, see the Advanced Search runbook.

You can toggle ActiveContext indexing by deactivating or activating the relevant Ai::ActiveContext::Connection.

To deactivate the active connection

This stops all ActiveContext-related workers.

⚠️ WARNING: This is a destructive action that will require a full reindex and should be used as a last resort. Pause indexing is the preferred method to use during incidents or maintenance.

Ai::ActiveContext::Connection.active.deactivate!

To reactivate a connection

⚠️ WARNING: if there is already an existing active connection, this will deactivate that other connection.

conn = Ai::ActiveContext::Connection.find_by(name: 'elastic')
conn.activate!

• Code Indexing and Embeddings pipeline

These are the same logs from the dashboard visualizations.

Code: Index State Management

• SaasInitialIndexingEventWorker - This worker marks namespaces as ready for ad-hoc indexing.
• ProcessPendingEnabledNamespaceEventWorker - This worker is not run in the regular pipeline with ad-hoc indexing. It processes pending namespaces, preparing the projects in the namespace for initial indexing.
• AdHocIndexingWorker - This worker kicks off initial indexing for a project ad-hoc. It is triggered on the first time a semantic search is performed on a project.
• MarkRepositoryAsReadyEventWorker - Once Initial Indexing is completed, this worker marks a project as ready. This means that the project is ready for subsequent Incremental Indexing after pushes or merges to the default branch.
• RepositoryIndexWorker - This worker executes indexing per repository. This includes both INITIAL and INCREMENTAL indexing.
  • Initial Indexing Service - triggered by the ProcessPendingEnabledNamespaceEventWorker for eligible projects
  • Incremental Indexing Service - after a project has been initially indexed and marked as ready, triggered after commits are pushed or merged to the default branch
  • Code Indexer - this is the actual class that calls the gitlab-elasticsearch-indexer

Code: Embeddings Generation

• BulkProcessWorker - This will allow you to trace the embeddings generation process.
  • Jobs - Tracks the start and finish of the bulk processing jobs
  • Embeddings Generation requests - Tracks the actual call to the embeddings generation model (Gitlab::Llm::VertexAi::Client)
  • Error log - Errors encountered during the bulk embeddings generation process
    • ContentNotFoundError - logged as a warning and the document is skipped
    • Other errors - logged as a warning and the documents are re-queued for processing

These are AI Gateway logs and dashboards that are relevant to ActiveContext Code Embeddings pipeline.

• Log: embeddings generations for Vertex AI text-embedding-005
• Dashboard: Vertex AI GCP Quota