Windows runners

The Windows shards run jobs on ephemeral Windows VMs in the GCP project gitlab-ci-windows. The managers are Linux pods on the Kubernetes runner manager clusters. Everything on that page about access, logs, deploys and rollbacks applies here. The difference from the Linux fleet is the executor. Instead of docker+machine, the runner uses the custom executor with the autoscaler driver, which creates a VM per job through the GCE API and runs the job scripts on it over WinRM.

Windows has its own ApplicationSet instead of a directory under the Linux gitlab-runner service because its readiness check boots a real Windows VM and takes minutes. Windows and Linux rollouts do not wait for each other. The sync order is gstg-ci, then gprd-ci.

1. The pod polls GitLab with the runner token from Vault, like every other k8s manager.
2. For each job the runner calls the driver’s prepare step. The driver creates a VM in gitlab-ci-windows from the image pinned in autoscaler.toml, attached to the shard’s subnet in the gitlab-ci-155816 shared VPC (saas-w-m-amd64-0, 10.25.0.0/16, for the m shard; saas-w-p-amd64-0, 10.27.0.0/16, for gstg and the private shards), in one of us-east1-b, c or d, with the windows-executor tag. The VM also gets an external IP for egress, like the Linux ephemeral VMs; nothing inbound is allowed on it. VM names contain the shard’s VMTag (wsrm, w21h2sm, w1809prm, w21h1prm).
3. The driver waits for WinRM (5985/5986) on the VM’s internal IP, reached through the peering between the shard network and ci, where the manager pods run (ConnectExternalIP = false). The windows-executor-allow-runner-manager-<network> rule allows WinRM from the runner-managers subnet and the manager cluster pod ranges.
4. The job scripts run over WinRM in PowerShell. The clone goes to ci-gateway (clone_url), so it arrives from 10.0.0.0/8, the range globally allowed for groups with IP restrictions. Artifacts and cache go through the runner helper on the VM. The gprd m shard uses the gitlab-com-windows-runner-cache bucket.
5. The driver’s cleanup step deletes the VM.

VMs that outlive their job (driver crash, pod eviction) are deleted by the reaper-windows instance of the reaper. The autoscaler driver does not set the heartbeat label the Linux VMs have, so the heartbeat rule never matches. The only rule is max_age: any VM in gitlab-ci-windows older than 24 h is deleted. The longest Windows job is 4 h.

Every pod runs a boot_verify canary before it reports ready: it creates one VM, waits for WinRM, runs a job on it and deletes it. The startupProbe allows 15 minutes because a cold Windows boot plus the WinRM handshake takes about 10. A pod that remains at 0/1 Running has a VM provisioning or connectivity problem. Read its logs first. Both environments also set verify_auth, so the canary checks the runner token.

A ready pod can create VMs and run a job on them. That says nothing about fetching jobs. If a ready manager runs nothing, check the logs for job request errors.

Images are pinned per shard in autoscaler.toml. The windows-containers pipeline builds them on the shared k8s runners: the job gets packer-runner credentials through Workload Identity Federation, packer creates a build VM in gitlab-ci-windows and provisions it over an IAP tunnel to WinRM. publish is a manual job on main. There is no dedicated build host. To roll a new build:

1. Wait for the windows-containers pipeline to publish the image into gitlab-ci-windows. gcloud compute images list --project gitlab-ci-windows --no-standard-images lists it.
2. Change the pin in the gstg m shard, merge, and let the gstg pods roll. Each new pod boot-verifies the new image. A broken image shows up as pods that never become ready while the old pods still run jobs.
3. Repeat for the gprd shards.

Reverting is the same change with the old image name.

Both come from the manager image. Renovate opens MRs on ci-images for the runner base image and the autoscaler version, then on argocd/apps for the image tag in the environment values. There is no in-place upgrade. The Deployment rolls new pods, each one boot-verifies, and the old pods finish their jobs within the termination grace period of 2h05.

Manager side, as for any k8s manager:

glsh kube use-cluster runner-managers-gprd-1
kubectl get pods -n gitlab-runner -l app=gitlab-runner-windows-m
kubectl logs -n gitlab-runner deploy/gitlab-runner-windows-m --since=1h

The driver writes its own log inside the pod, with the GCE calls, the WinRM connection attempts and the per-job VM names:

POD=$(kubectl get pods -n gitlab-runner -l app=gitlab-runner-windows-m -o name | head -1)
kubectl exec -n gitlab-runner ${POD#pod/} -- tail -100 /tmp/autoscaler/log/autoscaler.log

Executor side, from a workstation with access to gitlab-ci-windows:

gcloud compute instances list --project gitlab-ci-windows --filter='tags.items=windows-executor'
gcloud compute instances get-serial-port-output <vm> --zone <zone> --project gitlab-ci-windows

The serial console shows the Windows boot and whether WinRM came up.

The only inbound rule for the windows-executor tag is WinRM from the manager subnet and pod ranges, so commands on a VM go through a manager pod. The autoscaler binary in the pod is the WinRM client: its run stage sends a PowerShell script to the VM named in AUTOSCALER_VM_NAME, with the credentials from the VM’s metadata. Any pod of the shard can do this for any of the shard’s VMs. The three other variables are what the runner would set during a job. Without them the binary exits 0 with no output.

kubectl exec -it -n gitlab-runner deploy/gitlab-runner-windows-m -- sh

# inside the pod
printf 'Get-Service WinRM\nGet-ChildItem C:\\GitLab-Runner\\builds\n' > /tmp/debug.ps1
echo '{"id":0}' > /tmp/job.json
BUILD_FAILURE_EXIT_CODE=1 SYSTEM_FAILURE_EXIT_CODE=2 JOB_RESPONSE_FILE=/tmp/job.json \
  AUTOSCALER_VM_NAME=runner-znnc-lwgg-w21h2sm-2ea20540a129bbee079e \
  autoscaler --config /configmaps/autoscaler.toml custom run /tmp/debug.ps1 debug

The script’s output is printed, the driver’s own lines are in its log file. The port and service checks before the script are the same ones a job goes through, so a failure here is the failure the job would see. The VM has to exist: to look at one outside a failing job, run a job on the shard with a Start-Sleep in the script and use its VM name from the job log.

• Pods at 0/1 Running after a deploy: boot_verify is failing. Usually a bad image pin, a firewall or peering change (the manager cluster pod range missing from windows-executor-allow-runner-manager-<network>, or a peering left INACTIVE because only one side declares it), or a GCE quota in gitlab-ci-windows. The old pods run jobs until the new ReplicaSet is ready, so nothing is down yet.
• Jobs fail in prepare with a GCE permission error: the manager service account (runner-manager-k8s-<env>@gitlab-ci-155816) is bound to the runnerManager role in the windows-ci environment of config-mgmt, plus iam.serviceAccountUser on the default compute service account the VMs run as. Attaching to the host-project subnet needs compute.subnetworks.use in gitlab-ci-155816, which the project-wide runnerManagerK8s role carries.
• WinRM timeouts on every job: check the serial console of a fresh VM for the image’s WinRM setup, then the firewall rule and the ci to shard network peering.
• Clones time out on ci-gateway port 8989: the gprd/gstg ci-gateway side of the peering is missing (ci-gateway-network-peers in config-mgmt) or the shard range is not in ci-gateway-allow-runners.
• Many runner-* VMs older than a few hours: the reaper is not sweeping, see ReaperNotSweeping.