Windows runners

The Windows shards run jobs on ephemeral Windows VMs in the GCP project gitlab-ci-windows. The managers are Linux pods on the Kubernetes runner manager clusters. Everything on that page about access, logs, deploys and rollbacks applies here. The difference from the Linux fleet is the executor. Instead of docker+machine, the runner uses the custom executor with the autoscaler driver, which creates a VM per job through the GCE API and runs the job scripts on it over WinRM.

Windows has its own ApplicationSet instead of a directory under the Linux gitlab-runner service because its readiness check boots a real Windows VM and takes minutes. Windows and Linux rollouts do not wait for each other. The sync order is gstg-ci, then gprd-ci.

1. The pod polls GitLab with the runner token from Vault, like every other k8s manager.
2. For each job the runner calls the driver’s prepare step. The driver creates a VM from the image pinned in autoscaler.toml, in the executor-subnet of the windows-ci network, in one of us-east1-b, c or d, with an external IP and the windows-executor tag. VM names contain the shard’s VMTag (wsrm, w21h2sm, w1809prm, w21h1prm).
3. The driver waits for WinRM (5985/5986) on the VM’s external IP. Manager traffic leaves the cluster through the per-environment Cloud NAT. The runners-manager-to-executors firewall rule in windows-ci allows only those NAT addresses, so new NAT addresses have to be added to that rule.
4. The job scripts run over WinRM in PowerShell. Artifacts and cache go through the runner helper on the VM. The gprd m shard uses the gitlab-com-windows-runner-cache bucket.
5. The driver’s cleanup step deletes the VM.

VMs that outlive their job (driver crash, pod eviction) are deleted by the reaper-windows instance of the reaper. The autoscaler driver does not set the heartbeat label the Linux VMs have, so the heartbeat rule never matches. The only rule is max_age: any VM in gitlab-ci-windows older than 24 h is deleted. The longest Windows job is 4 h.

Every pod runs a boot_verify canary before it reports ready: it creates one VM, waits for WinRM, runs a job on it and deletes it. The startupProbe allows 15 minutes because a cold Windows boot plus the WinRM handshake takes about 10. A pod that remains at 0/1 Running has a VM provisioning or connectivity problem. Read its logs first. Both environments also set verify_auth, so the canary checks the runner token.

A ready pod can create VMs and run a job on them. That says nothing about fetching jobs. If a ready manager runs nothing, check the logs for job request errors.

Images are pinned per shard in autoscaler.toml. To roll a new windows-containers build:

1. Wait for the windows-containers pipeline to publish the image into gitlab-ci-windows. gcloud compute images list --project gitlab-ci-windows --no-standard-images lists it.
2. Change the pin in the gstg m shard, merge, and let the gstg pods roll. Each new pod boot-verifies the new image. A broken image shows up as pods that never become ready while the old pods still run jobs.
3. Repeat for the gprd shards.

Reverting is the same change with the old image name.

Both come from the manager image. Renovate opens MRs on ci-images for the runner base image and the autoscaler version, then on argocd/apps for the image tag in the environment values. There is no in-place upgrade. The Deployment rolls new pods, each one boot-verifies, and the old pods finish their jobs within the termination grace period of 2h05.

Manager side, as for any k8s manager:

glsh kube use-cluster runner-managers-gprd-1
kubectl get pods -n gitlab-runner -l app=gitlab-runner-windows-m
kubectl logs -n gitlab-runner deploy/gitlab-runner-windows-m --since=1h

The driver writes its own log inside the pod, with the GCE calls, the WinRM connection attempts and the per-job VM names:

POD=$(kubectl get pods -n gitlab-runner -l app=gitlab-runner-windows-m -o name | head -1)
kubectl exec -n gitlab-runner ${POD#pod/} -- tail -100 /tmp/autoscaler/log/autoscaler.log

Executor side, from a workstation with access to gitlab-ci-windows:

gcloud compute instances list --project gitlab-ci-windows --filter='tags.items=windows-executor'
gcloud compute instances get-serial-port-output <vm> --zone <zone> --project gitlab-ci-windows

The serial console shows the Windows boot and whether WinRM came up.

The only inbound rule for the windows-executor tag is WinRM from the manager NAT addresses, so commands on a VM go through a manager pod. The autoscaler binary in the pod is the WinRM client: its run stage sends a PowerShell script to the VM named in AUTOSCALER_VM_NAME, with the credentials from the VM’s metadata. Any pod of the shard can do this for any of the shard’s VMs. The three other variables are what the runner would set during a job. Without them the binary exits 0 with no output.

kubectl exec -it -n gitlab-runner deploy/gitlab-runner-windows-m -- sh

# inside the pod
printf 'Get-Service WinRM\nGet-ChildItem C:\\GitLab-Runner\\builds\n' > /tmp/debug.ps1
echo '{"id":0}' > /tmp/job.json
BUILD_FAILURE_EXIT_CODE=1 SYSTEM_FAILURE_EXIT_CODE=2 JOB_RESPONSE_FILE=/tmp/job.json \
  AUTOSCALER_VM_NAME=runner-znnc-lwgg-w21h2sm-2ea20540a129bbee079e \
  autoscaler --config /configmaps/autoscaler.toml custom run /tmp/debug.ps1 debug

The script’s output is printed, the driver’s own lines are in its log file. The port and service checks before the script are the same ones a job goes through, so a failure here is the failure the job would see. The VM has to exist: to look at one outside a failing job, run a job on the shard with a Start-Sleep in the script and use its VM name from the job log.

• Pods at 0/1 Running after a deploy: boot_verify is failing. Usually a bad image pin, a firewall or NAT change (new NAT addresses missing from runners-manager-to-executors), or a GCE quota in gitlab-ci-windows. The old pods run jobs until the new ReplicaSet is ready, so nothing is down yet.
• Jobs fail in prepare with a GCE permission error: the manager service account (runner-manager-k8s-<env>@gitlab-ci-155816) is bound to the runnerManager role in the windows-ci environment of config-mgmt, plus iam.serviceAccountUser on the default compute service account the VMs run as.
• WinRM timeouts on every job: check the serial console of a fresh VM for the image’s WinRM setup, then the firewall rule.
• Many runner-* VMs older than a few hours: the reaper is not sweeping, see ReaperNotSweeping.