Cluster Triple - GitLab Runner

2020-01-26

In my previous post, I outline the Kubernetes setup steps. Next up, I'm going install GitLab Runner via the standard helm chart. Beware: This was much harder than I hoped it would be!

Goals

Install GitLab Runner on the new Kubernetes Cluster running on the Cluster Triple + Base Pi.
Add a job to a GitLab project that will be executed by our private runner.

GitLab Runner Registration Token

In order to register a new GitLab Runner requires you use a registration token that is from either a certain GitLab project or a GitLab group that you are an owner of. For a certain project or group, you you can navigate to Settings » CI/CD and expand the *Runners section. From there, make a note of the registration token.

Installing the Helm Chart

Installing Helm

If you don't already have Helm installed, check out the installtion options to get the helm CLI installed. I used Helm V3, although this should work fine with Helm V2 as well.

GitLab Chart Repo

GitLab's helm charts can be found in their own Helm repository, so first we'll need to add the repo to your local setup:

$ helm repo add gitlab https://charts.gitlab.io/
"gitlab" has been added to your repositories

Helm Chart Values

To install the GitLab helm chart, there are some required configuration values that need to be passed. The full values.yaml file that worked for me was the following:

image: petejohanson/gitlab-runner:latest
gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "m-FbJnAoZLeEf_q_3akE"
runners:
  tags: "arm"
  runUntagged: false
  helpers:
    image: gitlab/gitlab-runner-helper:arm-latest
rbac:
  create: true

Let's break down those various values:

image - The default gitlab-runner Docker images is for x86_64, not using the manifest behaviour that allows multi-platform images. See the PS at the end of this article for how this image was created, if you don't want to use the various I've built.
gitlabUrl - Since I'm registering this to the SAAS public GitLab instance, I'm using the normal GitLab URL.
runnerRegistrationToken - This is the registration token noted in the start of our process. This is a private/secret value.
runners.tags - Because I only want certain special jobs picked up by this runner, I've added the arm tag to the runner.
runners.runUntagged - Same.
runners.helpers.image - This makes sure that the helper container that runs along-side the job containers using the version with the correct platform (Note: It would be nice if GitLab published images/tags using a manifest that pointed to the different architecture images properly)
rbac.create - Create the custom cluster role as part of the chart, since the runner needs to make calls into the Kubernetes API.

Install The Chart

We'll install the gitlab-runner helm chart into a separate namespace, named gitlab-runner, so we'll install the helm chart with the following:

$ helm -n gitlab-runner install -f values.yaml gitlab-runner gitlab/gitlab-runner

Once that completes, you can check on the status of the running by listing the pods in that new gitlab-runner namespace:

$ kubectl get pods -n gitlab-runner
NAME                                                 READY   STATUS              RESTARTS   AGE
gitlab-runner-gitlab-runner-f7d9cb88b-bhxc5          1/1     Running             0          6m13s

Once the runner has a status of Running, you can refresh the CI/CD page in GitLab and see the runner registered, with a green icon to show it live/active.

Running a Job

Adding A Tag To A Job

Since we registered our runner to not pick up untagged jobs, and added an arm tag to our runner, to get a job picked up by the runner we'll need to add the same tag to a job in our project. The tags setting for a job is an array of tags required to exist on the runner. Here's an example job:

build:rust:arm:
  stage: build
  image: rust:1-slim
  tags:
    - arm

  script:
    - cargo build

Note: Your job must use an image that works on the ARM architecture. Most images in Docker Hub that are part of the "core library" are actually a manifest that points to different docker images for different architectures, so should work. Others, e.g. rustlang/nightly-slim are only built for x86_64.

Running

Once you push a commit with that job definition, you can monitor your cluster for a new pod starting to run the job:

$ kubectl get pods -n gitlab-runner
NAME                                                 READY   STATUS              RESTARTS   AGE
gitlab-runner-gitlab-runner-f7d9cb88b-bhxc5          1/1     Running             0          6m13s
runner-p1uajn7v-project-12786723-concurrent-09xbxs   0/2     ContainerCreating   0          37s

You can see an example of a job completed with the GitLab Runner on my Cluster Triple Kubernetes cluster.

Summary

Once I got the custom gitlab-runner Docker image built, the final setup was really easy. With this configured, it should be very easy to build binaries or even Docker images that target the ARM architecture as part of your standard CI/CD pipeline.

PS - Building the `gitlab-runner` Docker image

In order to make this work, I need to build the gitlab-runner Docker image, but for the arm architecture. To do this, I took the following steps:

Installed qemu-system-arm on my Linux laptop.
Used libvert/GNOME Boxes to create a Fedora 30 VM running on the ARM architecture.
Logged into a shell in the new ARM based Fedora VM.
Installed various Fedora packages for Go, buildah to build Docker images, etc.
Checked out the gitlab-runer
Ran make build_simple to build gitlab-runner for ARM. This took a not inconsiderable amount of time.
Built dumb-init and git-lfs from their respective git repos, since banries for ARM weren't handy.
Copied the binaries for dumb-init, git-lfs, and GitLab Runner from out/binaries/gitlab-runer into the dockerfiles/alpine/ directory from the gitlab-runner repository.
Edit the dockerfiles/alpine/Dockerfile to not fetch various files over HTTP, but copy from the local filesystem.
Run buildah bud --build-arg DOCKER_MACHINE_VERSION=0.16.2 -t petejohanson/gitlab-runner:latest .
Wait
Run buildah login docker.io to log in with my Docker Hub credentials.
Do buildah push -D petejohanson/gitlab-runner:latest to push the images.
Whew!