Gitlab Pipeline Templates

For a while, I have built custom pipelines for my Gitlab projects, each starting from scratch and facing the same issues.

With the introduction of a new platform for running services, I needed to:

• Use custom versions of Terraform
• Inject CA certificates into each container
• Use a replacement docker registry, which now used authentication, provided by Vault
• Authenticate to vault for Terraform

I created a test application deployment, using base docker images, which resulted in more boiler-plate code than actual deployment logic, so I tested out the Official Gitlab Terraform templates (https://gitlab.com/gitlab-org/gitlab-foss/-/tree/master/lib/gitlab/ci/templates/Terraform).

This reduced the code a little, but still resulted in:

include:
  - template: Terraform/Base.gitlab-ci.yml  # https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Terraform/Base.gitlab-ci.yml
  - template: Jobs/SAST-IaC.gitlab-ci.yml   # https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml

variables:
  TF_ROOT: infrastructure/environments/auth
  TF_VAR_docker_image: "fare-docker-reg.mydomain:5000/nomad-test-image:v${CI_COMMIT_SHORT_SHA}"
  VAULT_ROLE: auth-test-app

stages:
  - validate
  - test
  - build
  - deploy
  - cleanup

fmt:
  extends: .terraform:fmt
  needs: []

validate:
  extends: .terraform:validate
  needs: []
  variables:
    VAULT_ADDR: https://vault.mydomain:8200
    no_proxy: ".mydomain,.dockstudios.co.uk"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.mydomain:8200
  before_script:
   # Update CA certs
   - echo 'dockstudios.crt' >> /etc/ca-certificates.conf
   - update-ca-certificates

   # Install vault
   # Unset HTTP proxy manually, as busybox might not support no_proxy
   - http_proxy= wget "http://archive.mydomain/hashicorp/vault_1.16.1_linux_amd64.zip"
   - unzip vault_1.16.1_linux_amd64.zip
   # Get public key
   # Vault login
   - export VAULT_TOKEN="$(./vault write -field=token auth/gitlab_jwt/login role=$VAULT_ROLE jwt=$VAULT_ID_TOKEN)"
   # Get AWS credentials
   - export AWS_ACCESS_KEY_ID=$(./vault kv get -mount=deployment_secrets_kv -field=access_key terraform/s3_credentials)
   - export AWS_SECRET_ACCESS_KEY=$(./vault kv get -mount=deployment_secrets_kv -field=secret_key terraform/s3_credentials)

build_image:
  stage: build
  image: docker:latest
  variables:
    http_proxy: "http://myproxy.mydomain:3128"
    https_proxy: "http://myproxy.mydomain:3128"
    HTTP_PROXY: "http://myproxy.mydomain:3128"
    HTTPS_PROXY: "http://myproxy.mydomain:3128"
  script:
    - docker build -f Dockerfile -t $TF_VAR_docker_image --build-arg http_proxy=$http_proxy --build-arg HTTP_PROXY=$http_proxy --build-arg https_proxy=$http_proxy --build-arg HTTPS_PROXY=$http_proxy .
    - docker push $TF_VAR_docker_image

build_ext-sec-gk:
  variables:
    VAULT_ADDR: https://vault.mydomain:8200
    no_proxy: ".mydomain,.dockstudios.co.uk"
    TF_STATE_NAME: "ext-sec-gk"
  needs:
    - build_image
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.mydomain:8200
  before_script:
   # Update CA certs
   - echo 'dockstudios.crt' >> /etc/ca-certificates.conf
   - update-ca-certificates

   # Install vault
   # Unset HTTP proxy manually, as busybox might not support no_proxy
   - http_proxy= wget "http://archive.mydomain/hashicorp/vault_1.16.1_linux_amd64.zip"
   - unzip vault_1.16.1_linux_amd64.zip
   # Get public key
   # Vault login
   - export VAULT_TOKEN="$(./vault write -field=token auth/gitlab_jwt/login role=$VAULT_ROLE jwt=$VAULT_ID_TOKEN)"
   # Get AWS credentials
   - export AWS_ACCESS_KEY_ID=$(./vault kv get -mount=deployment_secrets_kv -field=access_key terraform/s3_credentials)
   - export AWS_SECRET_ACCESS_KEY=$(./vault kv get -mount=deployment_secrets_kv -field=secret_key terraform/s3_credentials)
  extends: .terraform:build
  environment:
    name: $TF_STATE_NAME
    action: prepare

deploy_ext-sec-gk:
  variables:
    VAULT_ADDR: https://vault.mydomain:8200
    no_proxy: ".mydomain,.dockstudios.co.uk"
    TF_STATE_NAME: "ext-sec-gk"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.mydomain:8200
  before_script:
   # Update CA certs
   - echo 'dockstudios.crt' >> /etc/ca-certificates.conf
   - update-ca-certificates

   # Install vault
   # Unset HTTP proxy manually, as busybox might not support no_proxy
   - http_proxy= wget "http://archive.mydomain/hashicorp/vault_1.16.1_linux_amd64.zip"
   - unzip vault_1.16.1_linux_amd64.zip
   # Get public key
   # Vault login
   - export VAULT_TOKEN="$(./vault write -field=token auth/gitlab_jwt/login role=$VAULT_ROLE jwt=$VAULT_ID_TOKEN)"
   # Get AWS credentials
   - export AWS_ACCESS_KEY_ID=$(./vault kv get -mount=deployment_secrets_kv -field=access_key terraform/s3_credentials)
   - export AWS_SECRET_ACCESS_KEY=$(./vault kv get -mount=deployment_secrets_kv -field=secret_key terraform/s3_credentials)
  extends: .terraform:deploy
  script:
    - gitlab-terraform plan
    - gitlab-terraform plan-json
    - gitlab-terraform apply
  dependencies:
    - build_ext-sec-gk
  environment:
    name: $TF_STATE_NAME
    action: start

Another side affect of using the template is that Gitlab has now deprecated these pipelines, and will likely be removed, due to the license changes to Terraform (I won’t link to these, as this probably isn’t news to anyone).

So I created a new set of custom pipeline templates, which provide:

• Custom templates to be used in application deployment
• Custom docker images that contain Vault pre-installed and the custom CA certificates that I require
• A pipeline to build the custom deployment images

Custom pipeline templates

The custom pipelines themselves, were fairly simple, basically:

• Copying the functionality from the upstream Gitlab template and remove the deprecation warnings.

• Create child templates that would use the replicated template and provide the additional custom steps

  • One pipeline for Terraform

  include:

  • project: 'dockstudios/gitlab-templates' ref: main file: '/Terraform/Base.gitlab-ci.yml'

  variables: TF_VERSION: "1.8.2" TF_IMAGE: "harbor.mydomain/gitlab-templates/terraform:${TF_VERSION}-latest"

  .auth: variables: VAULT_ADDR: https://vault.mydomain:8200 no_proxy: ".mydomain,.dockstudios.co.uk" id_tokens: VAULT_ID_TOKEN: aud: https://vault.mydomain:8200 before_script:

  Vault login

  • export VAULT_TOKEN="$(vault write -field=token auth/gitlab_jwt/login role=$VAULT_ROLE jwt=$VAULT_ID_TOKEN)"

  Get AWS credentials

  • export AWS_ACCESS_KEY_ID=$(vault kv get -mount=deployment_secrets_kv -field=access_key terraform/s3_credentials)
  • export AWS_SECRET_ACCESS_KEY=$(vault kv get -mount=deployment_secrets_kv -field=secret_key terraform/s3_credentials)
  • mkdir ~/.aws
  • echo -e "[profile dockstudios-terraform]\nregion = main\noutput = json\nendpoint_url = https://s3.mydomain:9001\n" > ~/.aws/config
  • echo -e "[dockstudios-terraform]\naws_access_key_id=${AWS_ACCESS_KEY_ID}\naws_secret_access_key=${AWS_SECRET_ACCESS_KEY}\n" > ~/.aws/credentials
  • echo -e 'provider_installation {\n network_mirror {\n url = "https://harbor.mydomain:9443/mirror/v1/"\n }\n}' > ~/.terraformrc

  .terraform_fmt: extends: .terraform:fmt image: $TF_IMAGE needs: []

  .terraform_validate: extends:

  • .terraform:validate
  • .auth needs: [] image: $TF_IMAGE

  .terraform_plan: needs: [] image: $TF_IMAGE extends:

  • .terraform:build
  • .auth environment: name: $TF_STATE_NAME action: prepare

  .terraform_deploy: image: $TF_IMAGE extends:

  • .terraform:deploy
  • .auth script: - gitlab-terraform plan - gitlab-terraform plan-json - gitlab-terraform apply environment: name: $TF_STATE_NAME action: start

And another for docker:

variables:
  DOCKER_VERSION: "26.1.0"

.docker_build:
  variables:
    VAULT_ADDR: "https://myvault.com"
    no_proxy: ".mydomain,dind"
    NO_PROXY: ".mydomain,dind"
    HTTP_PROXY: "http://myproxy:8888"
    HTTPS_PROXY: "http://myproxy:8888"
    DOCKER_BUILDKIT: 0
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://myvault.com
  before_script:
   # Vault login
   - export VAULT_TOKEN="$(vault write -field=token auth/gitlab_jwt/login role=$VAULT_ROLE jwt=$VAULT_ID_TOKEN)"
   # Setup access to dind
   - export no_proxy="$no_proxy,dind,172.16.0.0/12"
   - export NO_PROXY=$no_proxy
   - export DOCKER_HOST="tcp://${DIND_PORT_2375_TCP_ADDR}:2375"
   - export DOCKER_TLS_CERTDIR=""
  image: harbor.mydomain/gitlab-templates/docker:${DOCKER_VERSION}-latest
  stage: build
  services:
    - name: harbor.mydomain/gitlab-templates/docker:${DOCKER_VERSION}-latest
      alias: dind
      variables:
        DOCKER_HOST: "unix:///var/run/dind.sock"
        DOCKER_TLS_CERTDIR: ""

Building custom images

The custom images were also very easy to create…

A simple set of docker files:

Dockerfile.docker
ARG DOCKER_VERSION
FROM harbor.mydomain/library/docker:${DOCKER_VERSION}-dind

ARG http_proxy
ARG no_proxy
ARG VAULT_VERSION

env http_proxy=$http_proxy
env https_proxy=$http_proxy
env HTTP_PROXY=$http_proxy
env HTTPS_PROXY=$http_proxy
env no_proxy=$no_proxy
env NO_PROXY=$no_proxy

RUN apk add jq

# Setup root CA cert
COPY dockstudios.crt /usr/share/ca-certificates/dockstudios.crt
RUN echo 'dockstudios.crt' >> /etc/ca-certificates.conf && \
    update-ca-certificates

# Install vault
# Unset HTTP proxy manually, as busybox might not support no_proxy
RUN http_proxy= wget "http://archive.mydomain/hashicorp/vault_${VAULT_VERSION}_linux_amd64.zip" && \
    unzip vault_${VAULT_VERSION}_linux_amd64.zip && \
    mv ./vault /usr/local/bin/ && \
    rm vault_${VAULT_VERSION}_linux_amd64.zip

env http_proxy=
env https_proxy=
env HTTP_PROXY=
env HTTPS_PROXY=
env no_proxy=
env NO_PROXY=


Dockerfile.terraform
ARG BASE_IMAGE

FROM harbor.mydomain/library/alpine:3.19.1

ARG TERRAFORM_BINARY_VERSION
ARG http_proxy
ARG no_proxy
ARG VAULT_VERSION

env http_proxy=$http_proxy
env https_proxy=$http_proxy
env HTTP_PROXY=$http_proxy
env HTTPS_PROXY=$http_proxy
env no_proxy=$no_proxy
env NO_PROXY=$no_proxy

RUN apk add --no-cache \
  curl \
  gcompat \
  git \
  idn2-utils \
  jq \
  openssh \
  jq

WORKDIR /tmp

# Setup root CA cert
COPY dockstudios.crt /usr/share/ca-certificates/dockstudios.crt
RUN echo 'dockstudios.crt' >> /etc/ca-certificates.conf && \
    update-ca-certificates

# Install vault
# Unset HTTP proxy manually, as busybox might not support no_proxy
RUN http_proxy= wget "http://archive.mydomain/hashicorp/vault_${VAULT_VERSION}_linux_amd64.zip" && \
    unzip vault_${VAULT_VERSION}_linux_amd64.zip && \
    mv ./vault /usr/local/bin/ && \
    rm vault_${VAULT_VERSION}_linux_amd64.zip

RUN ( curl -sLo terraform.zip "http://archive.mydomain/hashicorp/terraform-releases/${TERRAFORM_BINARY_VERSION}/terraform_${TERRAFORM_BINARY_VERSION}_linux_amd64.zip" && \
      unzip terraform.zip && \
      rm terraform.zip && \
      mv ./terraform /usr/local/bin/terraform \
    ) && terraform --version

env http_proxy=
env https_proxy=
env HTTP_PROXY=
env HTTPS_PROXY=
env no_proxy=
env NO_PROXY=

WORKDIR /

COPY src/bin/gitlab-terraform.sh /usr/bin/gitlab-terraform
RUN chmod +x /usr/bin/gitlab-terraform

# Override ENTRYPOINT since hashicorp/terraform uses `terraform`
ENTRYPOINT []

From here a small script to build both of the docker images:

#!/bin/bash

set -e

DOCKER_VERSION=$1
BUILD_NUMBER=$2

if [ "$DOCKER_VERSION" == "" ]
then
  echo "Usage: build_docker.sh <docker version>"
  exit 1
fi

if [ "$BUILD_NUMBER" == "" ]
then
  BUILD_NUMBER=$CI_JOB_ID
fi

if [ "$BUILD_NUMBER" == "" ]
then
  echo "Not running in CI - cannot find CI_JOB_ID and BUILD_NUMBER not passed"
  echo "Usage: build_docker.sh <docker version> <build number>"
  exit 1
fi

if [ "$VAULT_VERSION" == "" ]
then
  echo "Set VAULT_VERSION environment variable"
  exit 1
fi

# Get auth details for docker registry
DOCKER_DETAILS=$(./vault kv get -mount=deployment_secrets_kv -field=docker_registry -format=json pipelines/dockstudios/gitlab-templates)
# Perform docker login
echo $DOCKER_DETAILS | jq -r '.password' | docker login --password-stdin --username="$(echo $DOCKER_DETAILS | jq -r '.username')" "$(echo $DOCKER_DETAILS | jq -r '.registry')"

# Generate docker base docker image name
DOCKER_IMAGE="$(echo $DOCKER_DETAILS | jq -r '.registry')/$(echo $DOCKER_DETAILS | jq -r '.project')/docker"

if [ "${DOCKER_IMAGE}" == "" ]
then
  echo DOCKER_IMAGE env variable must be set
  exit 1
fi

DOCKER_TAG="$DOCKER_IMAGE:${DOCKER_VERSION}-${BUILD_NUMBER}"
LATEST_TAG="$DOCKER_IMAGE:${DOCKER_VERSION}-latest"
# Attempt to pull pre-existing image to use as build-cache
docker pull $LATEST_TAG || true

docker build \
  --cache-from=$LATEST_TAG \
  --build-arg=DOCKER_VERSION=${DOCKER_VERSION} \
  --build-arg=http_proxy=$HTTP_PROXY \
  --build-arg=no_proxy=$NO_PROXY \
  --build-arg=VAULT_VERSION=$VAULT_VERSION \
  -t $DOCKER_TAG \
  -f Dockerfile.docker \
  .

docker push $DOCKER_TAG

# Update latest tag
docker rmi $LATEST_TAG || true
docker tag $DOCKER_TAG $LATEST_TAG
docker push $LATEST_TAG

docker rmi $LATEST_TAG $DOCKER_TAG

A small Gitlab job to kick off this script was created (unfortunately, with a fairly similar amount of boiler-plate, since it couldn’t use itself as a template :D )

Result

Combining the use of the templates and the docker images, I was able to trim down the application deployment Gitlab pipelines to something much more reasonable:

include:
  - template: Jobs/SAST-IaC.gitlab-ci.yml   # https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml
  - project: 'dockstudios/gitlab-templates'
    ref: main
    file: '/terraform.yml'
  - project: 'dockstudios/gitlab-templates'
    ref: main
    file: '/docker.yml'

stages:
  - validate
  - test
  - build
  - deploy
  - cleanup

fmt:
  extends: .terraform_fmt

.service:
  variables:
    VAULT_ROLE: auth-test-app
    TF_STATE_NAME: service
    TF_ROOT: infrastructure/environments/prod

build_image:
  # Use the official docker image.
  extends:
   - .docker_build
   - .service
  when: manual
  # Set allow failure to allow this to optionally be run for the remainder of the pieline to work
  allow_failure: true
  script:
   - sh ./build.sh

validate_service:
  extends:
   - .terraform_validate
   - .service

plan_service:
  extends:
   - .terraform_plan
   - .service

deploy_service:
  extends:
   - .terraform_deploy
   - .service
  dependencies:
   - plan_service

Conclusion

This consolidation of deployment configuration into a single template and Docker image has quickly had many benefits:

• reduce noise in the individual pipelines means that they are easier to read
• fewer issues when deploying a new application
• changes can be made centrally which take immmediate affect
• reduced reliance on upstream templates and docker images

This has lead to quickly creating pipelines whilst attempting to move applications to a new platform.

If you’d like to see more about my Hashicorp setup with Vault, Consul and Nomad, see https://github.com/MatthewJohn/vault-nomad-consul-terraform