Binary Artifact Hosting Fun

To kick this off, I’ve not had a good way to host binaries generated during CI/CD. At the moment, I have:

• Gitlab itself - I avoid using this for any binaries/OCI images mostly because the thing is bloaty enough and I do try to have complete backups without added hundreds of GBs of binaries to it.
• Sonartype Nexus - This is used for apt/yum caching and some scripted public binary mirroring
• Minio - currently used for Terraform state, Terraform provider binary mirroring (Terrashine) and some public certificates

Given I’ve written Gitlab off, I’m left with bulky Nexus and now-unsupported Minio, so I wanted to try some different technologies and see what I could come up with

Options

That detail actually tightens the architecture a lot, because you’ve now clearly separated:

My requirements were quite hard set:

• JWT-like authentication, authenticating via Vault with claim-based policies
• Terraform provider support (I mean, I’d be willing to write one if the tool is that good)
• HTTP-based access (general pulling public artifacts)
• S3-based access (for file uploads and perhaps easiest for downloading private artifacts)

Feature	MinIO	Sonatype Nexus Repository	RustFS	Garage	SeaweedFS	Artifact Keeper
Vault JWT/OIDC integration (auth issuance)	🟢 (excellent STS mapping)	🔴 Supports LDAP and robot accounts	Support OIDC	🔴 (needs proxy)	🔴	🟢
Nomad artifact pull experience	🟢 (direct HTTP GET)	🟢	🟢	🟢	🟢	🟢
Terraform support	🟢 (AWS or Minio provider)	🟢	3rd-party Provider with a handful of resources	🔴	🔴	🔴
External DB required	❌	✅	❌	❌	❌	⚠️ PostgreSQL
Operational complexity	🟢 Low	🔴 High	🟢 Low	🟢 Very low	🟡 Medium	🟡 Medium–High
Artifact versioning semantics	🟢	🟢	🔴	🔴	🔴	🟢
JWT-first machine auth model	🟢	🔴	🟢	🔴	🔴	🟢
S3 compatibility (relevant for CI tools)	🟢	🟢	🟢	🟢	🟢	🟢

Based on this, Sonatype could be useful (it’s heavier and not the ideal, but I do already use it and the apt/yum (etc.) requirements won’t go away, so it’d still be here to stay meaning adding another technology.

The main caveat is that it only supports LDAP (or could create per-application write and read robot accounts).

So I decided to do two tests:

• Test RustFS using Vault as an OIDC provider
• Attempt to create a Vault plugin that provides a mock LDAP server as a secrets engine

Testing RustFS

After spinning up RustFS and Vault locally (if it doesn’t require much setup, this is much easier than setting it up in homelab with new VMs, firewall rules, ansible blah blah).

The Vault OIDC provider was easy to setup and RustFS was easy to point at it, however, authenticating using the Vault root token, I received:

missing 'code' query parameter

but this was actually due to an OIDC error thrown (RustFS didnt' pick up the error and just complained that it didn’t receive a code :facepalm:):

identity entity must be associated with the request

This is interesting - I’m not sure what I thought would happen when authenticating as root, but as least some basic identity. I presume this means that I can only use it if I’m authenticated to some identity-based auth engine (JWT etc?) - I wonder how this would work with pure Vault tokens - I presume fail(?!).

So, going further into an annoying local setup (I might need to start using Terraform!)… https://github.com/sbstjn/jwkserve to the rescue.. quick and easy JWT authentication:

podman run -p 3000:3000  sbstjn/jwkserve:latest
vault auth enable jwt
vault write auth/jwt/config \
    jwks_url="http://10.88.0.9:3000/.well-known/jwks.json" \
    jwt_supported_algs="RS256"
vault write auth/jwt/role/test-role \
  role_type=jwt \
  user_claim=sub \
  bound_subject="1234567890" \
  policies=default

Then test:

vault write auth/jwt/login role="test-role" jwt="eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Im1uWV9RT1RjN1pwRV9ndEZKNlNiS0NXenpzUEoyck1MYTdsTlkyOU54MVktUlMyNTYifQ.eyJleHAiOjE3ODI2NzM3NjMsImlhdCI6MTc4MjY3MTk2MywiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDozMDAwIiwibmFtZSI6IkpvaG4gRG9lIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.hlWWzpvAhEZbAhtQIwx14SNaWeXXYFPovB-cT4cOwCnVGPXPTNUY1CkwFeWBn1eLQ46nTiO3R5-wo-Z4tEYAexp4YnJILVrmwn_BLjwPaFRk1hVfiN1QKDtW1icl5guTobyH1qhvDzCed8v23LgI1I5_Gcdw8-milvj8bhsigKGjp_I97n7hk-77CYGuIBqNM8-Q3FLYwg-VYtjFj7xxU2_w2YELa23XaNr5-etryS6xkhK3gkGD1k2BjAKMcWySl3vyMifvEk8l8fgNq63ldUTdoO7nIr1gfK5HRaCd6g5-xXCcAm1NTA1LQOwG39QX0mzIQuMYv9oV--nxpuEMcA"
Key                  Value
---                  -----
token                hvs.CAESIENvpwBWyyAKmTLVF-9ZUNPWROQROpqYjfUwAb_ojztvGh4KHGh2cy56WU52OVYwd0NJWXJJWDQxOXJ2YmlQRVM
token_accessor       nGEPLRN7IPVFcqTZm4C0tDWp
token_duration       768h
token_renewable      true
token_policies       ["default"]
identity_policies    []
policies             ["default"]
token_meta_role      test-role

(oh my, a token, please use this expired token to authenticate to a vault instance that no longer exist on my laptop that’s turned off! Guaranteed I get some email from some heroic bot trying to do me a favour)

And tada, using the token during the auth flow, I was able to login to RustFS (granted I got a big fat “Access Denied”, but it worked)

It seems that the username used by RustFS is infact the Vault entity ID, not the sub from the JWT. Looking at the vault entity, it does have the name based on the sub.

So let’s mock the OIDC flow and see what we get:

import requests
import urllib.parse
import base64
import json
import jwt  # pip install pyjwt

ISSUER = "http://127.0.0.1:1234/v1/identity/oidc/provider/default"
AUTH_ENDPOINT = "http://127.0.0.1:1234/ui/vault/identity/oidc/provider/default/authorize"
TOKEN_ENDPOINT = "http://127.0.0.1:1234/v1/identity/oidc/provider/default/token"

CLIENT_ID = "Qh17xeGsksrrIGJ5pr4R6jOUh0IDbAz5"
CLIENT_SECRET = "hvo_secret_b9aH1YfRELKNALvONexKbgpayfgDitNOLALnENayM2I55jTQ1uDr4E1VWRw957eP"
REDIRECT_URI = "http://localhost:9001/rustfs/admin/v3/oidc/callback/default"

SCOPE = "openid"
RESPONSE_TYPE = "code"


def build_auth_url():
    params = {
        "response_type": RESPONSE_TYPE,
        "client_id": CLIENT_ID,
        "redirect_uri": REDIRECT_URI,
        "scope": SCOPE,
    }
    url = AUTH_ENDPOINT + "?" + urllib.parse.urlencode(params)
    return url


def exchange_code(code):
    data = {
        "grant_type": "authorization_code",
        "code": code,
        "client_id": CLIENT_ID,
        "client_secret": CLIENT_SECRET,
        "redirect_uri": REDIRECT_URI,
    }

    r = requests.post(TOKEN_ENDPOINT, data=data)
    r.raise_for_status()
    return r.json()


def decode_jwt(token):
    header, payload, signature = token.split(".")
    payload += "=" * (-len(payload) % 4)
    decoded = json.loads(base64.urlsafe_b64decode(payload))
    return decoded


if __name__ == "__main__":
    print("\n=== Vault OIDC Test ===\n")

    auth_url = build_auth_url()

    print("1. Open this URL in your browser:\n")
    print(auth_url)
    print("\n2. Complete login (GitLab/JWT/etc.)")
    print("3. Copy the `code` from redirect URL\n")

    code = input("Paste authorization code: ").strip()

    print("\nExchanging code...\n")
    token_response = exchange_code(code)

    print("Token response:\n")
    print(json.dumps(token_response, indent=2))

    if "id_token" in token_response:
        print("\nDecoded ID Token (user identity):\n")
        print(json.dumps(decode_jwt(token_response["id_token"]), indent=2))

Looking at the result:

{
  "at_hash": "MkikFxaYSW28j8cvO6aAlw",
  "aud": "Qh17xeGsksrrIGJ5pr4R6jOUh0IDbAz5",
  "c_hash": "xO5WFXcacOzf27rWdOZd2A",
  "exp": 1782763835,
  "iat": 1782677435,
  "iss": "http://10.88.0.7:1234/v1/identity/oidc/provider/default",
  "namespace": "root",
  "sub": "a5537670-e420-f229-f970-befc91679243"
}

Comparing this to the JWT servers':

{
  "sub": "1234567890",
  "name": "John Doe",
  "iss": "http://localhost:3000",
  "iat": 1782707217,
  "exp": 1782709017
}

We can see the sub has been rewritten to a random entity ID from Vault

Let’s try mapping the original sub to groups:

vault write auth/jwt/role/test-role \
  role_type=jwt \
  user_claim=sub \
  groups_claim=sub \
  bound_subject="1234567890" \
  policies=default

Vault LDAP plugin