· Blog - MattBits

blog.mattsbit.co.uk

🧾 1. INFRASTRUCTURE EVOLUTION TIMELINE (HIGH LEVEL)

Era	Approx timeframe	Key characteristics	Core systems
Early server era	(pre-~2007 → unknown)	First exposure to servers, NT domain	Compaq Windows NT server
Early homelab / ESX era	~2007–2013	Physical servers, early virtualization (RAM constrained)	DL380 G3 (x3), DL360 G3, ESX/vCenter, Whitebox NAS
Expansion / distributed experimentation	~2012–2016	Multi-server lab, PXE booting experiments, Jenkins (sheila)	x3455 servers, DL160 G6 (later), fileserver
VPS High-Availability Era	~2018	Multi-provider VPS cluster, Mesh VPN, HA MariaDB	Tinc, OpenVPN, Docker Swarm, HAProxy
Identity foundation era	~2016	First stable Linux identity system	FreeIPA (marge/homer)
Datacentre / colo era	~2017–2018	Formal cluster + SAN deployment	DL160 G6 cluster, Nexsan SAN
Consolidation era	~2018–2019	Transition (ThinkCentre) $\to$ R620/R720xd consolidation	ThinkCentre, R620, R720xd, Synology, Keycloak
Platform orchestration era	~2019–2024	Container orchestration, Rancher, early automation	GitLab (Rancher), Jenkins systems
Modern orchestration era	~2024–2026	Nomad-based unified service plane	Nomad, Consul, Vault, Keycloak migration

🧾 1.1 THE RANCHER/K8S ERA (RETIRED)

During this era, container orchestration was centralized through Rancher, managing Kubernetes clusters across the core nodes. This provided a GUI-driven approach to managing workloads but was eventually superseded by the more lightweight and unified Hashicorp stack (Nomad/Consul).

🛠️ Orchestration Stack

Platform: Rancher (RKE-managed Kubernetes)
Versions:
- naxos: v1.11.6-rancher1
- rhodes: v1.20.15-rancher1
CNI: canal (Calico + Flannel VXLAN)
Ingress: nginx-ingress-controller
Registry: Private registry at fare-docker-reg.dock.studios:5000
Identity: Keycloak-integrated OIDC/RBAC

📦 Workload Inventory by Node

`crete`

Core services and collaboration tools:

nextcloud
nexus
onlyoffice
perforce
seafile
terrareg

`naxos`

Development and monitoring tools:

cachet
freeipaselfservice
gitlab
mattermost
promailgate
sentry

`rhodes`

CI/CD and data services:

gitea
hsbc
int-jenkins
matomo
odot
oracle-neo4j
phabricator
sonarqube
wiki

🧾 1.5 LEGACY SERVICE STACKS (PRE-2012)

Service Category	Era/Tool	Notes
Email	`hMailserver` $\to$ `Zimbra` $\to$ `G-Suite`	Local `hMailserver` (~2011) $\to$ `Zimbra` $\to$ Cloud (`G-Suite`).
Web & App Hosting	`Kloxo` / Apache / IIS	Managed domains (`loveyourmusic.co.uk`, `mhsolutions.co.uk`, etc.). Hosted `eyeOS`, `phpMyAdmin`, and various custom apps via IIS/Apache.
Database Backend	MySQL	Hosted application data (`lymDB`, `subscribers`, `mailserver`, etc.).
File Storage	`FreeNAS 8.9` (`fileserver-01`)	Early storage via FreeNAS; provided iSCSI targets and Samba shares (`media$`, `downloads$`, `applications$`, etc.).

🧾 2. PHYSICAL / LEGACY SERVER HISTORY

System	Type	Role	Notes	Status
Packard Bell Club 30	Physical	First domain/server system	NT4-era	🔴
HP ProLiant 1U server	Physical	First dedicated server	Pre-virtualisation era	🔴
HP DL380 G3 (×3)	Physical	ESX cluster	VMware + vCenter (RAM constrained)	🔴
HP DL360 G3	Physical	Support compute	Same generation	🔴
Whitebox PC	Physical	File Server	TrueNAS/FreeNAS era	🔴
IBM x3455 (×2–4 approx)	Physical	Linux/KVM experimentation	Early oVirt exploration	🔴
Dell 2U server	Physical	Unused	Too loud / rejected	🔴
HP DL160 G6 (×3)	Physical	Datacentre compute	~Jan 2017 – Jan 2018 (Datacentre)	🔴
IBM ThinkCentre	Physical	VM Host	~May 2018 (Post-Datacentre)	🔴
ThinkPad	Physical	VM Host	~May 2018 (Post-Datacentre)	🔴
Dell R620 (×1–2)	Physical	Consolidation era compute	~June 2018 (Consolidation era)	🔴
hetznur01	Virtual	Former Hetzner node	Moved from cloud July 2018	🔴
VPS Cluster (Hetz/OVH)	Virtual	HA Web/DB services	Tinc, BIND, Consul, HAProxy, MariaDB	🟢
pfserver-04	Physical	Baddesley Fileserver	4 x 3TB drives	🟢
Nomad Cluster (tinos, etc)	Virtual	Current orchestration plane	Debian 12 (tinos), Ubuntu 18.04 (admts)	🟢

🧾 3. STORAGE & BACKUP EVOLUTION

Era/Phase	Technology	Role	Notes
Early Era	Local HDD	Primary storage	Local disks on servers
ESX Era	iSCSI (Home-built)	VM storage	VMware -> iSCSI
Laptop Era	iPXE + NFS	Boot/Root drives	Fileserver -> PXE -> Laptop servers
Datacentre Era	iSCSI (NEXSAN)	Enterprise SAN	oVirt -> iSCSI -> NEXSAN
Post-Datacentre	iSCSI (Synology)	VM storage backend	Libvirt -> iSCSI -> Synology
SSD Era (~2020)	Software RAID (SSD)	VM performance layer	Libvirt + SSD RAID (with LVM cache LVs) -> iSCSI -> Synology
ZFS Era (2026+)	ZFS + Enterprise HDD	High-perf/High-cap	Libvirt + ZFS (10K drives) + SSD L2ARC/SLOG -> iSCSI -> Synology

💾 Backup Strategy Evolution

Era	Method	Flow / Tools	Notes
Tape Era (~2011)	Symantec Backup Exec	Black server + LTO-2 Tape drive	Inherited server, found/cheap LTO-2 drive; physical tape rotation
Modern Tiered (Current)	Rsync + Cloud	`hans` -> `pfserver-04` -> Backblaze	`hans-solo` acts as MongoDB backup DB

🧾 4. NETWORKING EVOLUTION

System / Era	Type	Role	Notes	Status
Pre-2013 (Unmanaged)	Hardware	Basic connectivity	3× 10/100 3Com switches + Netgear Fibre switch (unmanaged)	🔴
2013 – Datacentre (Managed)	Hardware	Core switching	3× 3Com HP Baseline 2928-SFP Plus (VLAN/STP)	🔴
Interim Period	Hardware	Core switching	4× Stacked D-Link DGS-3324SR	🔴
Post-Datacentre	Hardware	Managed switching	Netgear 8-port managed switch	🔴
Tinc Mesh	Networking	Multi-provider VPN	Peer-to-peer mesh connectivity	🟢
OpenVPN	Networking	Home-lab interconnect	VPS ↔ Home-lab connection	🟢
Boundary	Networking	Zero-trust access	Identity-aware proxy/gateway	🟢

🧾 5. VIRTUALISATION / PLATFORM HISTORY

System	Type	Role	Notes	Status
VMware ESX (DL380 G3)	Virtualisation	First VM cluster	vCenter-managed	🔴
oVirt (DL160 G6 era)	Virtualisation	Datacentre cluster	Jan 2017 – Jan 2018 (Pre-R720xd)	🔴
ThinkCentre/ThinkPad Era	Virtualisation	VM Hosting	Pre-R720xd transition	🔴
HAProxy/Bind/Consul	Orchestration	HA VPS cluster	Multi-provider cluster	🟢
Rancher	Orchestration	Container platform	GitLab hosted here	🔴
Nomad	Orchestration	Current scheduler	Key migration target	🟢

🧾 6. CORE MODERN PLATFORM (R720XD ERA)

The R720xd serves as the primary consolidation host for the majority of the service plane.

🟢 Running VM Catalog (Categorized)

Category	Key Systems
Infrastructure & Network	`cosh-fw-1`, `cosh-host-5` (Libvirt + Auth API), `admts`, `apu`, `admin-vpn-interconnect`, `remote-interconnect`, `hetz-vpn-interconnect`, `init-connect-vpn-interconnect`, `netbird-gateway`, `inthetz`, `hetznur01`, `docker-man-1`
Identity & Security	`marge`, `homer`, `keycloak-1`, `albali` (Manual Vault), `vault` cluster (`lemon`, `orange`, `lime`), `crypt` (Nexus/Apt)
Consul Cluster	`banana`, `apple`, `cucumber`
Nomad Cluster	Servers: `raspberry`, `strawberry`, `blueberry`; Clients: `thassos`, `syros`, `gordon`, `kasos`, `anafi`, `printer` (Receipt API), `coder` (Coder dev environments), `zante` (Isolated Nomad)
CI/CD & Dev Platform	`fare-jenkins`, `pub-jenkins`, `ci-host-1`, `harbor`, `fare-docker-reg`
Application/Service Layer	`cartman`, `miram` (S3/Terrashine), `hans` (Rsync/Backup), `hans-solo` (MongoDB/Backup DB), `sync-disc`

🔴 Decommissioned / Legacy VMs

Category	Systems
Windows/Legacy	`w2k3-build-01`, `w2k3-dc-1`, `w2k3-dc-2`, `wilson`, `sharon` (W2K3 Replacement), `randy` (W2K3 Replacement)
Replaced / Obsolete	`assassin-interconnect`, `auth-proxy-1`, `ff-proxy`, `cuddy`, `dockglu-1/2/3`, `dockheketi-1`, `fare-docker-1/2/3/4`, `fare-docker-store`, `fare-rancher`, `gatekeeper-1`, `tarvos` (Test Firefly-III)
Other Legacy	`apu` (Puppet Master), `arnie` (Hetzner Node), `billet` (R720xd VM), `buid-osx-1` (OS X Build Slave), `cameron` (R720xd VM), `che` (Eclipse Che), `chief-wiggum` (Wazuh Security), `cloud-dr-1` (Hetz Replication), `corfu` (Web App 4 - R720xd), `cosh-fw-2` (Datacenter/Test Firewall), `cosh-fw-3` (Test Firewall), `crete` (PHPipam/Portainer - R720xd), `fare-fs-1` (Media Syncthing), `kyle` (Icinga2 Master), `lemnos` (R720xd VM), `metis` (R720xd VM), `mob-vpn-interconnect` (Mobile VPN), `mr-bump` (FreeIPA Client - R720xd), `naxos` (FreeIPA Self-Service - R720xd), `ross` (Icinga2 Distributed), `samos` (Rancher - R720xd), `sec-scanner` (Security Scanning?), `sherlock` (Location APIs)

🧾 7. IDENTITY / IAM SYSTEMS

System	Type	Role	Notes	Status
Windows NT Domain	Identity	Early auth system	Compaq era	🔴
FreeIPA (marge/homer)	Identity	Linux identity domain	CentOS, deployed Jun 2016	🟢
Keycloak-1	IAM	Legacy authentication	Deprecated VM instance	🟡
Keycloak (Nomad)	IAM	Primary auth system	login.dockstudios.co.uk	🟢
HashiCorp Vault (albali)	Secrets	Secrets management	Oct 13 2023 FS created	🟢

🧾 8. CI / DEVOPS SYSTEMS

System	Role	Notes	Status
pub-jenkins	Public CI	GitLab → GitHub sync, public builds	Ubuntu 18.04, Oct 2018 FS
fare-jenkins	Internal CI	Private pipelines (PrivateJenkins-Config)	🟢
ci-host-1	CI executor	Build worker	🟢

🧾 9. GIT / DEV PLATFORM EVOLUTION

System	Platform	Role	Notes	Status
GitLab	Rancher	Source control platform	Jul 15 2019 initial deployment	🟡
GitLab	Nomad	Orchestrated platform	Migration ~Jun 6 2026	🟢

🧾 10. EDGE / EXTERNAL CLUSTER

Node	Provider	Role	Notes	Status
hetznur02	Hetzner	Cluster node	Debian 11 (updated May 2026 from Buster). Managed via PortainerConfigs	🟢
hetzfal01	Hetzner	Cluster node	Debian 11 (updated May 2026 from Buster). Managed via PortainerConfigs	🟢
ovhlim01	OVH	Cluster node	Debian 11 (updated May 2026 from Buster). Managed via PortainerConfigs	🟢

🧾 11. NETWORK / SECURITY INTERCONNECT

System	Role	Notes	Status
admin-vpn-interconnect	VPN	Admin access	Debian 9 (Stretch) created Nov 2018 (formerly pfsense fare-fw-1)
remote-interconnect	VPN	Remote access	Debian 9 (Stretch) created Jan 2018 (formerly pfsense fare-fw-1)
hetz-vpn-interconnect	VPN	External link	Debian 9 (Stretch) created Nov 2018 (formerly pfsense fare-fw-1)
init-connect-vpn-interconnect	VPN	ComboPod component	Debian 9 (Stretch) created Nov 2018
netbird-gateway	Mesh VPN	Zero-trust networking
hetz-haproxy	Proxy	Cluster Load Balancer	Hetzner cluster ingress

🧾 12. LEGACY BUILD / WINDOWS 2003 PIPELINE

System	Role	Notes	Status
w2k3-build-01	Build server	Jenkins dependency system	🟡
w2k3-dc-1	Domain controller	Build auth system	🟡
w2k3-dc-2	Domain controller	Redundant auth	🟡

🧾 13. KEY ARCHITECTURAL TRANSITIONS

Transition	From	To	Meaning
Identity	NT domain	FreeIPA	Linux identity foundation
IAM	VM Keycloak	Nomad Keycloak	Orchestrated identity
CI exposure	internal only	pub-jenkins	public build gateway
GitLab hosting	Rancher	Nomad	platform consolidation
Compute model	VM-centric	orchestrated services	infrastructure abstraction
Proxy	Nginx	HAProxy	Better HA and config management
VPN	OpenVPN	Boundary	From clunky VPN to zero-trust
Orchestration	(Internal) Swarm	Nomad	Core cluster migration to Hashicorp stack
Security/Firewall	Various (ISA/VM)	pfsense / Distributed	Persistent network security layer
Virtualisation	oVirt	Nomad/Containerized	Transition from VM cluster to orchestration
Storage	Local/NAS	ZFS + iSCSI	From local disks to high-perf distributed storage

🧬 Application & Tooling Lifecycle

This section traces the evolution of key application domains, showing how the tools and hosting environments have matured over the years.

Domain	Early / Legacy Era (Physical/ESX)	Consolidation / Rancher Era (K8s/VMs)	Modern Nomad Era (Orchestrated)
Identity & Auth	Windows NT Domain	FreeIPA (marge/homer)	Keycloak (Nomad) + Vault
Orchestration	Manual / ESX / KVM	Rancher (RKE/K8s)	Nomad + Consul
CI/CD	Jenkins (sheila)	Private/Public Jenkins	GitLab (Nomad)
Version Control	Manual / Local	GitLab (Rancher)	GitLab (Nomad) / Forgejo
DNS/Networking	BIND (Manual)	BIND + Consul (Early)	BIND + Consul (Dynamic) + Netbird
Ingress/Proxy	Apache / Nginx	Nginx Ingress (K8s)	Traefik (Nomad)
Storage	Local HDD / NAS	iSCSI (Synology/Nexsan)	ZFS + iSCSI (Synology)

🧾 14. METHODOLOGICAL EVOLUTION

🛠️ Provisioning & Configuration Management

Era	Method	Key Systems	Notes
Manual Era (~2011–2014)	Direct SSH / Manual	`x3455`, `DL160`	Early manual provisioning
Config Management Era (~2015–2020)	Puppet / Ansible	`apu` (Puppet Master)	Automated configuration via Puppet/Ansible
IaC Era (~2021–Present)	Terraform / Ansible	`cosh-servers` (Terraform)	Declarative infrastructure via Terraform

📦 Artifact & Registry Management

Era	Method	Key Systems	Notes
Local Era (~2011–2015)	Local FS / Manual	`fare-fs-1`	Manual artifact management
Repository Era (~2016–2022)	Nexus / Apt	`crypt`	Centralized repository management
Registry Era (~2023–Present)	Harbor / Container Registry	`harbor`	Container image orchestration

👁️ Observability & Monitoring

Era	Method	Key Systems	Notes
Active Monitoring Era (~2012–2018)	Icinga2 / Nagios	`kyle`, `ross`	Legacy host-based monitoring
Centralized Era (~2018–2023)	Checkmk / Monitoring	[TBC]	[TBC]
Modern Observability Era (~2024–Present)	Prometheus / Consul / Nomad	[TBC]	[TBC]

💻 Development & Workspace Environments

Era	Method	Key Systems	Notes
Local/Jenkins Era (~2011–2017)	Jenkins Slaves / Manual VMs	`buid-osx-1`, `ci-host-1`	Early CI/CD execution
Containerized Era (~2018–2023)	Rancher / Docker Swswarm	`fare-docker-*`, `samos` (Rancher)	Transition from non-docker (`cuddy`) $\to$ `fare-docker` $\to$ Rancher
Ephemeral/Cloud-Native Era (~2024–Present)	Coder / Nomad	`coder`	Developer environment orchestration

🧠 15. CURRENT ARCHITECTURAL STATE (SUMMARY)

You currently operate a consolidated, orchestrated infrastructure based on the following layers:

🟢 Orchestration layer

Nomad (core scheduler and service plane)
Consul (service discovery and KV store)

🟢 Identity & Security layer

Keycloak (primary identity provider, Nomad-hosted)
FreeIPA (legacy identity dependency)
HashiCorp Vault (secrets management and control plane)
Boundary (zero-trust access gateway)

🟢 CI/CD & Dev Platform layer

Jenkins (split between public and internal pipelines)
GitLab (source control and orchestration, Nomad-hosted)
Coder (ephemeral developer environments)

🟢 Compute Layers

Core Compute

R720xd Consolidation Host: Primary host for the majority of the service plane.

Edge Compute

3-node symmetric cluster: Multi-provider (Hetzner/OVH) footprint (HAProxy, BIND, Consul, Tinc).

🟢 Storage & Backup Layer

Storage: ZFS-based high-performance storage (10K RPM HDDs + SSD L2ARC/SLOG) via iSCSI/Synology.
Backup: Tiered Rsync flow (hans $\to$ pfserver-04 $\to$ Backblaze).

🟢 Networking Layer

Cluster Interconnect: Tinc (decentralized mesh for Hetzner cluster linking)
DNS: BIND (Public) + Consul (Dynamic Public)
Mesh VPN: Netbird (zero-trust connectivity)
Secure Access: Boundary + Netbird + Jump-hosts
Ingress/Load Balancing: HAProxy for cluster ingress
Security: Distributed pfsense firewalling.

Feeds