Run Jenkins controller on the control node and move build execution to agents¶

Status¶

Accepted

1. Context¶

HybridOps needs a Jenkins operating model that keeps the control plane stable while allowing build execution to scale and change independently.

The platform already has two distinct execution surfaces:

the control node, which holds the long-lived automation entry point
the RKE2 cluster, which is the preferred runtime for elastic worker workloads

If Jenkins runs everything on the controller host, several problems follow:

the UI and scheduling layer compete with builds for CPU, memory, and disk
host-level failures affect both coordination and execution at the same time
the steady-state platform drifts away from the Kubernetes delivery model used elsewhere in HybridOps

At the same time, the earliest bootstrap phase cannot assume RKE2 pod agents are already online. The architecture therefore needs a steady-state answer and a controlled bootstrap exception.

2. Decision¶

HybridOps adopts the following Jenkins placement model:

the Jenkins controller runs on the control node as a containerised service
Jenkins build execution moves to agents rather than running on the controller
the preferred steady-state execution surface is Jenkins agents on RKE2
a Docker-based bootstrap agent on the control node is allowed only as a transitional path before RKE2 agents are available
controller executors remain at 0 in steady state

This separates:

control plane: UI, scheduling, credentials indirection, and JCasC
execution plane: pipeline steps, toolchains, and short-lived build work

3. Rationale¶

3.1 Keep the control plane stable¶

The controller is the long-lived coordination surface. It benefits from:

predictable host placement
straightforward backup and restore
limited workload variance

Running it on the control node preserves that stability.

3.2 Put execution where elasticity belongs¶

RKE2 is already the preferred runtime for platform workloads. Using it for Jenkins agents means:

execution can scale independently of the controller
worker environments can be replaced without rebuilding the control plane
the delivery architecture stays aligned with the rest of the platform

3.3 Make bootstrap explicit instead of accidental¶

Early in platform bring-up, RKE2 agents may not exist yet. A Docker-based agent on the control node is acceptable during that window, but it is a bootstrap bridge, not the target operating model.

4. Consequences¶

Positive¶

the Jenkins controller remains small and easier to operate
build execution can move between bootstrap and steady-state surfaces without redefining the controller role
controller recovery and agent recovery become separate concerns
the platform can explain a clean control-plane versus execution-plane split

Trade-offs¶

the bootstrap phase carries temporary complexity because a Docker agent may be required before RKE2 agents are ready
agent labels, secrets, and node definitions need disciplined management
pipelines that assume local controller execution must be corrected

5. Operating model¶

Controller¶

runs on the control node
uses Docker and Configuration as Code
remains reachable even when individual agents are replaced

Preferred agents¶

run on RKE2
carry the build toolchain needed for HybridOps delivery workflows
execute image, Terraform, Ansible, and validation jobs away from the controller

Bootstrap exception¶

a Docker-based inbound agent on the control node may be used during platform bring-up or controlled fallback
this agent is temporary and should be retired once RKE2 agents are available

6. Validation¶

This decision is considered implemented when:

the controller runs on the control node and is healthy
controller executors are disabled in steady state
at least one agent execution path exists outside the controller
the supported runbooks and HOWTOs use this placement model consistently

7. References¶

Maintainer: HybridOps
License: MIT-0 for code, CC BY 4.0 for documentation unless otherwise stated.

Runbooks¶

HOWTOs¶

Run records¶

Capability Map