Skip to content

CI/CD Pipeline — Jenkins Orchestrator

Purpose and scope

Orchestrates plan/apply flows across on-prem and cloud stacks, executes repeatable scenarios, and captures evidence outputs. Scope includes the Jenkins controller configuration, shared library, and pipeline definitions under this repository.

Repository layout

  • ci/jenkins/jobs/
    Groovy seed jobs and job definitions.

  • ci/jenkins/pipelines/
    Pipeline folders and Jenkinsfiles (parameterised orchestration entry points).

  • ci/jenkins/shared-library/
    Shared library used by pipelines to standardise bootstrap, credentials handling, and command execution.

Execution contract

Pipelines follow a common stage model:

  1. Bootstrap
    Load the bootstrap secret bundle, mint ephemeral runtime files, and export required environment variables for downstream tooling.

  2. Init
    Run provider init scripts in non-interactive mode where required to generate infra/env/*.credentials.tfvars.

  3. Plan / Validate
    Run Terragrunt/Terraform plan and validation steps (and Packer validation where applicable).

  4. Apply / Build
    Apply infrastructure changes or build artefacts (templates, images) using standard Makefile entry points.

  5. Verification
    Run smoke checks appropriate to the flow.

  6. Evidence
    Persist logs and redacted artefacts under output/, and archive selected artefacts in Jenkins.

Bootstrap and secrets contract

Bootstrap secret bundle

A single Ansible-Vault encrypted env-format bundle is used to supply bootstrap values without interactive prompts.

  • Default vault file: control/secrets.vault.env
  • Vault content: KEY=VALUE lines (comments and blank lines permitted)
  • Consumption pattern: decrypted to an ephemeral file and exported into the process environment for the duration of a step.

Standard bootstrap wrapper

Pipelines should use the shared library wrapper (for example runWithBootstrapEnv) to:

  • Ensure repository checkout is performed where required.
  • Materialise ephemeral runtime files:
  • vault password file (for ansible-vault view)
  • optional GCP runner key file (for ADC via GOOGLE_APPLICATION_CREDENTIALS)
  • Invoke control/tools/helper/secrets/with_bootstrap_env.sh to execute commands with environment enrichment derived from the vault bundle.

Vault file override is supported for enterprise-style paths:

  • Example: /var/lib/jenkins/bootstrap/secrets.vault.env

Credentials

Credential IDs are treated as part of the pipeline contract and are referenced by the shared library and JCasC templates.

Required for bootstrap automation

  • ansible-vault-password
    Type: secret text. Used to decrypt control/secrets.vault.env.

Required for GCP CI flows that use impersonation

  • gcp-runner-sa-json
    Type: secret file (preferred). Service account key used to authenticate the CI runner, then impersonate the Terraform runtime service account.

The runner key is generated by the interactive bootstrap on an operator workstation (see control/tools/provision/init/gcp/init-gcp-bootstrap.sh) and is then stored in Jenkins credentials.

Optional common credentials

  • github-token (secret text)
  • tfc-token (secret text)
  • netbox-token (secret text)
  • proxmox-api-token (secret text)
  • ssh-key (SSH private key)

These should not be required for controller bring-up. They are consumed by specific pipelines as needed.

Provider init scripts

Provider init scripts generate infra/env/*.credentials.tfvars for Terraform/Terragrunt and Packer.

  • control/tools/provision/init/init-azure-env.sh
  • control/tools/provision/init/gcp/init-gcp-ci.sh (CI validation for GCP runner + impersonation)
  • control/tools/provision/init/init-proxmox-env.sh
  • control/tools/provision/init/init-tfc-env.sh

Interactive bootstrap scripts must not run in CI. CI paths should use non-interactive init and validation flows.

Evidence outputs

Evidence is captured in two places:

  • Jenkins build logs and archived artefacts (per-job retention policy).
  • Repository evidence trees produced by tools and init scripts:
  • output/logs/...
  • output/artifacts/...
  • latest symlinks where implemented by the tooling

Pipelines should prefer invoking existing tooling that already writes to the standard output/ locations rather than inventing pipeline-local paths.

Failure modes and recovery

Common failure categories:

  • Missing or invalid credentials (vault password, GCP runner key, expired tokens).
  • Provider API permission gaps (IAM/RBAC not granted or not yet propagated).
  • Drift in runner images (missing CLIs, incompatible versions).
  • Terragrunt/Terraform state locks, workspace misconfiguration, or policy failures.

Recovery actions:

  • Re-run bootstrap and init scripts on an operator workstation when identities or permissions must be changed.
  • Rotate and re-seed Jenkins credentials when runner keys or tokens are rotated.
  • Rebuild CI runner images when tooling drift is detected.

Change guidelines

  • Prefer shared library helpers over duplicating shell logic in Jenkinsfiles.
  • Keep Jenkinsfiles thin: parameter parsing + one call into the shared library wrapper.
  • Do not embed secret values in pipeline code or JCasC YAML; reference Jenkins credentials by ID.
  • Add new credential IDs only when required, and update the shared library contract and controller JCasC templates together.