Secret Delivery Pipeline¶

Overview¶

Secret Delivery Pipeline follows the path from GCP Secret Manager to a live Kubernetes secret. The service account is provisioned in Terraform. On-prem bootstrap publishes the store credential to the cluster. Applications declare an ExternalSecret, and ESO delivers the materialized secret to the namespace.

On GKE, Workload Identity replaces the static key. The cluster-side delivery pattern remains the same.

Case study¶

Context: application secrets were spread across environment files and manual operator steps. There was no traceable path from GCP Secret Manager to the application namespace, and credentials were embedded in GitOps state in some services.
Challenge: on-prem clusters needed a secure, auditable path to cloud-held secrets without the static key appearing in Terraform state, Git history, or the operator workstation beyond the bootstrap moment.
Approach: org/gcp/gsm-eso-sa provisions the service account. The key is generated once via hyops init gcp --with-eso-sa and written only to the local bootstrap vault. platform/k8s/gsm-bootstrap publishes the credential to the cluster at bootstrap time. ESO then handles all subsequent syncs through a ClusterSecretStore, with no operator involvement at sync time. On GKE, Workload Identity eliminates the static key entirely.
Outcome: the recorded exercise confirmed the full GSM → ExternalSecret → Kubernetes Secret delivery cycle on both the on-prem RKE2 and GKE environments. The GKE path completed delivery and cleanup, and static credentials no longer appeared in state or history.

Covers SA creation in Terraform, key generation via hyops init, GSM bootstrap apply, ClusterSecretStore creation, ExternalSecret sync, and the verified secret landing in the application namespace.

Outcome¶

The result is a repeatable, auditable secret delivery path that works across on-prem and cloud surfaces.

GCP Secret Manager holds the authoritative secret values.
ESO reads from GSM and delivers secrets into the application namespace without operator involvement at sync time.
The service account credential is provisioned once, stored as a Kubernetes secret, and reused across store operations.
On GKE, Workload Identity eliminates the static key while preserving the same store configuration.

Operating model¶

org/gcp/gsm-eso-sa (Terraform) creates the GCP service account (eso-gsm-reader@...) with Secret Manager read access. No key is generated at this stage.
hyops init gcp --with-eso-sa runs gcloud iam service-accounts keys create, captures the key JSON, and writes it to the operator's local encrypted bootstrap vault as HYOPS_GSM_SA_KEY_JSON. It is an operator bootstrap step. The key never touches Terraform state.
platform/k8s/gsm-bootstrap reads HYOPS_GSM_SA_KEY_JSON from the bootstrap vault at cluster bootstrap time and creates the gsm-sa-credentials Kubernetes secret in the external-secrets namespace.
platform/k8s/gcp-secret-store creates the ESO ClusterSecretStore, backed by gsm-sa-credentials.
Applications declare an ExternalSecret in their namespace. ESO handles the fetch from GCP Secret Manager and delivers a Kubernetes secret without further operator action.

The GKE environment uses Workload Identity instead of a static service account key. The ClusterSecretStore configuration and delivery behaviour from step 4 onward are the same.

Architecture¶

Secret delivery pipeline showing the GCP reader identity, bootstrap handoff, ClusterSecretStore contract, application delivery through ExternalSecret, and Workload Identity as the GKE auth variant.

The static key never touches Terraform state or Git history. Each stage of the pipeline is a discrete, reviewable platform action.

Platform state¶

IP addresses, hostnames, and instance identifiers visible in screenshots and recordings reflect the ephemeral infrastructure provisioned during the recorded exercise.

Each stage is backed by a recorded run record. The captured exercise included a full GSM → ExternalSecret → Kubernetes Secret round trip on both delivery surfaces.

Stage	Module	Run ID	Status
SA provisioning	`org/gcp/gsm-eso-sa`	`apply-20260317T193309Z-465b4e8e`	ok
Bootstrap	`platform/k8s/gsm-bootstrap`	`apply-20260313T230910Z-739e848c`	ok
GKE secret store	`platform/k8s/gcp-secret-store#gke_burst_secret_store`	`apply-20260318T120046Z-bc73643d`	ok

The recorded GKE exercise completed a full GSM → ExternalSecret → Kubernetes Secret round trip and cleaned up the validation objects afterwards. Representative execution records are retained as part of the platform log.

Implementation¶

Surface	Auth mechanism	Notes
On-prem RKE2	Static SA key via `gsm-sa-credentials`	Provisioned by `gsm-bootstrap` at cluster bootstrap
GKE burst	Workload Identity	No static key; WI annotation on ESO service account

Key components¶

SA provisioning: org/gcp/gsm-eso-sa (Terraform)
Key generation: hyops init gcp --with-eso-sa (writes to the bootstrap vault)
Bootstrap: platform/k8s/gsm-bootstrap
Secret store: platform/k8s/gcp-secret-store
GKE variant: platform/k8s/gcp-secret-store#gke_burst_secret_store
ESO: deployed via the public workload GitOps root alongside the cluster

Where it fits¶

Application secrets with a single authoritative source in GCP Secret Manager
On-prem clusters that need a secure path to cloud held secrets
Platform services that share one ClusterSecretStore across multiple application namespaces
Teams moving secrets out of environment files and into a controlled delivery path

References¶

What was verified¶

Verified during the recorded HybridOps v1.0.1 secret-delivery exercise. Service account provisioning was confirmed in Terraform, the bootstrap key path was validated, and gsm-bootstrap and ClusterSecretStore were verified on both the on-prem RKE2 and GKE surfaces.