Skip to content

GCP Secret Store Lifecycle (HyOps Module)

  • Purpose: Establish a reusable ClusterSecretStore in GKE for External Secrets, backed by Google Secret Manager and authenticated through GKE Workload Identity.
  • Trigger: A cloud burst or cloud-native workload lane needs secret delivery without static service-account keys inside the cluster.
  • Impact: Enables External Secrets to resolve secrets from Google Secret Manager through a Kubernetes service account bound to a Workload Identity principal.
  • Severity: P1
  • Rollback strategy: Remove the ClusterSecretStore, remove the workload-identity IAM binding, and remove the Kubernetes service account if the cluster should no longer resolve cloud secrets.

Context

Module ref: platform/k8s/gcp-secret-store

This module is the GKE-native alternative to the on-prem platform/k8s/gsm-bootstrap pattern:

  • on-prem pattern: seed a static service-account key secret into the cluster
  • GKE pattern: use Workload Identity and a ClusterSecretStore

That split is intentional. The cluster keeps a short trust chain and avoids long-lived cloud credentials in Kubernetes Secrets.

Preconditions

  • GKE cluster is reachable and kubeconfig is already published
  • platform-external-secrets is installed in the target cluster
  • gcloud and kubectl are installed on the controller
  • an active gcloud account can operate on the target project
  • the cluster has Workload Identity enabled

Typical state inputs:

  • kubeconfig_state_ref: platform/gcp/gke-kubeconfig#gke_burst_kubeconfig
  • cluster_state_ref: platform/gcp/gke-cluster#gke_burst_cluster

Execute

hyops apply --env dev \
  --module platform/k8s/gcp-secret-store \
  --state-instance gke_burst_secret_store \
  --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__k8s__gcp-secret-store/instances/gke_burst_secret_store.inputs.yml"

What the module does

  1. Confirms kubeconfig and controller-side tooling are present.
  2. Enables secretmanager.googleapis.com when requested.
  3. Ensures the External Secrets CRDs are present.
  4. Waits for the External Secrets operator deployments to become ready.
  5. Creates the Kubernetes service account used by the store.
  6. Grants the Workload Identity principal roles/secretmanager.secretAccessor.
  7. Applies the ClusterSecretStore.
  8. Waits for the store to report Ready=True.

Verification

Check module state:

jq '.status, .outputs' \
  "$HOME/.hybridops/envs/dev/state/modules/platform__k8s__gcp-secret-store/instances/gke_burst_secret_store.json"

Check the store directly:

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/gke-burst.yaml" \
kubectl get clustersecretstore gcp-secret-manager -o yaml

Check the workload-identity Kubernetes service account:

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/gke-burst.yaml" \
kubectl get serviceaccount -n external-secrets eso-gcp-secret-manager -o yaml

Check API enablement:

GCP_PROJECT_ID="$(jq -r '.context.project_id' "$HOME/.hybridops/envs/dev/meta/gcp.ready.json")"

gcloud services list --enabled \
  --project "$GCP_PROJECT_ID" \
  --filter='config.name:secretmanager.googleapis.com' \
  --format='value(config.name)'

Success indicators:

  • module state ends with status=ok
  • cap.k8s.gcp-secret-store = ready
  • the ClusterSecretStore condition shows Ready=True
  • the eso-gcp-secret-manager service account exists in external-secrets
  • secretmanager.googleapis.com is enabled on the secret project

End-to-end secret sync validation

To prove the path beyond store readiness, use a disposable GSM secret and a disposable namespace:

  1. Create or update a temporary secret in Google Secret Manager.
  2. Apply an ExternalSecret in a disposable namespace that targets gcp-secret-manager.
  3. Wait for the ExternalSecret condition Ready=True.
  4. Read the materialized Kubernetes Secret and confirm the value matches the GSM source.
  5. Delete the validation namespace, ExternalSecret, and temporary GSM secret.

This path has already been exercised successfully against the current burst lane. Keep the validation flow disposable; do not leave long-lived validation secrets behind in the public burst cluster.

Troubleshooting

External Secrets CRDs exist but the operator deploys under a prefixed release name

The module resolves deployment names dynamically by suffix, so prefixed release names like platform-external-secrets are supported. If this still fails, confirm the operator deployments are present:

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/gke-burst.yaml" \
kubectl -n external-secrets get deploy

ClusterSecretStore does not become Ready

  • Confirm the cluster has Workload Identity enabled.
  • Confirm the IAM binding was created on the target project.
  • Check External Secrets logs:
    KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/gke-burst.yaml" \
kubectl -n external-secrets logs deploy/platform-external-secrets

Secret Manager API is disabled

Re-run with ensure_secretmanager_api: true, or enable it explicitly before rerunning.

Store is Ready but no Kubernetes Secret appears

  • Confirm the ExternalSecret references gcp-secret-manager.
  • Confirm the GSM secret exists in the target project and the exact key name matches the remoteRef.key.
  • Check ExternalSecret status:
    KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/gke-burst.yaml" \
kubectl -n <namespace> get externalsecret <name> -o yaml

References