Skip to content

Verify Secret Delivery Pipeline

Intent

  • Purpose: Confirm that the secret-delivery path is healthy across both the on-prem and burst delivery surfaces.
  • Trigger: Pre-demo validation, post-change validation, or routine confirmation before relying on cluster-delivered secrets.
  • Impact: Verifies the GCP service-account authority layer, bootstrap material, ClusterSecretStore readiness, and application-side ExternalSecret sync.
  • Severity: P2
  • Rollback: Verification only. If the path is unhealthy, stop dependent rollout and reconcile the relevant bootstrap, store, or application state first.

Preconditions

  • The service-account authority and at least one cluster-side store have already been deployed.
  • The on-prem RKE2 cluster and the GKE burst cluster are reachable.
  • Do not print or persist secret payload material during verification.

Verify authority and on-prem bootstrap

hyops show module org/gcp/gsm-eso-sa --env dev
hyops show module platform/k8s/gsm-bootstrap --env dev

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/rke2.yaml" \
kubectl -n external-secrets get secret gsm-sa-credentials

Expected:

  • org/gcp/gsm-eso-sa is status=ok
  • the ESO reader service account email is published
  • platform/k8s/gsm-bootstrap is status=ok
  • gsm-sa-credentials exists in external-secrets

Verify on-prem store and sync

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/rke2.yaml" \
kubectl get clustersecretstore -A

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/rke2.yaml" \
kubectl get externalsecret -A

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/rke2.yaml" \
kubectl -n keycloak get externalsecret,secret

Expected:

  • gcp-secret-manager reports READY=True
  • representative ExternalSecret resources report SecretSynced and READY=True
  • the materialized application secret exists alongside its ExternalSecret

Verify GKE burst store and sync

hyops show module platform/k8s/gcp-secret-store#gke_burst_secret_store --env dev

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/gke-burst.yaml" \
kubectl get clustersecretstore -A

KUBECONFIG="$HOME/.hybridops/envs/dev/state/kubeconfigs/gke-burst.yaml" \
kubectl get externalsecret -A

Expected:

  • the burst secret-store module is status=ok
  • cap.k8s.gcp-secret-store: ready
  • the GKE gcp-secret-manager store reports READY=True
  • representative burst-lane ExternalSecret resources report SecretSynced and READY=True

References