Skip to content

Init GCP credentials with hyops init gcp

  • Purpose: Initialise GCP runtime inputs so Terraform/Terragrunt stacks can run with ADC + impersonation. Owner: Platform operations

  • Trigger: First-time environment bootstrap, new workstation/runner.

  • Impact: Without GCP auth and impersonation, GCP stacks cannot apply.

  • Severity: P3 Pre-reqs: gcloud installed; access to the target project; permission to impersonate GCP_TERRAFORM_SA_EMAIL when steady-state impersonation is configured.

  • Rollback strategy: Remove the generated credentials file under <root>/credentials/ and re-run init.

Context

hyops init gcp writes:

  • Non-secret config: <root>/config/gcp.conf
  • Credentials tfvars: <root>/credentials/gcp.credentials.tfvars (mode 0600)
  • Readiness marker: <root>/meta/gcp.ready.json

The tfvars file is designed for Terraform provider impersonation flows.

The readiness marker may also carry a non-secret SSH public key for cloud VM bootstrap flows. This lets shipped blueprints stay free of embedded operator keys.

Preconditions and safety checks

  • Ensure you are targeting the correct environment:

    echo "HYOPS_ENV=$HYOPS_ENV"

  • Confirm required tools exist:

    command -v gcloud

Notes: - If you are bootstrapping a brand new project (project does not exist yet), set GCP_ADC_QUOTA_PROJECT_ID in <root>/config/gcp.conf to an existing accessible project to avoid interactive quota project selection during ADC flows. - GCP_BILLING_ACCOUNT_ID is optional in gcp.conf, but useful when the same env will run org/gcp/project-factory to create a new project.

Steps

1) Generate the config template (first run only)

  • Command:

    hyops init gcp --env dev

  • Expected result:

  • If the config does not exist, HyOps writes a template and exits with guidance.
  • Edit <root>/config/gcp.conf (required: GCP_PROJECT_ID, GCP_REGION) and re-run.
  • Optional: set GCP_BILLING_ACCOUNT_ID if you already know which billing account should be used for project creation.
  • Optional but recommended: set GCP_SSH_PUBLIC_KEY if the local machine does not already have ~/.ssh/id_ed25519.pub or ~/.ssh/id_rsa.pub.
  • GCP_TERRAFORM_SA_EMAIL is optional (recommended for steady-state). If omitted, HyOps can derive it from org/gcp/project-factory module state after that module has been applied.

2) Interactive bootstrap (workstations)

  • Option A (manual login):

    gcloud auth login gcloud auth application-default login hyops init gcp --env dev

  • Option B (allow HyOps to invoke login flows):

    hyops init gcp --env dev --with-cli-login

  • Existing env repair (review and override current config safely):

    hyops init gcp --env dev --with-cli-login --force

Note: - with --with-cli-login, HyOps can derive GCP_PROJECT_ID and GCP_REGION from the active gcloud context, and can also let you select an open billing account for org/gcp/project-factory - when those values are only auto-detected from gcloud, HyOps now shows them and lets you keep or override them before persisting them into the env-scoped config - the raw gcloud login message Your current project is [...] is Google CLI output, not HyOps accepting that project silently; HyOps follows it with an explicit review prompt before writing gcp.conf - with --with-cli-login --force, HyOps also rechecks values already present in gcp.conf, so an existing env does not silently stay pinned to an old project, region, or billing account - if the selected project is no longer accessible to the active Google identity, HyOps now fails explicitly instead of leaving the env in a misleading ready state; in interactive repair mode it also prompts for a replacement project id before persisting new config - if you explicitly target a brand new project id and a billing account is available, HyOps can continue in bootstrap mode so org/gcp/project-factory can create the project; in that mode other GCP-backed modules fail fast until you rerun hyops init gcp --force after project creation - in that same bootstrap path, if ADC exists but lacks billing.resourceAssociations.create on the chosen billing account while the active gcloud identity can access it, hyops init gcp --with-cli-login now drives gcloud auth application-default login inline instead of leaving you to discover the ADC mismatch later during project-factory - during that ADC refresh, Google may warn that it could not add an older project as the ADC quota project. That is usually harmless during bootstrap if the old project is no longer accessible. After the new project exists, rerun hyops init gcp --force or set the quota project explicitly with gcloud auth application-default set-quota-project <new-project-id> - if GCP_TERRAFORM_SA_EMAIL is explicitly set, impersonation validation still fails hard when permissions are wrong - if the service account is only auto-derived from org/gcp/project-factory state, HyOps can use direct ADC temporarily during bootstrap or IAM propagation, then converge to impersonation after project creation and a final hyops init gcp --force - when the derived Terraform service account exists in the target project and the active operator identity can manage it, HyOps now attempts the bootstrap roles/iam.serviceAccountTokenCreator grant automatically before rechecking impersonation - in the same current-project bootstrap path, HyOps also ensures roles/servicenetworking.networksAdmin on the Terraform service account and checks the private-service-access permission pair needed by Cloud SQL private networking - that bootstrap only covers the current project selected by init; if a module later uses a Shared VPC host project, the host project still needs the corresponding network roles for the effective Terraform identity

3) Non-interactive mode (CI/runners)

  • Action: ensure ADC is already configured on the runner and impersonation is permitted. Common CI pattern:
  • set GOOGLE_APPLICATION_CREDENTIALS to a runner service-account JSON key file
  • ensure the runner identity can impersonate GCP_TERRAFORM_SA_EMAIL (roles/iam.serviceAccountTokenCreator)

  • ensure GCP_TERRAFORM_SA_EMAIL is set (via <root>/config/gcp.conf or GCP_TERRAFORM_SA_EMAIL env var)

  • Command:

    hyops init gcp --env dev --non-interactive

Verification

  • Confirm the readiness marker includes context.ssh_public_key when a public key is discoverable or configured:

    jq -r ".context.ssh_public_key" ~/.hybridops/envs/dev/meta/gcp.ready.json

  • Confirm readiness marker exists:

    cat ~/.hybridops/envs/dev/meta/gcp.ready.json

Expected interpretation: - context.auth_mode="impersonation" with context.impersonation_validated=true means the environment is in the preferred steady-state mode. - context.auth_mode="direct-adc" with context.impersonation_validated=false is only a transient bootstrap or IAM-propagation state. It is acceptable immediately before or after org/gcp/project-factory, but the normal end state should be a rerun of hyops init gcp --force that moves the env into impersonation.

  • Confirm credentials file exists:

    ls -la ~/.hybridops/envs/dev/credentials/gcp.credentials.tfvars

Notes: - HyOps validates ADC and impersonation without persisting access tokens to run-record paths. - If the env remains in direct-adc after project creation, treat that as a recovery step still in progress and rerun hyops init gcp --force before continuing with steady-state GCP workflows.

Troubleshooting

gcloud organizations list returns Listed 0 items.

This is common on personal/trial Google accounts without a Cloud Organization.

  • You can still use HyOps with an existing project:
  • Create a project manually in the Cloud Console (or ask an admin to create one).

  • Set GCP_PROJECT_ID in <root>/config/gcp.conf and re-run hyops init gcp.

  • You can still use org/gcp/project-factory if you have permission to create projects and attach billing:

  • Leave org_id and folder_id unset in the project-factory inputs (project lands under “No organization”).
  • You still need a valid/open billing account ID.

gcloud beta billing accounts list is empty / billing is closed

org/gcp/project-factory needs an open billing account to create a new project.

  • Confirm your billing account is open:

    gcloud beta billing accounts list

  • If you do not have one, create a billing account in the Cloud Console (or ask an admin).

  • If you have multiple open accounts, hyops init gcp --with-cli-login can persist the selected account as GCP_BILLING_ACCOUNT_ID in the env-scoped gcp.conf.

Project-factory apply fails with permission errors

Typical causes:

  • You cannot create projects (resourcemanager.projects.create), or
  • You cannot link the project to the billing account.

Options:

  • Use an existing project and skip project-factory.
  • Ask an admin to grant the required permissions, or have them run project-factory once for you.

ERR: cannot access configured GCP project

Meaning:

  • the current GCP_PROJECT_ID exists only in local config, or
  • it belongs to another Google account or organization, or
  • your current identity no longer has IAM access to it

This is different from gcloud merely remembering an old project name in its local config.

Recommended repair path:

hyops init gcp --env dev --with-cli-login --force \
  --project-id <new-project-id> \
  --region <region>

Then verify:

hyops show init --env dev gcp

References

License: MIT-0 for code, CC-BY-4.0 for documentation