Initialise Terraform Cloud credentials¶

Purpose: Establish Terraform Cloud authentication for Terraform/Terragrunt runs (cloud backend). Owner: Platform operations
Trigger: First-time Terraform Cloud onboarding, token rotation, new workstation/runner bootstrap.
Impact: When using Terraform Cloud state, stacks cannot run without a valid token.
Severity: P2 Pre-reqs: Terraform CLI installed, network access to Terraform Cloud. Vault is optional (only needed if you want HyOps to persist the token into the runtime vault bundle).
Rollback strategy: Revoke token in Terraform Cloud, remove local Terraform credentials, and optionally remove the runtime vault entry.

Context¶

hyops init terraform-cloud:

Reads config from <root>/config/terraform-cloud.conf.
Finds a token in this order:
runtime vault bundle (<root>/vault/bootstrap.vault.env) when it exists and is decryptable
Terraform CLI credentials file (default: ~/.terraform.d/credentials.tfrc.json)
optionally runs terraform login when --with-cli-login is set
Always writes/updates the Terraform CLI credentials file (mode 0600).
Writes a readiness marker: <root>/meta/terraform-cloud.ready.json.
Only writes TFC_TOKEN into the runtime vault bundle when either:
the vault file already exists, or
you pass an explicit vault password source (--vault-password-file/--vault-password-command).
Does not create Terraform Cloud workspaces (workspace lifecycle is handled separately).
Allows hyops runner blueprint ... to project Terraform Cloud auth into the remote runner job automatically when a valid local runtime token or Terraform CLI credentials file exists.

Notes: - <root> is ~/.hybridops/envs/<env> when you pass --env <env> (recommended). - If you are running with local Terraform state (HYOPS_TERRAFORM_BACKEND_MODE=local), you can skip Terraform Cloud init.

Preconditions and safety checks¶

Confirm required tools exist:

command -v terraform curl

Steps¶

1) Create config template (first run) - Command:

     hyops init terraform-cloud --env dev

Expected result:
- If config is missing, HyOps writes a template and exits with code 10.
- HyOps prints the path to edit and the exact re-run command.

2) Populate config - File: <root>/config/terraform-cloud.conf - Required: TFC_ORG - Optional: TFC_HOST (Terraform Enterprise), WORKSPACE_PREFIX, TFC_CREDENTIALS_FILE

3) Run init (workstation path) - Option A (you already ran terraform login previously):

     hyops init terraform-cloud --env dev

Option B (allow HyOps to invoke terraform login):

hyops init terraform-cloud --env dev --with-cli-login

If you add --force on an existing environment, HyOps now reviews the current effective Terraform Cloud defaults before continuing. That includes values already present in terraform-cloud.conf, not only values detected from the local Terraform CLI state.

4) Optional: persist token into the runtime vault bundle This is useful when you want HyOps to keep TFC_TOKEN alongside other per-environment secrets.

Prereq (once per workstation):

 hyops vault bootstrap
 hyops vault status-verbose

Then (explicitly request vault persistence for the environment):

 hyops init terraform-cloud --env dev --vault-password-command 'hyops vault password'

5) Optional: create/ensure Terraform Cloud workspaces headlessly (no GUI) Use this when you want deterministic workspace creation before first terragrunt init/apply.

Direct workspace name:

 hyops tfc workspace-ensure platform-gcp-dev-org--gcp--project-factory-gcp-org-00-project-factory-v1-0 \
   --org hybridops-tech --execution-mode local

Derived workspace name (recommended, keeps dev/shared/... namespaces explicit):

 hyops tfc workspace-ensure \
   --org hybridops-tech --execution-mode local \
   --provider gcp \
   --module-ref org/gcp/project-factory \
   --pack-id gcp/org/00-project-factory@v1.0 \
   --env dev \
   --workspace-prefix hybridops

Shared environment example:

 hyops tfc workspace-ensure \
   --org hybridops-tech --execution-mode local \
   --provider proxmox \
   --module-ref platform/onprem/platform-vm \
   --pack-id onprem/proxmox/20-platform-vm@v1.0 \
   --env shared \
   --workspace-prefix hybridops

Verification¶

Readiness marker exists:

cat ~/.hybridops/envs/dev/meta/terraform-cloud.ready.json
Terraform credentials file exists and is locked down:

ls -la ~/.terraform.d/credentials.tfrc.json
Optional: verify a workspace exists and is configured without using the TFC GUI:

hyops tfc workspace-ensure \ --org hybridops-tech --execution-mode local \ --provider gcp \ --module-ref org/gcp/project-factory \ --pack-id gcp/org/00-project-factory@v1.0 \ --env dev \ --workspace-prefix hybridops
Optional: confirm runner-dispatched jobs can reuse this init without a second remote login:

hyops runner blueprint preflight --env dev \ --runner-state-ref platform/linux/ops-runner#gcp_ops_runner_bootstrap \ --file "$HOME/.hybridops/envs/dev/config/blueprints/gcp-ops-runner.yml"
If you enabled vault persistence, verify TFC_TOKEN is present in the vault bundle:

ansible-vault view \ --vault-password-file <(hyops vault password) \ ~/.hybridops/envs/dev/vault/bootstrap.vault.env | grep '^TFC_TOKEN='

Troubleshooting¶

`ERR: TFC_ORG is required`¶

Fix: Set TFC_ORG in <root>/config/terraform-cloud.conf and re-run init.

`ERR: Terraform Cloud token not available`¶

Fix:

Run interactive login:

terraform login app.terraform.io
hyops init terraform-cloud --env dev

Or let HyOps run it:

hyops init terraform-cloud --env dev --with-cli-login

`runner remote hyops command failed` with Terraform Cloud token hint¶

Cause: the selected runtime has no usable local Terraform Cloud auth for runner dispatch to project into the remote job.
Fix:

hyops init terraform-cloud --env dev --with-cli-login

Then re-run the runner-dispatched blueprint/module command.

`ERR: terraform login requires a TTY`¶

Cause: you ran with --with-cli-login in a non-interactive context.
Fix: run terraform login manually in an interactive shell, then re-run hyops init terraform-cloud.

`--with-cli-login --force` keeps the wrong host, org, or workspace prefix¶

Meaning: the environment already had values in terraform-cloud.conf, and you want to repair them instead of silently carrying them forward.
Current behaviour:
HyOps now pauses and asks you to keep or override the effective values for:
- TFC_HOST
- TFC_ORG
- WORKSPACE_PREFIX
This review step applies to values that came from the existing config file as well as values derived from the local Terraform CLI state.
Recommended repair path:

hyops init terraform-cloud --env dev --with-cli-login --force

`WARN: vault persistence skipped: ...`¶

Cause: you did not explicitly request vault persistence and no vault bundle exists yet under <root>/vault/.
Fix: re-run with an explicit vault password source:

hyops init terraform-cloud --env dev --vault-password-command 'hyops vault password'

`Terraform Cloud workspace binding mismatch for module state slot ...`¶

Cause: the same HyOps module state slot (same module + state_instance) is now deriving a different Terraform Cloud workspace/backend than the prior successful run.
Common causes:
changed --env / context_id
changed WORKSPACE_PREFIX / name_prefix
changed TFC_ORG
switched backend mode (cloud vs local)
Note: HyOps already preserves the known compatibility cases for legacy workspace names, including the historical hybridops-* to platform-* prefix migration and the shortened formatter introduced for long names. If you still see this error, the derived workspace is materially different.
Fix (recommended): restore the original naming/backend inputs so the derived workspace matches the existing state slot.
Intentional migration only:

HYOPS_ALLOW_BACKEND_BINDING_DRIFT=1 hyops apply --env --module --inputs

Then import/reconcile Terraform resources before continuing normal operations.

`state lock was held by another run` or destroy/apply stops on a stale Terraform Cloud lock¶

Cause: a prior Terraform/Terragrunt run exited without releasing the remote lock.
First confirm you are targeting the correct module slot and workspace derivation. HyOps already preserves the known compatibility cases for legacy workspace names, so a lock on an older workspace name can still be the expected binding for that slot.
Preferred recovery path:

hyops state unlock --env \ --module \ --state-instance \ --inputs \ --lock-id \ --force
Then rerun the original preflight, validate, apply, or destroy command.

`workspace policy non-fatal (workspace_not_found): workspace not found (it may not be created yet)`¶

Cause: Terragrunt workspace policy checked execution mode, but the workspace does not exist yet.
Fix (headless, no GUI):

hyops tfc workspace-ensure \ --org hybridops-tech --execution-mode local \ --provider \ --module-ref \ --pack-id provider/...@vX.Y \ --env \ --workspace-prefix hybridops

Re-run the HyOps module/blueprint after workspace creation.

Post-actions and clean-up¶

Record the init run_id and run-record path from the command output.
If rotating tokens, revoke old tokens after verifying the new token works.

Initialise Terraform Cloud credentials¶

Context¶

Preconditions and safety checks¶

Steps¶

Verification¶

Troubleshooting¶

ERR: TFC_ORG is required¶

ERR: Terraform Cloud token not available¶

runner remote hyops command failed with Terraform Cloud token hint¶

ERR: terraform login requires a TTY¶

--with-cli-login --force keeps the wrong host, org, or workspace prefix¶

WARN: vault persistence skipped: ...¶

Terraform Cloud workspace binding mismatch for module state slot ...¶

state lock was held by another run or destroy/apply stops on a stale Terraform Cloud lock¶

workspace policy non-fatal (workspace_not_found): workspace not found (it may not be created yet)¶