OpenTofu runner and state

OrkStage provisions AWS with OpenTofu modules in infra/tofu/modules/aws/*, one module per logical resource type. ADR 0005 records the design.

The isolated runnerCopy Anchor Link

• The worker never runs OpenTofu in its own process. Every tofu command runs in a throwaway container: the pinned ghcr.io/opentofu/opentofu:1.12.7 image, or the tofu-runner image with the modules baked in.
• The module library is mounted read only, or baked into the image. Providers come from a plugin cache volume that all runners share.
• Credentials reach that one container only, as environment variables. They never appear in files, command lines or state.
• Each command has a hard timeout (TOFU_TIMEOUT_MINUTES, 45 by default) and can be cancelled.

One state per resourceCopy Anchor Link

The state key is orkstage/<physical-name>.tfstate. There is one state for each resource, not one for each instance or placement slot. This design:

• keeps the platform's own DAG, with progress, retries and failure isolation for each resource;
• limits the damage of a broken state file to one resource;
• passes dependencies as module outputs that become inputs of the next module. Every module emits a kind output, so the adapter wires them by type;
• stores the exact root configuration next to the state (.config.json), so a destroy never depends on a template that has changed since.

Backend: S3, with use_lockfile for locking, and encryption. On real AWS, you create the bucket in advance with versioning and encryption, and OrkStage validates it before use (see Connecting an AWS account). On an emulator, OrkStage creates the bucket on first use.

No secrets in stateCopy Anchor Link

An OpenTofu ephemeral resource generates each RDS master password, and the module writes it through write only attributes (password_wo, secret_string_wo). Services receive the ARN of the Secrets Manager secret as an ECS secret. We verified that the state holds password: null.

Provider plan before applyCopy Anchor Link

In the Generate plan phase, the worker runs a real tofu plan for every existing resource whose desired state changed. It writes a summary for each resource to the deployment's event log:

INFO  [phase:plan] api: 0 to add, 1 to change, 0 to destroy
WARN  [phase:plan] ingress: 1 to add, 0 to change, 1 to destroy; replace aws_lb_target_group.route[0]

• A deletion or replacement makes the deployment high risk. A routine new revision of an ECS task definition (for a new image digest) does not.
• For a production instance, high risk escalates the deployment to AWAITING_APPROVAL, even when the platform's own diff predicted only an update.
• A resource the provider could not plan also escalates a production deployment to approval: nobody approves changes nobody has seen.
• When no change is destructive, the approval step is marked skipped, with the reason.
• The deployment stores the provider plans (plan.providerPlans) for review.

What is approved is what is appliedCopy Anchor Link

Every OpenTofu command runs tofu validate after init, so a configuration error stops before any plan or change.

The plan phase saves each provider plan to a file named after the deployment, in that resource's working directory, next to a hash of the configuration it was made from. The apply phase then runs tofu apply <plan file> instead of planning again:

• If the configuration changed in between (for example a dependency was replaced earlier in the same deployment), or the state changed (someone deployed or edited the resource), a production step fails with a message that says so, and nothing is applied. Deploy again to plan and approve the current changes. Other environments log a warning, plan again and apply.
• The plan file holds attribute values, so it is used once and deleted, whether the apply worked or not. Retries of a failed step plan again.
• Resources this deployment creates have no saved plan: the platform plan already lists them as new.

Previews and drift scans plan in working directories of their own, without the state lock, so they never touch the directory of a running deployment.

Releasing a stale state lockCopy Anchor Link

Locking uses a lock file next to the state in the bucket (use_lockfile). A worker that is killed in the middle of an apply can leave it behind; the next deployment of that resource then fails with Error acquiring the state lock and prints the lock id. Before releasing it, make sure no deployment of that instance is still running. Then, from the worker host:

docker run --rm -it -v orkstage-tofu-work:/workspace -w /workspace/<directory>   -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN   ghcr.io/opentofu/opentofu:1.12.7 force-unlock <lock id>

The directory is the resource's working directory under TOFU_WORK_DIR (its name is printed in the failing step's log), and the credentials are the target's (for AWS, a session of the deployment role). Write down who released which lock and why in the instance's incident or change record: the release itself happens outside OrkStage, so the audit log does not see it.

Drift scansCopy Anchor Link

Every DRIFT_CHECK_INTERVAL_MINUTES (360 by default, 0 turns scans off), the worker plans each applied resource on real providers again, from the snapshot it was deployed with. Any planned action means someone changed the resource outside OrkStage. The resource then becomes DRIFT_DETECTED, OrkStage writes an audit event and sends instance.drift_detected. The scan skips resources with pending changes. A new deployment restores the configuration from the template.

Recovery from deleted resourcesCopy Anchor Link

When someone deletes a resource outside OrkStage, its recorded state outlives it. The AWS adapter has a live existence probe for this case:

• During verification, OrkStage probes any resource that is not healthy. If the resource no longer exists, the step fails at once with Resource no longer exists at the provider. Redeploy to re-create it. instead of waiting.
• Verification waits in one settle window of 5 minutes per deployment, not one per resource, so a slow service does not multiply the wait.
• On the next deployment, the executor asks the provider whether converged resources still exist. It recreates the missing ones instead of skipping them as unchanged.

Verified against MiniStackCopy Anchor Link

The AWS adapter has been tested end to end against MiniStack with: VPC, subnets and security groups, ECR, SQS with a dead letter queue, S3, Secrets Manager, RDS Postgres (real container), an ElastiCache Valkey replication group (real container), an ALB with target groups and listener rules (live data plane at /_alb/<name>/), an ECS Fargate service (real container, registered in the target group and healthy), Route 53 and ACM. The adapter contract suite passes against it.