CloudFront + OAC cut-over for a Lambda Function URL, dev-only, with deploy gates and rollback

What they did

The author prepared an AWS cut-over that puts CloudFront with Origin Access Control in front of a Lambda Function URL using AWS_IAM auth, and enabled it for dev only while staging and prod stay on enable_cdn = false. A bare enable_cdn = true flip would have broken several things, so the PR fixes each: the frontend module now forwards x-amz-content-sha256 and stops forwarding Authorization (dead config under signing_behavior = "always" that only widened the cache key); the custom_error_response 404 -> 200 /index.html rewrite is removed because it turned JSON API 404s into HTML 200s and the server already does SPA fallback; a new aws_lambda_permission.function_invoke_cloudfront grants cloudfront.amazonaws.com both lambda:InvokeFunctionUrl and lambda:InvokeFunction scoped to the distribution ARN. Deploy gates were reworked: test-deployment reads the URL AuthType, resolves the distribution domain from the cloudfront_distribution_id job output, fails loudly if missing, probes the CloudFront URL and asserts the raw URL returns exactly 403; rollback.yml probes frontend_url. Client IP handling behind the CDN was fixed in internal/server/source_ip.go via clientip.Resolve, trusting X-Forwarded-For only when CUDLY_TRUSTED_PROXY_HOPS is set, which Terraform sets to 1 only when enable_cdn = true, and trailing XFF entries equal to the peer are dropped first. Tests were added at env and module level, and mutation probes (OAC detached, Authorization restored, duplicate statement id, InvokeFunctionUrl swapped in, auth forced to NONE) each made a test fail. A guard test pins the 26 CDN apply actions the deploy role needs, with new statements in policy_compute_b.tf because policy_compute.tf hit the 6144-character limit. Docs were updated to use terraform output -raw frontend_url.

How it went

Local validation was clean: terraform fmt, init -backend=false and validate on environments/aws; terraform test with 15 passed in environments/aws and 1 in modules/frontend/aws; go test on the aws environment package; actionlint on both workflows. No terraform plan ran against the real backend because no credentials were available, so the first dev plan in CI is the real check. Nothing was applied; a human must first apply ci-cd-permissions for dev, then merge, deploy dev, and immediately set the dev DASHBOARD_URL secret to the CloudFront frontend_url and redeploy. A few minutes of 403 on dev during the apply is accepted.

Worth knowing

The cut-over has a known outage window: in one apply the URL flips to AWS_IAM and the public permission is removed before the distribution and CloudFront permissions exist. Between deploy and the DASHBOARD_URL update, the OIDC issuer and JWKS still point at the raw URL, so federated token verification fails until redeploy, and any Azure/GCP workload identity federation trusting the old issuer must be re-registered. Machine API clients calling the raw Function URL with an API key break and must switch to X-Authorization plus x-amz-content-sha256. Rollback via rollback.yml with disable_cdn=true extends the outage by the CloudFront disable time and can leave the URL closed if the distribution delete fails. Open maintainer decisions remain on custom domains and on splitting the flip across dev, staging and prod.

Wiring

Terraform environments/aws drives the change: enable_cdn = true in dev tfvars feeds the compute module (Function URL auth type) and the frontend module (CloudFront distribution, OAC, forwarded headers). The distribution uses the Function URL as origin, so the URL must exist first; the auth type derives from enable_cdn, which is why a single apply has a brief 403 window. The new aws_lambda_permission.function_invoke_cloudfront (statement id FunctionInvokeAllowCloudFront, principal cloudfront.amazonaws.com, scoped to the distribution ARN) is required alongside the existing invoke permission. CI workflows: test-deployment reads the AuthType, pulls cloudfront_distribution_id from job outputs, probes the CloudFront URL and asserts the raw URL returns 403; rollback.yml takes disable_cdn=true, applies with -var=enable_cdn=false, then requires AuthType NONE and raw /health 200. The deploy role needs the 26 CDN actions pinned by cdn_apply_permissions_test.go, added in policy_compute_b.tf. Runtime: CUDLY_TRUSTED_PROXY_HOPS=1 set by Terraform only when enable_cdn = true, consumed by clientip.Resolve in internal/server/source_ip.go; CUDLY_ISSUER_URL is unset so resolveOIDCIssuerURL falls back to DASHBOARD_URL, which must be updated to the CloudFront frontend_url after the first apply.

Try it yourself

In this repo, prepare a dev-only cut-over of our Lambda Function URL to CloudFront with Origin Access Control and AWS_IAM auth, leaving staging and prod on enable_cdn = false. Do the following: (1) in the frontend module, forward x-amz-content-sha256 and stop forwarding Authorization; (2) remove the custom_error_response 404 -> 200 /index.html rewrite so JSON API 404s stay JSON; (3) add an aws_lambda_permission for cloudfront.amazonaws.com granting both lambda:InvokeFunctionUrl and lambda:InvokeFunction scoped to the distribution ARN; (4) update test-deployment to read the Function URL AuthType, resolve the distribution domain from the cloudfront_distribution_id job output, fail loudly if missing, probe the CloudFront URL and assert the raw URL returns exactly 403; (5) fix client IP resolution in internal/server/source_ip.go to trust X-Forwarded-For only when CUDLY_TRUSTED_PROXY_HOPS is set, and set that to 1 in Terraform only when enable_cdn = true; (6) add env and module tftests asserting AWS_IAM, OAC on every origin, forwarded headers, and distinct statement ids; (7) add a guard test pinning the CDN apply permissions the deploy role needs. Run terraform fmt -check -recursive, terraform validate, terraform test and go test, and report what could not be checked without credentials.