AWS reference architecture

This guide describes the customer-controlled AWS deployment used for an Overlay pilot. Complete Docker On EC2 first when the goal is a quick connectivity and product sanity check.

Do not turn the EC2 sanity host or the disposable AWS rehearsal stack into production by changing its name. Production infrastructure must follow the customer’s networking, security, backup, observability, and change-control standards.

Reference architecture

Users
  -> customer DNS and HTTPS
  -> WAF or approved edge
  -> Application Load Balancer
  -> ECS Fargate web service (2+ tasks)
       -> self-hosted convex-backend service (realtime + functions)
       -> RDS/Aurora PostgreSQL: Better Auth data
       -> ElastiCache Redis over TLS
       -> private encrypted S3 bucket
       -> approved OIDC provider and model gateway

ECS convex-backend service (1 active task + standby)
  -> RDS/Aurora PostgreSQL in the same AZ (Convex persistence engine)
  -> S3 buckets for files, modules, exports, snapshot imports, and search
  -> internal ALB on ports 3210 (client/realtime) and 3211 (HTTP actions)

Customer CI -> ECR immutable digest -> one-off migration -> all ECS roles
Secrets Manager -> ECS task secret references and BYOK provider-key records
CloudWatch, Container Insights, and SNS -> logs, metrics, alarms, and operators

Convex is the application backend. The self-hosted convex-backend service executes functions, realtime sync, scheduled jobs, and durable work; there are no separate worker or scheduler services. Postgres remains only as the Convex backend’s persistence engine and as Better Auth’s separate auth database — the app never runs SQL against application data. Deploy and operate the backend per Self-hosted Convex backend.

Required ownership

Before provisioning, name the owners for:

AWS account, Region, VPC, quotas, tags, and budget.
Source repository, protected branches, CI/CD, and release approval.
DNS, ACM certificates, ingress, WAF, and outbound egress.
RDS, Redis, S3, KMS, backups, restore tests, and retention.
Google Workspace or other OIDC application and pilot users.
Secrets Manager values and rotation.
CloudWatch alarms and the incident notification destination.
Model gateway and every optional external processor.
Pilot acceptance, rollback, and go-live.

Use a dedicated nonproduction account when possible. Human access should be federated, named, MFA-protected, time-limited, least-privilege, and audited by CloudTrail. CI should use GitHub Actions OIDC or CodeBuild roles, not static AWS access keys.

AWS resources

Layer	Minimum pilot requirement
Network	At least two Availability Zones; public ALB subnets; private ECS subnets; isolated RDS/Redis subnets; approved NAT, proxy, or VPC endpoints.
Compute	ECS cluster; web service with at least two tasks; convex-backend service (active + standby); one-off migration task.
Database	Encrypted Multi-AZ RDS PostgreSQL or Aurora PostgreSQL writer serving the convex-backend persistence engine, plus a separate auth-scoped database and role for Better Auth; automated backups and restore test. Backend and RDS must be in the same region, ideally the same AZ.
Redis	TLS ElastiCache with fail-closed application behavior; Multi-AZ automatic failover for the target pilot.
Objects	Private encrypted, versioned S3 bucket; block public access; lifecycle, logging, backup/recovery, and deletion policy.
Ingress	HTTPS ALB listener, healthy target group, customer DNS, ACM certificate, and WAF/edge controls required by policy.
Delivery	Customer repository; CI OIDC or CodeBuild; immutable ECR tags; image scanning; protected staging and production environments.
Operations	CloudWatch log groups, Container Insights, dashboard, alarms, SNS destination, RDS metrics, ALB access logs, and defined retention.

Secrets Manager

Create separate paths for staging and production. ECS injects application secrets as environment variables. For BYOK, the application makes narrowly scoped runtime calls to Secrets Manager using its ECS task role; each user API key is a separate opaque secret ARN stored in Convex, never a plaintext database value. Keep non-secret selectors and hostnames in task-definition environment variables. Enable AWS-backed BYOK in the customer runtime configuration:

OVERLAY_PROVIDER_SECRETS=aws-secrets-manager
AWS_REGION=<customer-region>
OVERLAY_AWS_SECRETS_PREFIX=jpgs/overlay/byok
OVERLAY_AWS_SECRETS_KMS_KEY_ID=<customer-kms-key-arn>

Grant the web ECS task role only secretsmanager:CreateSecret, GetSecretValue, PutSecretValue, and DeleteSecret for that prefix, plus kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey on the customer KMS key. Do not grant list, broad read, cross-environment, or wildcard customer secret permissions. At minimum, create entries for:

SESSION_SECRET
SESSION_TRANSFER_KEY
SESSION_COOKIE_ENCRYPTION_KEY
INTERNAL_API_SECRET
INTERNAL_SERVICE_AUTH_SECRET
MCP_CREDENTIAL_ENCRYPTION_KEY
API_KEY_HASH_SECRET
PROVIDER_KEYS_SECRET
HOOKS_TOKEN_SALT
BETTER_AUTH_SECRET
BETTER_AUTH_OIDC_CLIENT_SECRET
BETTER_AUTH_DATABASE_URL
OVERLAY_REDIS_URL
AI_GATEWAY_API_KEY

Add provider credentials only for approved, enabled features. On ECS or EC2, prefer a bucket-scoped task or instance role and omit S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY. For non-AWS S3-compatible storage, configure both keys, store them in Secrets Manager, and rotate them. Do not place any secret in a NEXT_PUBLIC_* variable, source file, task-definition plaintext environment value, build log, document, ticket, email, or chat.

Deployment procedure

Prepare the customer repository

Fork or mirror the public LayerNorm/overlay-web repository into the customer’s GitHub organization. Clone that customer-controlled repository, retain the public repository as upstream, and branch from an approved published release. The example defaults to the tested v0.1.0-rc.1 release.

export CUSTOMER_REPOSITORY_URL="https://github.com/<CUSTOMER_ORGANIZATION>/<CUSTOMER_REPOSITORY>.git"
export OVERLAY_RELEASE="${OVERLAY_RELEASE:-v0.1.0-rc.1}"

git clone "$CUSTOMER_REPOSITORY_URL" overlay-web
cd overlay-web
git remote add upstream https://github.com/layernorm/overlay-web.git
git fetch upstream --tags --force
git checkout -b "release/${OVERLAY_RELEASE#v}-customer.1" "$OVERLAY_RELEASE"
git push -u origin "release/${OVERLAY_RELEASE#v}-customer.1"

Merge only through a reviewed pull request with required checks.

Establish customer-controlled CI access

Configure GitHub Actions OIDC or CodeBuild. Restrict the deployment role to the exact repository, branch/environment, ECR repository, ECS services, task roles, log groups, and approved infrastructure stacks. Restrict iam:PassRole to the named ECS execution and task roles.

Provision infrastructure as code

Use the customer’s approved Terraform, CloudFormation, CDK, or platform catalog. Keep environment values outside application source. Review the generated plan or change set before applying it.Required network flows are:

Internet/edge -> ALB: HTTPS only
ALB -> web tasks: application port only
web tasks -> convex-backend internal ALB: 3210 and 3211
convex-backend -> RDS: 5432 by security-group reference (same AZ)
migration -> RDS: 5432 by security-group reference (Better Auth database)
web tasks -> Redis: TLS port by security-group reference
convex-backend -> S3: VPC endpoints or approved egress
runtime -> Secrets/ECR/CloudWatch: VPC endpoints or approved egress
runtime -> OIDC/model/customer APIs: explicit approved HTTPS destinations

Build and push one immutable image

Build from the reviewed commit in CI, not on a production host:

export AWS_REGION=<REGION>
export AWS_ACCOUNT_ID=<ACCOUNT_ID>
export ECR_REPOSITORY=overlay-web
export GIT_SHA="$(git rev-parse HEAD)"
export IMAGE="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$GIT_SHA"

aws ecr get-login-password --region "$AWS_REGION" | \
  docker login --username AWS --password-stdin \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"

docker buildx build \
  --platform linux/amd64 \
  --file docker/Dockerfile.enterprise-runtime \
  --build-arg NEXT_PUBLIC_CONVEX_URL="https://<CONVEX_BACKEND_CLOUD_ORIGIN>" \
  --build-arg NEXT_PUBLIC_APP_URL="https://<OVERLAY_HOST>" \
  --tag "$IMAGE" \
  --push .

export IMAGE_DIGEST="$(aws ecr describe-images \
  --repository-name "$ECR_REPOSITORY" \
  --image-ids imageTag="$GIT_SHA" \
  --query 'imageDetails[0].imageDigest' \
  --output text)"

test -n "$IMAGE_DIGEST" && test "$IMAGE_DIGEST" != None
printf 'Release %s -> %s@%s\n' "$GIT_SHA" "$ECR_REPOSITORY" "$IMAGE_DIGEST"

Record the digest and use repository@sha256:... in every task definition. Do not rebuild the image between staging and production. NEXT_PUBLIC_* values are inlined at build time, so the baked NEXT_PUBLIC_CONVEX_URL must be the origin web tasks and browsers can both reach — the backend’s internal ALB hostname works when web tasks, operator browsers, and the backend share routable DNS; otherwise use the public backend origin.

Define the ECS roles

All task definitions use the same image digest and configuration family:

Role	Command	Scaling
Web	`node server.js`	At least two tasks behind the ALB.
Convex backend	Official `convex-backend` image	One active task plus a standby; see the self-hosted runbook.
Migration	See the command below	One-off task only; never an ECS service.

Migration command:

NODE_OPTIONS=--conditions=react-server node_modules/.bin/tsx scripts/db/better-auth-migrate.ts

Convex schema and function deployment is a separate npx convex deploy step against the self-hosted backend; see Self-hosted Convex backend.Configure health checks, log groups, CPU/memory, graceful stop time, read-only filesystem exceptions, database pool budgets, and secret ARN references per role.

Snapshot and run migrations once

Record the current image digest, task-definition revisions, schema version, a pre-deploy RDS snapshot, and a npx convex export of the Convex deployment.Run the Better Auth migration task in the private ECS subnets with the direct RDS writer endpoint. Wait for the task to stop, inspect its exit code, and retain its redacted logs. Do not deploy the services when the migration exit code is nonzero. Deploy the Convex revision per the self-hosted runbook and watch the backend logs for Executing Migration n/m.

Deploy the same digest to every service

Register new task-definition revisions containing the recorded digest, then update the convex-backend and web services. For web rolling deployments use:

desiredCount: 2 or more
minimumHealthyPercent: 100
maximumPercent: 200
ALB stickiness: disabled
ALB idle timeout: at least 60 seconds

Wait for every ECS service to become stable and every ALB target to become healthy. The convex-backend service must hold its persistence lease; a promoted standby acquiring the lease is expected HA behavior, not a split brain.

Configure DNS, TLS, and OIDC

Point the approved hostname to the ALB or approved edge. Register the exact production values with the OIDC owner:

Callback: https://<OVERLAY_HOST>/api/better-auth/sso/callback/<PROVIDER_ID>
Logout:   https://<OVERLAY_HOST>
Origin:   https://<OVERLAY_HOST>

Set BETTER_AUTH_URL, trusted origins, JWT issuer/audience, and JWKS URL to that same HTTPS origin. Store only the public client ID in plaintext config; the client secret remains in Secrets Manager.

Qualify staging

At minimum verify:

/api/v1/capabilities, /api/auth/session, and the ALB health path.
OIDC sign-in, refresh, new tab, sign-out, and account deletion.
Chat create, stream, stop, refresh recovery, rename, and delete across two web tasks.
Files, notes, projects, memory, knowledge, and citations when enabled.
Convex cron execution and realtime subscriptions observed in backend logs.
Redis fail-closed behavior and recovery.
RDS failover and convex-backend lease recovery; automatic web recovery.
S3 signed URL expiry, ownership checks, deletion, and reconciliation.
Backup restore to a new database and application connection to the restore.
CloudWatch alarms and delivery to the named operator.
Convex connectivity: web tasks reach the backend on 3210/3211 and no other deployment URL.

Keep the test accounts and data disposable. Run destructive contracts only against a separate contract-test database.

Rollback

Before every release, record the previous image digest and task definitions. When the new schema remains backward-compatible, roll all services back to the previous digest and verify auth, chat, files, and queue state. If the previous runtime is not schema-compatible, do not improvise a down migration. Restore the pre-deploy snapshot into a new database, point the previous release at the restored writer, and preserve the failed database and logs for investigation.

Product updates and customer customizations

Overlay updates never deploy automatically into the customer account:

git fetch upstream --tags
git checkout -b upgrade/overlay-vX.Y.Z main
git merge --no-ff <APPROVED_UPSTREAM_TAG>
# Resolve conflicts, update customer extensions, and run all gates.
git push -u origin upgrade/overlay-vX.Y.Z

Reusable fixes should go upstream. Customer policy, consent, curriculum, and proprietary data-lake logic should remain behind provider interfaces, service contracts, APIs, MCP servers, or extension modules where practical. Every customer customization needs tests, an owner, a data-flow description, an upgrade note, and rollback behavior.

Paste into a coding agent

Use this prompt only after the agent is authenticated through the customer’s approved AWS and source-control mechanisms:

Deploy Overlay to the customer-controlled AWS staging environment by following
docs/self-hosting/install/aws.mdx.

Operating constraints:
- Work only in the approved account, Region, repository, and staging environment.
- Start by reporting `aws sts get-caller-identity`, Git remote URLs, branch, and SHA.
- Use existing customer IaC and security standards. Do not create a parallel VPC
  or bypass approval because it is faster.
- Use federated access and CI OIDC/CodeBuild roles. Never create static AWS keys.
- Never print, echo, retrieve into logs, commit, or return secret values.
- Reference Secrets Manager ARNs from ECS. Return only names and ARNs.
- Keep RDS, Redis, the convex-backend internal ALB, and ECS private. Expose only
  the approved HTTPS ingress.
- Build one immutable image and use its exact digest for migration and web. The
  convex-backend uses the pinned official image; run migrations and the Convex
  deploy once before service rollout.
- Stop and request approval before applying IaC, running migrations, changing
  DNS/OIDC, triggering failover, deleting data, or promoting to production.

Execution checkpoints:
1. Inventory customer inputs, existing infrastructure, quotas, and missing owners.
2. Produce an architecture delta and IaC plan/change set; wait for approval.
3. Validate config and tests, build the reviewed commit in CI, push to ECR, scan,
   and record the immutable digest.
4. Record backup and rollback state; run the one-off migration and verify exit code.
5. Deploy the digest to the convex-backend and web services; wait for stable services.
6. Pause for interactive OIDC setup/login where required.
7. Execute staging product, isolation, recovery, backup, alarm, deletion, and
   Convex-connectivity QA from the guide.
8. Return a redacted deployment report: account/Region, resource IDs, Git SHA,
   image digest, task revisions, schema version, QA results, alarm/restore proof,
   costs, known issues, rollback instructions, and PASS/FAIL. Never return secrets.

Handover gate

The deployment is not handed over until the customer PoC can:

Identify the deployed Git SHA, image digest, schema version, and task revisions.
Redeploy the current digest through CI without changing source.
Find web, convex-backend, migration, RDS, Redis, ALB, and S3 telemetry.
Trigger and receive a test alarm.
Roll back to the previous digest and roll forward again.
Locate backups, restore instructions, secret rotation ownership, and incident contacts.
Revoke or reduce the temporary Overlay operator role while CI continues to work.

For configuration details and provider matrices, continue with Self-Hosting and Security.

Overview

Install

Configure

Operate

Enterprise

Licensing

AWS reference architecture

Reference architecture

Required ownership

AWS resources

Secrets Manager

Deployment procedure

Rollback

Product updates and customer customizations

Paste into a coding agent

Handover gate

​Reference architecture

​Required ownership

​AWS resources

​Secrets Manager

​Deployment procedure

​Rollback

​Product updates and customer customizations

​Paste into a coding agent

​Handover gate

Reference architecture

Required ownership

AWS resources

Secrets Manager

Deployment procedure

Rollback

Product updates and customer customizations

Paste into a coding agent

Handover gate