Operations Runbook

Operational procedures for the AW Chat backend: deploy, rollback, backup/restore, monitoring, and incident response.

Topology (template)

Replace hostnames with your environment:

Role	Example	Notes
API	https://chat-api.example.com	@aw-chat/server — /api/*, /admin/* API, /webhooks/intercom, /files/*
Admin SPA	https://chat-admin.example.com	@aw-chat/admin (nginx); proxies or CORS to the API
Optional demo	https://chat-demo.example.com	examples/web for QA — not required for production apps
Database	Managed PostgreSQL	Migrations on API start
Object storage	S3-compatible	Required in production
Upstream	Intercom	Support engine for chat/tickets

Host mobile/web apps embed the SDK and call only the API origin.

Deploy

1. Promote a known-good build/image (CI in this repo: lint → typecheck → test → build).
2. Container starts under tini (PID 1); drains on SIGTERM (graceful ~10s).
3. Post-deploy smoke:
   • GET /health → 200
   • GET /ready → 200 with { db: true, … }
   • Admin login → session token; load CSAT/complaints
   • Admin origin security headers (CSP, X-Frame-Options) if served via nginx

Rollback

API runs from source in the image — rollback = redeploy a prior commit/image:

git revert <bad-sha>   # preferred when you control the git deploy path
# or redeploy the previous image tag from your platform

Migrations are forward-only. A schema change is not undone by a code rollback; restore from backup if you must reverse data shape.

Database

Migrations

• Generate: pnpm --filter @aw-chat/server db:generate
• Apply: automatic on start, or db:migrate
• Never hand-edit an applied migration file.

Backup

pg_dump --format=custom --no-owner "$DATABASE_URL" > awchat-$(date +%F).dump

Schedule daily; store off-host.

Restore

createdb awchat_restore
pg_restore --no-owner --dbname awchat_restore awchat-YYYY-MM-DD.dump

Validate, then repoint DATABASE_URL.

Retention

Append-only analytics/audit (post_button_clicks, admin_logs) prune only when DATA_RETENTION_DAYS is set. Intercom data follows Intercom’s retention. Comments / CSAT / complaints are not auto-deleted — see Data model.

Monitoring

Watch for:

• /ready failing (DB) or open Intercom circuit breaker
• Log markers: [shutdown], [db] idle client error, [unhandledRejection], [storage] S3 is not configured…, [error] code=… at=METHOD /path
• Intercom 429s — workspace quota is the capacity ceiling

Alert on readiness failures, restarts, and error-rate spikes.

Secrets

• Inject at runtime; never bake into images.
• Rotating JWT_SECRET invalidates all client sessions.
• Rotating ADMIN_SESSION_SECRET invalidates admin sessions.
• Rotating FILE_URL_SECRET breaks existing public file links.
• Intercom / S3 / FCM / Telegram: rotate at the provider, update env, redeploy.

Known follow-ups (hardening backlog)

• Pin CI Actions to commit SHAs; expand SAST/secret scanning.
• Compile server to JS so the final image can drop the TS toolchain.
• Before multi-replica: Redis for cache/rate-limit/lockout; single scheduler for channel publish.