Deployment#

A short overview of how the service runs in production. For provisioning, see Infrastructure bootstrap and Set up a new environment.

Runtime topology#

The service runs as a single container on Azure App Service, one App Service plan per environment (dev, staging, prd). The container is built from the Dockerfile in the repo root and pushed to the shared Azure Container Registry by CI.

Environment

Plan SKU

vCPU

RAM

dev

B2

2

3.5 GiB

staging

B2

2

3.5 GiB

prd

P0v3

1

4 GiB

These values live in var.app_service_plan_sku_by_env (infra/variables.tf) and are resolved per Terraform workspace by local.app_service_plan_sku; a workspace absent from the map falls back to B2. Why prd differs, and when to escalate it to P1v3, is recorded in ADR-019. Note that the container runs a single gunicorn worker, so exactly one ONNX embedding session is resident per instance — raising the worker count would multiply that memory floor, so it is not a way to spend prd’s extra RAM. Resizing a plan restarts the app; the procedure is in Operational how-tos § Resize the App Service plan.

        flowchart LR
    user[Caller<br/>EspoCRM, etc.]
    appsvc[Azure App Service]
    user -->|HTTPS| appsvc
    appsvc -->|managed identity| kv[Key Vault]
    appsvc -->|AAD token or password| pg[(PostgreSQL)]
    appsvc -->|HTTPS| llm[LLM provider<br/>via LiteLLM]

Container lifecycle#

entrypoint.sh runs two things, in order:

  1. python -m qfa.cli.migrate — applies any pending Alembic migrations. Uses a Postgres advisory lock (pg_advisory_lock(LOCK_KEY)) scoped to the connection, so concurrent replicas wait for one migrator to finish and a crashed migrator’s lock auto-releases when its connection closes.

  2. gunicorn qfa.main:app --worker-class asgi — binds the HTTP server. No -w, so one worker.

Putting migrations before the server means the App Service health probe doesn’t see a half-migrated database. The trade-off: container start time grows with migration time, so keep migrations small.

Database authentication#

Two modes, selected by DB_AUTH_MODE:

  • password — DB_PASSWORD is read from settings. Simple; suitable for local dev.

  • entra — the SQLAlchemy connection acquires an AAD access token via _AadTokenProvider, caching it and refreshing 120s before expiry. The App Service system-assigned managed identity must be granted the PostgreSQL role (configured by Terraform). This is the production default.

Secrets#

Secrets reach the App Service via Key Vault references, not as plain environment variables:

Secret

Used by

llm-api-key

LLMSettings.api_key

llm-api-base

LLMSettings.api_base

auth-api-keys

AuthSettings.api_keys (parsed as JSON)

Seeding is described in Set up a new environment § 6. Rotation of auth-api-keys is described in API key management.

CI / CD#

Infrastructure changes:

  • PR touching infra/ → CI runs terraform plan automatically.

  • Merge to main → trigger terraform apply manually from the Actions tab.

Application changes:

  • Merge to main → CI builds the container, pushes to ACR, and deploys to the target App Service. Branch protection and deployment gating are configured per environment in the GitHub Actions workflow.

Recovering from common situations#

  • Migration is stuck. Check pg_locks for the advisory lock; if it’s held by a dead session, the lock will release on connection close (a few seconds at most). If it doesn’t, manually kill the holding backend with pg_terminate_backend(pid).

  • Managed identity recreated (after terraform destroy/rebuild) — re-run steps 4 and 5 of Set up a new environment for the affected environment to refresh AZ_CLIENT_ID.

  • Usage tracking is broken but app must keep serving requests. No action is needed to keep core operations running: TrackingLLMAdapter logs recording failures but never raises, so analysis continues even when the database is unreachable — the only impact is dropped usage records and 503s from the /v1/usage* endpoints. Note there is no switch to disable the database: a connection is required for the app to boot (see settings reference), so a totally misconfigured DB blocks startup rather than degrading gracefully.