ADR-020: mistral-medium-3-5 as the judge model#

Status#

Accepted

Context#

#258 made the judge connection configurable independently of the generation model (JUDGE_LLM_*, settings reference), but left it unset everywhere: judge calls ran on the primary model, azure/gpt-5.4, in every deployed environment. #259 is the follow-up decision — which model the judge connection should actually point at.

The product owner directed mistral-medium-3-5, already deployed on the existing Azure AI Foundry resource. Per src/qfa/resources/model_prices.yaml, it is materially cheaper than the incumbent: $1.50/1M input, $7.50/1M output, versus $2.50/1M and $15.00/1M for azure/gpt-5.4 — roughly 1.7× cheaper on input, 2× cheaper on output. Judge calls happen at four call sites, one judge per generation call on the analyze and summarize paths, so this is a material share of total LLM spend, not a rounding error.

Decision#

azure_ai/mistral-medium-3-5, on the existing Azure Foundry resource, serves all four judge call sites in every deployed environment: the analyze judge, the hierarchical leaf judge, and the judges in summarize and summarize_bulk. Configured via JUDGE_LLM_MODEL + JUDGE_LLM_API_BASE (var.judge_llm_model, var.judge_llm_api_base in infra/variables.tf), no new credential — the judge connection inherits LLM_API_KEY from the primary connection, same Azure Foundry resource, same trust boundary.

How it was verified#

Not yet run. This decision is shipped without live evaluation evidence: no automated evaluation harness exists in this repo, and no environment available at implementation time held Azure credentials to run one against the real deployment. The rollout below is staged specifically to catch a bad judge model with limited blast radius in the absence of that evidence, and the rollback conditions are stated as objective log/metric signals a human can act on without needing a benchmark.

Evidence#

Limitations#

  • No offline or live evidence that mistral-medium-3-5 discriminates good judge inputs from bad ones as reliably as azure/gpt-5.4 did. The staged rollout and rollback conditions below are the mitigation for shipping without that evidence, not a substitute for it.

  • The two free-text judge sites (summarize_bulk, summarize) parse a bare float and raise AnalysisError on anything else — a weaker or differently-tuned model is more likely to break that contract than the two structured sites. See the follow-up issue below.

Rollout#

dev → staging → prd, one environment at a time via the per-environment AZ_JUDGE_LLM_API_BASE GitHub Actions variable and a terraform apply, each soaking at least one working day before promoting to the next environment.

Rollback#

Set the environment’s AZ_JUDGE_LLM_MODEL GitHub Actions variable (and AZ_JUDGE_LLM_API_BASE, if set) to "" and re-apply. Because JUDGE_LLM_MODEL unset means “use the primary client” (src/qfa/settings.py), rollback is a config change with no code change and no redeploy of the image — the cheapest rollback the JUDGE_LLM_* mechanism allows.

Rollback conditions#

Stated as observables, not feelings:

  1. Any AnalysisError with "LLM judge returned invalid quality score" in the logs — a failed /v1/summarize or /v1/summarize-bulk request caused by the judge, at a site that is supposed to degrade, not fail.

  2. A sustained rise in the 5xx alert on the summarise routes (infra/observability.tf).

  3. A sustained rise in "Analyse judge call failed" warnings, i.e. quality_score coming back null on /v1/analyze.

  4. SELECT model, count(*), sum(cost_usd) FROM llm_calls WHERE model = 'azure_ai/mistral-medium-3-5' GROUP BY model returning sum(cost_usd) = 0 — pricing didn’t register and cost attribution is silently broken. tests/scripts/test_infra_judge_config.py guards this at build time; this catches a deployment-time mismatch it can’t see (e.g. a Foundry deployment renamed after this ADR shipped).

When to revisit#

  • Any rollback condition above fires.

  • An evaluation harness is built and produces evidence that contradicts this decision.

  • The follow-up issue below lands and changes the failure mode at the two free-text sites.

Follow-up#

Opened as #299: the two free-text judge sites (summarize.py’s summarize_bulk and summarize) parse a bare float and raise AnalysisError on anything else, unlike the two structured analyze judge sites which degrade to quality_score=None. Converting them to a structured response_model is proven feasible against Azure AI Mistral by _provider_safe_response_format (src/qfa/adapters/llm_client.py). Not done in this change.

Participants#

Olaf