ADR-014: EmbeddingPort and self-hosted BGE-M3 ONNX embedding model#

Status#

Accepted

Context#

mode: "hierarchical" (#124) analyses corpora larger than the single-call token cap by clustering feedback records into thematically coherent chunks before a map-reduce over the LLM. Clustering needs a vector per record, so we need a text-embedding model. The model is the one genuinely external, swappable dependency in the new path; everything downstream of the vectors (clustering, the coding-trend table) is deterministic in-process computation.

Constraints from the domain: feedback is multilingual, so the model must be multilingual (a monolingual model clusters by language, not theme); the service runs CPU-only; and we must not weaken our data-handling posture (anonymisation runs before embedding, and the model must not be able to execute arbitrary code or exfiltrate data).

Decision#

  1. Introduce a single new driven port, EmbeddingPort (qfa.domain.ports), a synchronous typing.Protocol: embed(texts: tuple[str, ...]) -> tuple[tuple[float, ...], ...]. Synchronous because encoding is CPU-bound local computation, not I/O (contrast LLMPort.complete). Domain-pure (plain floats, no numpy in the signature).

  2. Implement it with BgeM3OnnxEmbedder (qfa.adapters.embedding), running BGE-M3, ONNX, int8, in-process via onnxruntime, dense-1024-d output only. The adapter explicitly inherits EmbeddingPort per AGENTS.md.

  3. Artifact provenance: a community pre-built ONNX-int8 build, pinned by revision hash, mirrored into our own artifact store, and loaded from that mirror — never fetched from HuggingFace at runtime in production. Validated once against official BAAI/bge-m3 (cosine >= 0.999) to catch a botched/altered conversion.

  4. Security flags asserted at construction: trust_remote_code=False; no custom-operator libraries registered; keep onnxruntime patched. A standard-op ONNX graph cannot execute arbitrary code or perform I/O (unlike a pickle .bin checkpoint), so the residual surface is parser CVEs (patching) and conversion correctness (the one-time validation).

  5. Concurrency: a single batched session.run() with intra_op_num_threads left at the core count. No thread/process pool.

  6. Clustering and the coding-trend table get no port — they are deterministic services logic with nothing external to swap, unit-tested with hand-built vectors and zero mocks.

Why BGE-M3 specifically#

Multilinguality is the hard requirement, and it is where most strong embedding models fall short — many are English-centric and would cluster our feedback by language instead of theme. BGE-M3 embeds 100+ languages into one shared vector space and is competitive on multilingual retrieval benchmarks at a size we can run on CPU. We use only its dense 1024-d output; its sparse and multi-vector (ColBERT) heads are unused. The official weights are MIT-licensed, which keeps the self-conversion escape hatch (Option C) open.

Why ONNX int8 (not PyTorch / fp32)#

Running the ONNX graph through onnxruntime means no PyTorch at runtime — we avoid the ~2 GB torch toolchain in the image and infer on CPU directly. int8 quantisation shrinks the artifact to ~558 MB (from ~2.2 GB fp32) and speeds up CPU inference, at roughly ~1% recall loss — acceptable here because clustering depends on relative similarity between records, not absolute precision. The format also underpins the security property in point 4: a standard-op ONNX graph is a pure math graph that cannot execute code or perform I/O, unlike a pickle .bin checkpoint that runs arbitrary code on load.

Options considered#

A. EmbeddingPort + self-hosted ONNX adapter (chosen)#

  • Pro: keeps the heavyweight model out of the application ring; the port preserves the option to externalise the model later without touching services. CPU-only, no torch at runtime, <800 MB footprint.

  • Pro: self-hosting avoids sending feedback to a third-party embedding API — consistent with the anonymise-before-LLM posture.

  • Con: we own the mirroring + one-time validation as an ops concern.

B. Hosted embedding API (e.g. provider endpoint) — rejected#

  • Con: sends (even anonymised) feedback off-box for a task we can do in-process; adds an I/O dependency and per-call cost for no quality win at our scale.

C. Self-convert official MIT weights via optimum up front — deferred#

  • The artifact source lives entirely behind the port, so self-conversion is a build/ops change, not an architecture change (same code). It is build-expensive (re-introduces the ~2 GB torch toolchain, ~2.2 GB download, ~4-6 GB peak RAM) for no runtime benefit, so it is documented as a later, compliance-driven option only.

Consequences#

  • qfa.domain.ports gains exactly one new driven port; the driving/driven split is preserved (ADR-011).

  • qfa.api.app is the only place that constructs the adapter (allowlisted in the import-linter layers contract, like the other composition-root imports).

  • Deployments that do not use mode=hierarchical set no EMBEDDING_* variables; the orchestrator gets embedder=None and hierarchical requests degrade to a 502 analysis_unavailable rather than requiring a model on disk.

  • A future externalised or self-converted model is a drop-in: a new adapter behind the same port, no services change.

Addendum (2026-06-01): configurable model family, e5-base default#

Embedding latency (CPU-bound encoding) is the dominant cost on the hierarchical path for the typical short-text corpus, so the adapter was generalised from BGE-M3-only to a configurable family without changing the port or services:

  • BgeM3OnnxEmbedder became OnnxEmbedder (qfa.adapters.embedding), still explicitly inheriting EmbeddingPort. The only behavioural fork is output handling: pooling="pre_pooled" (BGE-M3’s already-pooled dense_vecs) vs pooling="mean" (mean-pool token-level last_hidden_state over the attention mask, for E5), plus an optional query_prefix (query: for E5). The security posture (points 4–5 above) is unchanged and written once.

  • EMBEDDING_MODEL_KIND selects the family; EMBEDDING_DENSE_DIM and EMBEDDING_MAX_TOKENS are per-artifact knobs (so e5-base 768-d and e5-small 384-d share kind=e5). The dimension is validated per batch — a mismatched artifact/config fails loud.

  • The default model is now intfloat/multilingual-e5-base (768-d, mean-pooled): ~2–3× faster CPU encoding than BGE-M3 at a modest cross-lingual quality trade, which is the right default for the short multilingual feedback this service sees. It is the official repo’s own int8 ONNX export, pinned by hash and validated the same way (point 3). BGE-M3 is no longer baked into the image: deployments that prefer its stronger cross-lingual quality add a fetch step to the Dockerfile model stage and override the EMBEDDING_* env.

The conversion-correctness check (point 3) now applies per artifact; the e2e cosine-vs-official test still covers BGE-M3 via the retained build_bge_m3_embedder entry point.

Participants#

Marius