qfa.services.clustering#

HDBSCAN clustering + token-budget chunking for hierarchical analysis.

Deterministic services logic (no port): turns dense embedding vectors into a set of Chunk objects for the map step. HDBSCAN needs no preset cluster count, no fixed eps, and labels outliers as noise (-1).

Two invariants, both unit-tested:

  1. Full coverage — the union of all chunk records equals the input set; no record is dropped (outliers go into uncategorised chunks).

  2. Budget — no returned chunk exceeds max_total_tokens; an over-budget group is split into budget-sized sub-chunks.

Functions

cluster_records(*, records, vectors, ...[, ...])

Cluster records by their embedding vectors into budget-sized chunks.
qfa.services.clustering.cluster_records(*, records: tuple[FeedbackRecordModel, ...], vectors: tuple[tuple[float, ...], ...], min_cluster_size: int, max_total_tokens: int, chars_per_token: int, metric: str = 'euclidean', target_chunk_tokens: int | None = None) → tuple[Chunk, ...][source]#

Cluster records by their embedding vectors into budget-sized chunks.

Parameters:

  • records (tuple[FeedbackRecordModel, ...]) – The records to cluster (same order/length as vectors).

  • vectors (tuple[tuple[float, ...], ...]) – Dense embedding vector per record.

  • min_cluster_size (int) – HDBSCAN min_cluster_size.

  • max_total_tokens (int) – Per-chunk token ceiling — the hard limit of what one LLM call can hold. No returned chunk ever exceeds it.

  • chars_per_token (int) – Char-to-token conversion ratio for the budget estimate.

  • metric (str) – HDBSCAN distance metric (default euclidean).

  • target_chunk_tokens (int | None) – Desired chunk granularity, decoupled from the ceiling. HDBSCAN clusters are uneven, so a dominant theme can fit the ceiling whole and become one fat, slow map call. When set, a cluster larger than this is split into roughly equal sub-chunks. The effective split budget is min(target_chunk_tokens, max_total_tokens), so the ceiling always wins. None keeps the old behaviour (split only at the ceiling).

  • their (Records within every chunk are always ordered chronologically by)

  • last) (created metadata (undated records)

  • time (so a chunk reads as a)

  • time-windows. (series and a split cluster yields contiguous)

Returns:

Chunks whose records partition the input exactly. Noise points are collected into uncategorised chunk(s) with label == -1.

Return type:

tuple[Chunk, …]

Raises:

ValueError – If records and vectors differ in length.