Vladimir Ivanov

Independent software engineer & researcher

I build and rigorously evaluate retrieval infrastructure for AI coding agents — and I treat the evaluation as a deliverable, not an afterthought.

Senrah on GitHub Read the evaluation Contact via GitHub

Senrah flagship project

Precedent retrieval for AI coding agents. Senrah is an open-source, self-hosted tool that indexes a repository’s merged-PR history — each problem together with the diff that actually solved it — and serves it to any coding agent (Claude Code, Codex, Cursor) over the Model Context Protocol.

The problem. Coding agents have zero context about how your codebase has historically been changed. In a mature or legacy repository, the conventions, workarounds, and edge-case lore live in years of merged pull requests — exactly the knowledge base models were never trained on. So agents guess. Every mainstream context tool embeds what your code is; Senrah embeds how your team has actually changed it, and hands the agent real precedents instead of plausible inventions.

How it works.

ingest merged PRs (resumable, rate-limit-aware)  →  embed problem + solution  →  pgvector (HNSW, your own Postgres)  →  MCP search_prs_v1

The agent asks a question in natural language; Senrah returns the most relevant merged PRs — problem, diff, and a qualitative confidence label — ranked by a weighted problem/solution similarity with ANN oversampling and Python-side re-ranking. Read-only, stateless over the database, never touches your source at read time.

Measured, not demoed

0.90
recall@5
0.79
MRR@10
0.71
recall@1
218
held-out queries

Known-item retrieval on a deduped dotnet/efcore corpus (575 merged PRs; backport clusters scored per-cluster so cherry-picks can’t inflate recall). Manifest hash-pinned; the run is deterministic and frozen.

The honest framing: these numbers measure ranking quality — when a relevant target exists, it ranks near the top. They do not measure forward coverage: whether a new task can find a precedent that genuinely predates it, with no leakage from the future. A leak-free temporal-holdout instrument for that harder question is built and human-calibrated; the full-power number is still open and is tracked, not hidden. Details below.

Evaluation as a first-class artifact

“Retrieval is easy to demo and hard to measure. Most of the engineering effort here went into the measurement, not the retrieval.”

The interesting part of Senrah is not the 0.90 — it is what happened when I refused to stop there. The full account is in docs/EVAL.md, including the negative results. The arc:

  1. Refused to ship the flattering number alone. Known-item recall answers “can it rank a target it is told exists.” It says nothing about coverage for genuinely new work — and on a deeper corpus it can even fall as near-duplicate distractors multiply. A second instrument was required.
  2. Built a leak-free temporal holdout. Corpus strictly merged before a cutoff T, queries strictly after; the split frozen on merged_at and original ingest timestamps (PR bodies are editable after the fact, so current state can’t be trusted); relevance labels frozen together with the corpus boundary; the scorer applies the product’s own below-threshold cutoff so the eval measures what users actually get. The full efcore history was ingested for it: 487 → 8,449 PRs.
  3. Hit a statistical-power failure — and recorded it. With relevance defined by metadata links, only 5 of 278 post-T tasks were answerable, against a pre-registered floor of ≥ 80. Five is noise, not a measurement. But n=5 has two opposite explanations — the label is too strict, or precedents are genuinely rare — and pivoting back to the good-looking number would have hidden the distinction.
  4. Diagnosed it with a labeled A/B probe. Hand-checking 30 random post-T tasks showed a genuine prior precedent existed for ≥ 14/30 (strict) up to ~29/30 (lenient): the metadata label was undercounting true relevance by roughly 14–29×. Verdict: the instrument was rescuable; the label was the disease.
  5. Rebuilt relevance with TREC-style pooling. Per-task judgment pools union three independent retrieval legs — the production embedding retriever, BM25-unique, and a second-family embedder (bge-m3). A blind human gold set of 270 (task, candidate) pairs was labeled before looking at any result: 29 of 30 tasks answerable under judged relevance. Per-leg precision (embedding retriever 0.51, bge-m3 0.12, BM25 0.09) validated the pool — the low-precision legs still contribute 29% of relevant precedents the main leg missed.
  6. The LLM judge failed its pre-registered gate — and stayed failed. The plan to scale labeling with an LLM judge was conditional on Cohen’s κ ≥ 0.6 against human labels. Best result: κ = 0.39 — below the floor (precision 0.77, recall 0.35; the misses concentrated exactly on the semantic and convention-transfer cases the pool was widened to capture). The judge is kept advisory-only and can never override the recall guardrail. A slower, human-anchored result beats a number laundered through a labeler measured to be unreliable.
Judge (framing)κ vs humanVerdict
Sonnet 4.6 (Phase-9 gold)0.27below floor
Opus 4.8 (Phase-9 gold)0.39below floor
Sonnet 4.6 (temporal gold)0.39below floor

Principles held throughout: pre-register the gate before the results exist; label blind before you look; treat the first number as a symptom, not a finding; and report negative results as results — because a measurement you cannot trust is worse than no measurement.

For engineering organizations

Senrah is built for the environment where agent guessing hurts most: large codebases with years of accumulated, company-specific convention. Base models degrade exactly where your history is deepest — but the precedents are the conventions, and retrieval over them is model-agnostic.

Cross-service planning from one index

Senrah indexes any number of repositories into one database, and the search tool takes an optional per-repo filter — so one central, read-only index serves every team’s agents. For tasks that span services, that enables a two-step pattern:

plan query all repos  →  repo spread of hits = impact hint  →  decompose  →  execute per-repo queries

A planning agent queries the whole index — «add order cancellation flow» — and the repository spread of the results shows how the last similar change actually landed: an endpoint PR in gateway, a domain method + migration in orders, an event consumer in notifications. That is a decomposition drawn from your own history, before any code is written. Each sub-task then goes to an agent querying its own repository for house-style precedents. No cross-team ticket conventions required — the signal is semantic similarity of problems, not issue-tracker links. (And stated plainly: a brand-new service with no merged history will not appear; this complements a dependency graph rather than replacing it.)

Current limitations, stated plainly

  • Validated on a single corpus (dotnet/efcore); the connector seam exists but only a GitHub connector ships today.
  • The full-power forward-coverage number is still open — the harness is built and leak-checked, but scaling the labeling is human-bound because the LLM judge failed calibration.
  • Embeddings default to OpenAI; the endpoint is pluggable via embed.base_url to any OpenAI-compatible provider, including a local one — but fully air-gapped deployment has not been exercised end-to-end.

Links & contact