Truth
Symptom production data a few weeks out of date.
4.6M
rows, one writer.
Root cause A nightly dev→prod sync upserted on surrogate keys. The same logical row had a different ID in each database, so the sync overwrote production.
Fix A single-writer topology: every scraper and pipeline writes the primary, reads go to replicas. I reconciled 125 tables with a foreign-key-resolved content-hash diff, gated the reload on dev being a content superset of prod, and cut over by flipping one secret, reversible in minutes. 0 FK orphans. Running the copy in-region took it from 3,643s to 691s.
Rejected Repairing the sync. Surrogate IDs are not cross-database keys, so no upsert keyed on them can be correct.
Cost
Symptom an extra $12.6k on the LLM bill.
$12.6k
of overspend, traced to one loop.
Root cause KEDA scaled on Redis list length, and a task leaves the list once a worker takes it. The queue read as empty, pods scaled to zero mid-task, and late acks redelivered every killed task: 62k retries on 94 documents.
Fix Designed the workers for at-least-once delivery: claim each document before the LLM call, commit before ack, leases with heartbeats, fencing tokens so a stale worker’s write is rejected, and attempt caps. The new bulk pipeline runs on this design.
Rejected Pinning one replica. It hides the blind scaler instead of fixing it, and scaling down still kills tasks in flight.
Scale
~$0.06
per scanned document, 11 sources.
Every page is classified, extracted and cross-checked on key fields, and mismatches go to human review. A cheap classifier rejects wrong documents before extraction. Worker capacity went from 16 to 64 concurrent documents, and a scanned-guidance backfill transcribed 657 pages for $0.13 with zero errors.
Quality
Symptom search misses on plain-English queries.
88%
search accuracy, up from 62%.
Root cause The same errors reproduced on three different models, so the gap was prompt and vocabulary, not model quality. 41 of 176 benchmark queries needed filters the parser couldn’t express.
Fix Eval-gated prompt, vocabulary and deterministic pre-pass work across three callers, scored exact-match on a 176-query benchmark. A set frozen before tuning scores 92%. Any regression blocks the release. 0.9s median.
Rejected Upgrading the model, and big closed vocabularies as schema enums, which cost latency and determinism.
Architecture
10 → 1
search bars, one policy gateway.
A client saw 6 results on a company’s public page and 20+ in the app: three matching engines used different trigram thresholds. I replaced ten search bars and three engines with one server-driven registry across 18 sources behind a single policy gateway. Parity is pinned by tests that fail if the frontend and backend contracts drift. About 7k lines of duplicate code removed.
Rejected A count teaser for locked sources. It would leak data through counts.
Clients’ AI
MCP
our platform, inside clients’ AI.
An MCP server with 14 tools and 971 automated tests: OAuth 2.1 with PKCE and dynamic client registration, consent-gated write scopes, per-user admission control and kill switches. At a live demo a Fortune Global 500 pharma rated it deeper than its in-house agent. Submitted to the Claude connector directory.
Result
2×
partner renewal.
A partner wanted our data queryable inside its own warehouse. I designed a 13-table BigQuery star schema of 422k events across 7 jurisdictions, loaded through federated queries on a read replica and shared natively into theirs. It supported a 2× renewal.
