A citation that points at the wrong page is worse than no citation

I spent three years building AI features for other people's products. The thing that cost me the most sleep was never the model. It was a citation that used to be right.

Here is how it goes. You ship retrieval. Answers come back with a filename and a page number, and the first time a customer clicks one and lands exactly on the paragraph the answer paraphrased, you have won something you cannot buy back later: they stop checking. That is the whole point of a citation. Not that it exists — that after a while, nobody feels the need to open it.

Then, four months in, you change your embedding model. Something cheaper, or something better, or the old one gets deprecated with sixty days' notice. You reindex. And unless you were careful in a way most of us were not the first time, some fraction of your stored citations now point at the wrong span of the wrong page.

Nothing errors. The answers still read fine. The number next to the quote is still a number.

A customer clicks one, lands on a page about something else, and now they check every single one. You have not lost a feature. You have lost the trust that made the feature worth having, and you get it back much more slowly than you lost it.

Why this happens even when you are being careful

The failure is boring, which is why it is common. A citation is not one fact. It is at least four, and they are stored in different places at different times:

• the document, which has an id
• the chunk, which has an index inside that document
• the page or offset the chunk came from
• the vectors, which belong to whichever embedding model produced them

Re-embedding touches the last one. Re-parsing touches the middle two. And a "reindex" in most homegrown pipelines quietly does both, because the parse and the embed are one function that runs when a file arrives. Change the model, re-run the function, and the chunk boundaries move — same document, same page, different spans, and the offsets you stored last quarter now describe text that has shifted a few hundred characters to the left.

The version of this that gets you is subtler still. You do not reindex everything at once, because you have 400,000 documents and it costs real money. So you reindex in batches, over a week. Now half your corpus is embedded by model A and half by model B, they live in the same vector collection, their distances are not comparable, and retrieval quality drops in a way that looks like the new model being worse.

What we ended up doing

Two things, and neither is clever. They are just the things you eventually decide to make the system responsible for instead of yourself.

A document records which model's vectors it is in. Every document in Oberik carries embedding_model and pipeline_version — the reader and the embedder that produced its current vectors, as one string. A document embedded by an older model lives in its own collection rather than being mixed in with the new one. Which means "what does a reindex still owe me" is a query, not an audit:

const stale = (await ai.documents.list({ limit: 500 }))
  .items.filter((d) => d.embedding_model !== current);

That is the whole trick. Not preventing the drift — you cannot, you are going to change models — but making the half-migrated state legible while you are in it, rather than discoverable through a support ticket.

A document tells you how it was read. read_with names what actually happened to each file: text-layer for a PDF whose own text was extracted locally, local-ocr for OCR on our machines, document-parser for a native format, and otherwise the name of the model that read it. pdf_type and ocr_page_count sit next to it.

That field started as a billing answer. Someone asks why ingesting 2,000 PDFs cost what it did, and "we OCR'd 40 pages out of 90,000, here they are" is a better answer than a shrug. But it turned out to matter more for the other question, the one that arrives as "why is this answer empty." A 200-page report with two scanned exhibits should cost two pages of OCR. If ocr_page_count says 200, something upstream decided your text layer was untrustworthy, and that is a very different bug from a retrieval bug — but from the outside, both look like a bad answer.

The rule underneath

A file that nothing could read comes back failed, with a reason. Never ready with chunk_count: 0.

That sounds like a small API decision. It is the entire philosophy. A ready document with zero chunks is a document that will never appear in an answer and will never appear in an error either — it just quietly is not there, in a corpus of 400,000 things, and you find out when a customer asks about the one contract that mattered and gets a confident answer built from four other contracts.

The same instinct runs through the parts of this that are less obvious. Retrieval returns the passages that matched — a ranked fragment, never the whole document. So anything that depends on all of a file (a total, a count, a max, any per-row arithmetic) has to fetch the file whole and compute over it, because arithmetic over a fragment produces an exact-looking number that is simply short. Row count, filename, three decimal places. Reads as verified. Is wrong.

That one is not solvable by being careful either. It is solvable by the agent having a sandbox to pull the file into, and by being told, repeatedly and in the retrieval results themselves, that a fragment is not a file.

What I would tell myself in 2023

Store the pipeline version on the row from day one. It costs you one column and it is the difference between a migration you can watch and a migration you can only survive.

And decide early what a wrong citation costs you, because it is not the same for every product. If you are summarising internal Slack, a citation off by a page is an annoyance. If you are pointing a lawyer at a clause in a contract they are about to rely on, it is the product failing at the one thing it claimed to do — and the failure is silent, and it lands on the customer, and they are the one who finds it.

That asymmetry is why we made groundedness a check you can enforce server-side rather than a prompt instruction the model may or may not follow. Turn it on and an answer the retrieved sources do not support says so instead of asserting it. Start it on flag rather than block, watch it against your real corpus for a week, and only then decide what a violation should cost — because it is a model judgement, and every wrong judgement on block is a refused answer to a question your corpus could have answered.

Oberik is free while we launch, and you bring your own provider keys — so if you want to point it at a corpus you already have and see what read_with says about it, that costs you the price of your own embeddings and nothing else. oberik.com