Mimiron AI

What we got wrong

The most useful results in this project are the places it was wrong and how that was found. Each one below was measured before and after.

01

The evaluation was testing the wrong search

What we believed. Runs 2 to 5 measured the search the site uses.

What the measurement showed. Every chunk stores two kinds of vector. The evaluation searched with the older MiniLM vectors; the site searches with the arctic ones. Nothing failed, because both were there.

02

Recall went down, and that was the good news

What we believed. The graded pool held every passage worth grading.

What the measurement showed. Passages that only the arctic search finds had never been graded, so they counted as misses. 745 such passages were then graded; the first 568 alone held 52 that answer a question and had never been marked.

03

A production bug nobody could see

What we believed. Each search question was embedded the way the model expects.

What the measurement showed. The search instruction was being added in front of every question twice: once by our code, once by Weaviate. Nothing errored. Embedding 48 questions locally three ways showed which matched production: no instruction 0 of 48, once 5 of 48, twice 46 of 48.

04

The model refused questions it had the answer to

What we believed. A refusal meant the documents did not cover the question.

What the measurement showed. 6 of 42 answerable questions got “The provided context does not answer this question.” In 5 of them, the answering passage was in the prompt.

05

“Hybrid is best” stopped being true

What we believed. Hybrid search beats keyword and vector, which held on every chunker with MiniLM.

What the measurement showed. On the arctic index, vector and hybrid are level, and keyword search alone is the weakest on every chunker.

06

The evaluation took the live search down

What we believed. Running an evaluation could not affect the site.

What the measurement showed. On 2026-10-12 the evaluation scripts used up the shared daily embedding limit, and the site's arctic search answered 429 until it reset.

07

Over half the chunks started with a full stop

What we believed. Twelve passing unit tests meant the chunker was right.

What the measurement showed. 333 of 576 chunks began with a bare full stop. The tests checked structure, the evaluation still found the content, and a code review read a rule that sounded right. A person reading one passage caught it.

08

The first winner was a product of the test set

What we believed. BM25 led the first comparison.

What the measurement showed. Ten held-out questions, committed before they were run, put BM25 last on every chunker. A stopword tweak that fixed one failing question was also tested and reverted, because it was chosen after seeing which question failed.

09

The judge is wrong sometimes

What we believed. An LLM judge's grades can be used as they come.

What the measurement showed. It credited a passage with a kubectl command the passage does not contain, gave 0 to a correct answer whose reference was one word, and scored a perfect one-fact answer as half noise.