Mimiron AI

Retrieval evaluation

Run 10: 112 questions (106 answerable) × 18 configurations (3 chunkers × 3 retrieval modes × reranker on/off), embedding model snowflake-arctic-embed-m-v1.5, k = 5. Relevance labels are graded 0/1/2 per (question, chunk) by an LLM judge (Claude Haiku 4.5) over a TREC-style pool; a chunk counts as relevant at grade 2.

Winner · run 10

Recursive chunking + Hybrid retrieval + Cross-encoder reranker

0.70Recall@5 (joint highest)
0.89MRR
101/106Hit rate@5

Why this combination wins

18 configurations: 3 chunkers (fixed, recursive, semantic) × 3 retrieval modes (BM25, vector, hybrid) × reranker on/off. Each choice below is decided by the numbers in the table further down.

  1. Hybrid over BM25. BM25 alone reaches recall@5 0.61 at best; hybrid reaches 0.70. Users describe symptoms in their own words, so dense retrieval catches the paraphrase, while hybrid keeps BM25's exact matching for error strings and field names.
  2. Cross-encoder reranker on. It raises MRR in 9 of 9 configurations (production: 0.84 → 0.89). Recall@5 does not change, because it only reorders the candidates; it moves the answering chunk to the top, which is what the LLM weighs most.
  3. Recursive over semantic and fixed. The top configurations are statistically tied: paired bootstrap of the nearest challenger (semantic / vector / rerank) against the winner, recall@5 −0.015 (95% CI −0.07 to +0.04). Semantic led on the first 58 questions and lost on the next 54, which is why a small test set is not trusted. Within the tie, recursive has the joint-highest recall@5 and the cheapest indexing; semantic chunking embeds every sentence.
  4. Recall over precision. Fixed chunks score the best F1@5 because they are smaller and more precise, but a chunk that is never retrieved cannot be used, while an extra one costs a few tokens and the LLM ignores it. Retrieval is optimised for recall@5; the reranker fixes the order.

How chunk quality is graded

  1. Questions. 112 Kubernetes questions written the way people ask (error messages, symptoms, keywords), typed lexical, semantic or mixed; 6 are deliberately unanswerable from the corpus. Each was checked against the docs and kept or dropped by a person.
  2. Pooling. Grading the whole corpus per question is not feasible, so the top 10 chunks from every configuration are pooled (TREC-style) and only that union is graded: about 5,000 (question, chunk) pairs. Pooling 10 deep gives a relevant chunk at rank 7 a label, so recall@5 has an honest denominator.
  3. LLM judge. Claude Haiku 4.5 at temperature 0 with a fixed rubric grades each pair 0 (off topic), 1 (related) or 2 (answers it). Only grade 2 counts as relevant. A reply that does not parse stays unlabelled rather than guessed.
  4. Human audit of the judge. 20 grades drawn at random and re-graded blind by a person: 13 of 20 exact, 20 of 20 within one grade, never 0 against 2. Where they differed the judge was the generous one, so recall leans slightly optimistic.
  5. Metrics. Hit rate@5 (anything useful retrieved), recall@5 (how much of the evidence), precision@5 and F1@5 (how clean the context is), MRR (rank of the first relevant chunk), nDCG@5 (graded, position-weighted ranking).
  6. End-to-end check. On the generated answers: faithfulness 0.94, context relevance 0.97. Without retrieval, factual recall drops from 0.83 to 0.54, so the chunks carry information the model does not have.

All configurations

Winner pinned first; the rest sorted by recall@5, then MRR. F1@5 is the harmonic mean of mean precision@5 and mean recall@5. Fixed chunks score higher precision and F1@5 because they are smaller; the winner is picked on recall@5, since the LLM needs the answer present in the five and can ignore an irrelevant chunk.

ChunkerRetrievalRerankerPrecision@5Recall@5F1@5MRRnDCG@5Hit rate@5
RecursiveHybridon0.440.700.540.890.85101 / 106winner
RecursiveHybridoff0.440.700.540.840.84101 / 106
SemanticHybridon0.450.690.540.910.86100 / 106
SemanticVectoron0.430.690.530.910.84102 / 106
SemanticHybridoff0.450.690.540.840.84100 / 106
SemanticVectoroff0.430.690.530.840.83102 / 106
FixedHybridon0.400.680.500.900.76102 / 106
RecursiveVectoron0.430.670.520.880.84101 / 106
FixedHybridoff0.390.670.490.850.74102 / 106
RecursiveVectoroff0.430.670.520.830.82101 / 106
FixedVectoron0.370.640.470.870.74100 / 106
FixedVectoroff0.370.640.470.840.73100 / 106
FixedBM25on0.350.610.440.870.6998 / 106
FixedBM25off0.350.610.440.820.6898 / 106
RecursiveBM25on0.380.580.460.820.7692 / 106
RecursiveBM25off0.380.580.460.780.7492 / 106
SemanticBM25on0.370.560.450.810.7690 / 106
SemanticBM25off0.370.560.450.730.7390 / 106

Per-question drill-down

Pick a question and a configuration to see the retrieved chunks and the judge's grade for each.

Liveness probe failed: HTTP probe failed with statuscode: 500

Uses the manual’s own words
Chunking strategy
Retrieval mode
Reranker
  1. Query
  2. Retrieve
  3. Rerank
  4. Judge
  5. Generate

Retrieved chunks

BM25 and vector run together; scores are normalised and fused at alpha 0.5, so a chunk strong on either signal can reach the top 5. A cross-encoder (gte-reranker-modernbert-base) then scores each (question, chunk) pair and reorders the top 5.

  1. 1Define a liveness HTTP requestGrade 2 · relevant

    For the first 10 seconds that the container is alive, the `/healthz` handler returns a status of 200. After that, the handler returns a status of 500. ```go http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { duration := time.Now().Sub(started) if duration.Seconds() > 10 { w.WriteHeader(500) w.Write([]byte(fmt.Sprintf("error: %v", duration.Seconds()))) } else { w.WriteHeader(200) w.Write([]byte("ok")) } }) ``` The kubelet starts performing health checks 3 seconds after the container starts. So the first couple of health checks will succeed. But after 10 seconds, the health checks will fail, and the kubelet will kill and restart the container. To try the HTTP liveness check, create a Pod: ```shell kubectl apply -f https://k8s.io/examples/pods/probe/http-liveness.yaml ``` After 10 seconds, view Pod events to verify that liveness probes have failed and the container has been restarted:

    Judge: The passage directly explains what causes an HTTP liveness probe to fail with statuscode 500 and demonstrates how the kubelet responds to such failures by killing and restarting the container.

  2. 2Types of probeGrade 1 · partial

    Incorrect implementation of liveness probes can lead to cascading failures. This results in restarting of container under high load; failed client requests as your application became less scalable; and increased workload on remaining pods due to some failed pods. Understand the difference between liveness and readiness probes and when to apply them for your app. ### Readiness probe Readiness probes determine when a container is ready to accept traffic. This is useful when waiting for an application to perform time-consuming initial tasks, such as establishing network connections, loading files, and warming caches. Readiness probes can also be useful later in the container’s lifecycle, for example, when recovering from temporary faults or overloads. If the readiness probe returns a failed state, the EndpointSlice controller removes the Pod's IP address from the EndpointSlices of all Services that match the Pod.

    Judge: The passage discusses liveness and readiness probes conceptually and their consequences, but does not explain what causes an HTTP 500 status code or how to troubleshoot this specific liveness probe failure.

  3. 3Define a gRPC liveness probeGrade 1 · partial

    ```shell kubectl apply -f https://k8s.io/examples/pods/probe/grpc-liveness.yaml ``` After 15 seconds, view Pod events to verify that the liveness check has not failed: ```shell kubectl describe pod etcd-with-grpc ``` When using a gRPC probe, there are some technical details to be aware of: - The probes run against the pod IP address or its hostname. Be sure to configure your gRPC endpoint to listen on the Pod's IP address. - The probes connect over plaintext by default. See [Use TLS with gRPC probes](#grpc-probe-tls) for how to enable TLS. - There are no error codes for built-in probes. All errors are considered as probe failures. - If `ExecProbeTimeout` feature gate is set to `false`, grpc-health-probe does **not** respect the `timeoutSeconds` setting (which defaults to 1s), while built-in probe would fail on timeout. ### Use TLS with gRPC probes FEATURE STATE: feature gate GRPCContainerProbeTLS

    Judge: The passage discusses gRPC probes and mentions that 'all errors are considered as probe failures,' providing context about probe failures, but does not specifically address HTTP 500 status code failures or how to diagnose and fix them.

  4. 4Types of probeGrade 1 · partial

    Liveness probes determine when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs. If a container fails its liveness probe more times than the configured tolerance, the kubelet restarts that container. Liveness probes do not wait for readiness probes to succeed. If you want to wait before executing a liveness probe, you can either define `initialDelaySeconds` or use a [startup probe](#startup-probe). Liveness probes can be a powerful way to recover from application failures, but they should be used with caution. Liveness probes must be configured carefully to ensure that they truly indicate unrecoverable application failure, for example a deadlock.

    Judge: The passage explains what liveness probes are and their purpose, but does not explain what an HTTP 500 status code means or why the probe failed.

  5. 5Define a liveness HTTP requestGrade 1 · partial

    Define a liveness HTTP request Another kind of liveness probe uses an HTTP GET request. Here is the configuration file for a Pod that runs a container based on the `registry.k8s.io/e2e-test-images/agnhost` image. ```yaml apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-http spec: containers: - name: liveness image: registry.k8s.io/e2e-test-images/agnhost:2.40 args: - liveness livenessProbe: httpGet: path: /healthz port: 8080 httpHeaders: - name: Custom-Header value: Awesome initialDelaySeconds: 3 periodSeconds: 3 ```

    Judge: The passage explains how to define an HTTP liveness probe but does not explain what a 500 status code means or how to troubleshoot it.

The piece that answers the question came back first.Open any piece to read it and the judge’s reason.

Rank of the first relevant chunk, all configurations

Position of the first grade-2 chunk in the top 5, across all 18 configurations (× = not retrieved). Click a cell to load it.

BM25VectorHybrid
no rerankrerankno rerankrerankno rerankrerank
Fixed
Recursive
Semantic

The answer, with and without looking it up

Both written by claude-haiku-4.5 for this question. On the left it answers from memory. On the right it is given the 5 pieces first and told to answer only from them.

From memory

This means your container is running, but the liveness probe detected it's unhealthy (returning HTTP 500). Kubernetes will restart the pod.

Common causes:

  • Application crashed or is in a bad state
  • Dependency unavailable (database, cache, etc.)
  • Resource exhaustion (memory, CPU)
  • Configuration error

Troubleshooting:

  • Check pod logs: kubectl logs <pod-name>
  • Check previous logs if pod restarted: kubectl logs <pod-name> --previous
  • Describe the pod for events: kubectl describe pod <pod-name>
  • Test the endpoint manually if possible
  • Check if the liveness probe is too aggressive (low initialDelaySeconds or periodSeconds)

Quick fixes:

  • Increase initialDelaySeconds to give the app time to start
  • Check application health endpoint is actually returning 200
  • Verify dependencies are accessible from the pod

No sources. Recorded 2026-10-14.

After looking it up

Based on the passages, an HTTP liveness probe failure with status code 500 indicates that the kubelet received a failure response from the container's health check endpoint.

According to the HTTP probe specification, any status code greater than or equal to 200 and less than 400 indicates success. Any other code indicates failure. A 500 status code falls outside the success range, so the kubelet will consider the container unhealthy.

When a liveness probe fails, the kubelet's response is to kill the container and restart it. This is the expected behavior of a liveness probe—it determines whether a container should be restarted based on whether it is still running and healthy.

The example in the passages demonstrates this: a test server returns a 200 status for the first 10 seconds after the container starts, then returns a 500 status after that. The kubelet, which begins health checks 3 seconds after container startup, will eventually receive the 500 response and restart the container.

If you're seeing this failure unexpectedly, you should verify that your application's health check endpoint is functioning correctly and returning appropriate status codes during normal operation.

Sources

  • probes.md — Define a liveness HTTP request

Sources: probes.md — Define a liveness HTTP request; pod-probes-concept.md — Types of probe; probes.md — Define a gRPC liveness probe. Recorded 2026-10-13.