ConclusiveEXP-055 September 2026iones labUpdated on 16 October 2026

RAG & Hermes Agent: from 11% to 2% hallucinations across 1,200 internal documents

The questionCan we give our teams a reliable, sourced answer on any project, in under three seconds, from our existing documentation?

Verdict

Yes, provided you enforce source citation and refuse to answer when no passage supports the answer. Reranking makes the difference, not the model.

DecisionAdopted. Integrated into Jarvis, our internal assistant, and used daily by the team.

Status: Conclusive
Duration: 4 weeks
Team: 2 developers, 1 architect
Stack: Hermes Agent
pgvector
BM25
Cohere Rerank
Claude
MCP

0.84precision at 5 resultsagainst 0.72 without reranking, 0.61 with vector search only

2%answers with a hallucinationagainst 11% on the first prototype

2.1 smedian latencyfrom question to complete answer

1,200documents indexed38,000 passages

Context

Three years of projects add up to a lot of documentation: specifications, meeting minutes, deployment procedures, architecture decisions, tickets. It all exists, but finding “why we chose this VAT calculation rule on project X in 2023” often takes longer than asking the colleague who remembers. And they are not always there.

A first similarity-search prototype, built in two days, gave convincing answers. Too convincing: reviewing about a hundred answers, we found eleven that invented a detail, with perfect confidence. The lab therefore took the subject on with a clear objective: measure this rate, then reduce it to a level acceptable for daily use.

What we tested

We built a corpus of 1,200 documents from six completed projects, split into 38,000 passages, and a set of 180 questions written by the project managers, each with the expected answer and the source document. The set also included questions whose answer is not in the documentation. Each variant of the pipeline was evaluated on this same set.

V1: vector search only (embeddings, pgvector), 5 passages sent to the model.
V2: hybrid search, vectors + BM25 merged, to catch exact terms (table names, error codes, ticket references).
V3: V2 plus reranking of the top 30 passages by a rerank model, 5 kept.
V4: V3 plus a strict instruction: every statement must cite a passage, and the model must answer “I can’t find it” if no passage supports the answer.
Hermes Agent skills: Hermes Agent is the open-source agent framework from Nous Research, and its skills are reusable capabilities. We wrote one per question type (procedure, decision, figure), which adapts the query and the answer format.

Results

0.61
precision@5 with vector search only (V1): 0.72
precision@5 with hybrid search (V2): 0.84
precision@5 with rerank (V3): 2%
hallucinations with mandatory citations (V4)

Reranking is the most profitable lever. Going from V2 to V3 costs 180 ms of latency and improves precision by 12 points. Changing the embeddings model, on the other hand, moved the measure by only 2 points. We stopped comparing models and concentrated our effort on the search chain.

From 11% to 2% hallucinations, across the whole chain. The 11% was recorded on the first prototype, the 2% on V4, which also adds hybrid search and reranking: the gap comes from the whole, not from the citation instruction alone. With this instruction, the model answers “I can’t find this in the documentation” on 9% of the questions in the set. It is frustrating, but nine times out of ten the answer genuinely did not exist in the corpus. The remaining 2% of hallucinations are shortcuts between two close passages, on projects with similar business rules.

Hybrid search fixes almost all failures on exact identifiers: an error code or a table name cannot be found through semantic similarity.
Hermes Agent skills mainly improve the form: a procedure rendered as numbered steps, a decision with its rejected alternatives. Small precision gain, real usability gain.
The median latency of 2.1 s is driven by generation, not by search (280 ms all included).
Passage splitting: 400 tokens with an 80-token overlap gave the best results. Shorter loses context, longer dilutes the answer.

What we conclude

A reliable RAG is first of all a reliable search engine. The language model only rephrases what it is given; if the search brings back the right passage, the answer is good. We spent 80% of the lab’s time on indexing, merging and reranking, and that is where the gains were.

The citation constraint changes the relationship of trust. A sourced answer can be checked in one click; an unsourced answer has to be taken on trust. We will never expose a client to an assistant that does not cite its sources.

“The day the tool answered “I can’t find it” to a question I was sure I knew the answer to, I checked: the documentation didn’t exist. We wrote it.”
– Project manager, BUILD team

What’s next

The V4 pipeline is integrated into Jarvis, our internal assistant, and now indexes the documentation of all our active projects, with reindexing every night.
The “I can’t find it” questions are reported to the project managers every week: this has become our best detector of missing documentation.
Next step: evaluate the same pipeline on a client’s documentation, with strict separation per project.

Jarvis then went from an assistant that answers to an agent that triages our support tickets: see the report EXP-06, Jarvis & OpenClaw.

Let’s keep in touch

Social