FREN

Let’s keep in touch

Social

iones lab · experiment register

iones lab: we test before offering it to you.

AI agents, RAG, runtimes, developer tooling, assisted SEO: everything the market promises goes through our teams first. This register documents each test: the starting question, the protocol, the measurements, and what we concluded. Failures are published just like successes.

Experiments
06
Conclusive
04
Mixed
02
In progress
00
Discarded
00
  1. EXP-05Conclusive
    AI

    RAG & Hermes Agent: from 11% to 2% hallucinations across 1,200 internal documents

    Can we give our teams a reliable, sourced answer on any project, in under three seconds, from our existing documentation?

    VerdictYes, provided you enforce source citation and refuse to answer when no passage supports the answer. Reranking makes the difference, not the model.

    • Hermes Agent
    • pgvector
    • BM25
    • Cohere Rerank
    • Claude
    • MCP
    0.84precision at 5 resultsagainst 0.72 without reranking, 0.61 with vector search onlyRead the report
  2. EXP-04Conclusive
    Performance

    Groq versus GPU runtimes: when speed really changes the product

    Does Groq’s inference speed justify integrating it into our products, and for which uses?

    VerdictYes for anything interactive and short: classification, autocompletion, input assistants. No for long generation and reasoning, where the quality of the available models matters more than speed.

    • Groq
    • Llama 3.3 70B
    • vLLM
    • Claude
    • k6
    • Grafana
    0.21 stime to first tokenagainst 0.9 s on a GPU runtimeRead the report
  3. EXP-02Conclusive
    Performance

    Node, Go, Rust, PHP: the same API, four runtimes, the numbers

    At what load level does the choice of runtime become the limiting factor of a conventional business API?

    VerdictOn an API with a database, the gap between runtimes is 1 to 4 in raw throughput, but the database saturates well before that. The runtime only becomes the bottleneck beyond 3,000 requests per second per instance, a threshold most of our clients do not come close to.

    • Node 22
    • Go 1.23
    • Rust (Axum)
    • PHP 8.3
    • FrankenPHP
    • Postgres
    • k6
    38,400req/s · Rust, no databaseGo 31,200 · Node 12,900 · PHP 9,800Read the report
  4. EXP-01Conclusive
    DX

    Skills & MCP: equipping eight developers and measuring what changes

    Does shared AI tooling (skills and MCP servers) genuinely improve a team’s productivity and quality, or only that of a few developers who are already convinced?

    VerdictThe gain is real and measurable on standard tasks, provided you strictly frame what the agents may do. Without a shared catalogue, everyone reinvents their own rules and quality diverges.

    • Cursor
    • Claude Code
    • MCP
    • Redmine
    • Postgres
    • Playwright
    • Git
    −32%time on standard tasksCRUD, tests, migrations, across 140 ticketsRead the report

Method

How an idea becomes a lab report

  1. 01

    A question

    Every experiment starts from a precise question, raised by a real project or by a market promise we want to check.

  2. 02

    A protocol

    Scope, dataset, duration and success criteria are fixed before we start. What we measure is decided in advance, not afterwards.

  3. 03

    Measurements

    Figures recorded under lab conditions, on cases drawn from our own work. We publish disappointing results just like the others.

  4. 04

    A verdict

    Adopted, retained subject to conditions or discarded. And what we actually do with it in our teams and for our clients.

A topic you would like to see go through the lab?

A technology to assess, an AI use case to validate, a promise to check before investing: we can test it with you, following the same protocol.

Suggest an experiment