Eyobed Feleke

<start of post>

I just open-sourced a new evaluation framework for Agentic RAG.

Here is why it matters.

Most developers are still testing their RAG pipelines using simple similarity scores. But in a production environment, the failure mode isn't just a "low score"—it is the agent hallucinating a tool call that deletes a production database.

That is the gap I wanted to bridge.

Over the weekend I released RAG-Stress-Test v0.2 — a suite of 150 edge-case scenarios designed to break agentic workflows:

Does the agent detect when the retrieved context is intentionally contradictory?

Does it refuse to execute a tool call when the user's intent is ambiguous?

Does it maintain state across a 10-turn conversation without losing the original constraint?

Is the final output formatted strictly according to the schema?

A base GPT-4o model without custom prompting failed 42% of the "ambiguity" tasks. After implementing a recursive verification loop, that failure rate dropped to 8%.

The bottleneck isn't the LLM's intelligence. It is the lack of "guardrail" prompts that force the model to pause and reflect before calling a tool. We are still in the "move fast and break things" phase of AI agents, but for B2B, we need to move fast and "verify everything."

GitHub: https://lnkd.in/example1

Documentation: https://lnkd.in/example2

If you are building autonomous agents that have "write" access to your data — how are you stress-testing them?

#AI #RAG #LLMs #SoftwareEngineering #AgenticAI #OpenSource

<end of post>