Evals for AI systems,
as an MCP.

What makes it different

The metrics aren't new; DeepEval and Ragas cover those. The shape is: a standalone, local-first MCP an agent can call mid-workflow, with RAG scored by stage.

Or just run evals by chatting

No pipeline required. Connect retriEVAL in Claude or another chat UI, toggle it on, and a manual tester can score outputs, author metrics, and pull up charts in plain language.

Try it live

Run a real eval right here, no signup. Pick a sample (or paste your own input, context, and output), choose a free judge model, and watch retriEVAL score it. Nothing is saved.

Quickstart

Run it locally over stdio, or connect the hosted server from anywhere.

copy# install
pip install retrieval-mcp
export ANTHROPIC_API_KEY=sk-ant-...

# add to claude_desktop_config.json
{
  "mcpServers": {
    "retrieval": {
      "command": "retrieval-mcp"
    }
  }
}

copy# Claude.ai → Settings → Connectors
Add custom connector:
  URL: https://YOUR-APP.up.railway.app/mcp
  Auth: Bearer <token>

# then just ask Claude
"Load my golden set and run
 faithfulness + answer_relevancy."

Pick your judge: free local Ollama (RETRIEVAL_JUDGE_BACKEND=ollama), your Anthropic key, your OpenAI key, or any OpenAI-compatible endpoint via OPENAI_BASE_URL. Full deploy guide (Supabase + Railway + Vercel) is in the repo's DEPLOY.md.

See it in action

Walkthroughs and demos: connecting retriEVAL, authoring a metric, and building QE pipelines in Cursor.

YouTube demos will be embedded here.

How it compares

Honest positioning: not "better metrics," but a different shape: standalone, local, and usable as a tool inside an agent.