Evals for AI systems, as an MCP.

Try it live

Run a real eval right here, no signup. Pick a sample (or paste your own input, context, and output), choose a free judge model, and watch retriEVAL score it. Nothing is saved.

What makes it different

The metrics aren't new; DeepEval and Ragas cover those. The shape is: a standalone, local-first MCP an agent can call mid-workflow, with RAG scored by stage.

Or just run evals by chatting

No pipeline required. Connect retriEVAL in Claude or another chat UI, toggle it on, and a manual tester can score outputs and author metrics in plain language. Every result links straight to the run in the dashboard.

Quickstart

Self-hosted. Clone it, point it at a judge, run it. Your data never leaves your machine.

copygit clone https://github.com/hcarrillo001/retrieval-mcp
cd retrieval-mcp
pip install -r requirements.txt

# pick a judge, or use a local one below
export ANTHROPIC_API_KEY=sk-ant-...
python server.py

copy# in the chat: paste your cases inline
"Score these cases with faithfulness:
 [{"input": "...",
   "actual_output": "...",
   "retrieval_context": ["..."]}]"

# nothing is stored, no setup step

Pick your judge: free local Ollama (RETRIEVAL_JUDGE_BACKEND=ollama) to keep everything on your machine, or your Anthropic key, your OpenAI key, or any OpenAI-compatible endpoint via OPENAI_BASE_URL. Full setup, including your own always-on instance with a dashboard, is in the repo README.

See it in action

A full walkthrough, plus the three steps to run it yourself in Cursor.

Connect retriEVAL and run an eval

Add the connector, load a golden set, score it, and open the run in the dashboard.

Cursor setup

Run retriEVAL locally as an MCP server in Cursor.

What you're aiming for: connected in Cursor, green dot, 14 tools enabled.

Copy, paste, run. About two minutes, and you need Docker Desktop plus a judge key (an Anthropic key here, or skip it and run a local model, see below).

1

Start Docker Desktop and wait for the whale icon to stop animating. Nothing to download: the first run pulls the image for you. Docker has to be running before Cursor starts, or the server won't launch and Cursor won't tell you why.

2

Run this in your terminal. It creates the config Cursor reads:

copymkdir -p ~/.cursor && cat > ~/.cursor/mcp.json <<'EOF'
{
  "mcpServers": {
    "retrieval-mcp": {
      "command": "docker",
      "args": ["run", "-i", "--rm", "-e", "ANTHROPIC_API_KEY",
               "ghcr.io/hcarrillo001/retrieval-mcp:1.0.0"],
      "env": { "ANTHROPIC_API_KEY": "sk-ant-PUT-YOUR-KEY-HERE" }
    }
  }
}
EOF

Already have a ~/.cursor/mcp.json? This overwrites it, so paste the retrieval-mcp block into your existing file instead.

3

Put your key in and check it parsed:

copyopen -a Cursor ~/.cursor/mcp.json

# replace sk-ant-PUT-YOUR-KEY-HERE, save, then:
python3 -c "import json;json.load(open('$HOME/.cursor/mcp.json'));print('valid json')"

4

Restart Cursor, open Settings → Customize, and look for retrieval-mcp with a green dot and 14 tools enabled. Then paste this into the chat:

copy# in the Cursor chat: expect 1.00, all claims supported
Score this with faithfulness:
 input "What colour is the Golden Gate Bridge?",
 output "It is painted International Orange.",
 retrieval context ["The Golden Gate Bridge is
  painted International Orange."]

No green dot? Docker wasn't running when Cursor started. Start it and restart Cursor. Want to keep everything local? Add "RETRIEVAL_JUDGE_BACKEND": "ollama" to the env block, drop the API key, and no data leaves your machine. Runs are written inside the container, never to a hosted database.

Prefer to try it yourself? Run a live eval in the sandbox. No signup.

How it compares

Honest positioning: not "better metrics," but a different shape: standalone, local, and usable as a tool inside an agent.