LLM-as-Judge¶

Evaluate answer quality using Prometheus 2, a specialized evaluation model.

Overview¶

LLM-as-Judge provides multi-criteria evaluation:

Setup¶

Install Prometheus 2 via Ollama:

ollama pull hf.co/RichardErkhov/prometheus-eval_-_prometheus-7b-v2.0-gguf:Q5_K_M

Requirements:

• ~5GB disk space
• 16GB RAM recommended
• Ollama running (ollama serve)

Basic Usage¶

Python API¶

from ragnarok_ai.evaluators.judge import LLMJudge

judge = LLMJudge()

# Evaluate faithfulness
result = await judge.evaluate_faithfulness(
    context="Paris is the capital of France. It has a population of 2.1 million.",
    question="What is the capital of France?",
    answer="Paris is the capital of France.",
)

print(f"Verdict: {result.verdict}")      # PASS
print(f"Score: {result.score:.2f}")       # 0.85
print(f"Explanation: {result.explanation}")

CLI¶

ragnarok judge \
  --context "Paris is the capital of France." \
  --question "What is the capital of France?" \
  --answer "Paris is the capital of France."

Evaluation Criteria¶

Faithfulness¶

Checks if the answer is grounded in the provided context.

result = await judge.evaluate_faithfulness(
    context="Python was created by Guido van Rossum in 1991.",
    question="Who created Python?",
    answer="Python was created by Guido van Rossum.",
)
# PASS - answer is supported by context

Relevance¶

Checks if the answer addresses the question.

result = await judge.evaluate_relevance(
    question="What is the capital of France?",
    answer="Paris is the capital of France.",
)
# PASS - directly answers the question

Hallucination Detection¶

Checks for fabricated information not in the context.

result = await judge.detect_hallucination(
    context="Python was created by Guido van Rossum.",
    answer="Python was created by Guido van Rossum in the Netherlands in 1991.",
)
# PARTIAL - "1991" and "Netherlands" not in context

Completeness¶

Checks if the answer covers all aspects of the question.

result = await judge.evaluate_completeness(
    question="What is Python and who created it?",
    answer="Python is a programming language.",
    context="Python is a programming language created by Guido van Rossum.",
)
# PARTIAL - missing creator information

Batch Evaluation¶

Evaluate multiple items from a file:

# items.json
# [
#   {"context": "...", "question": "...", "answer": "..."},
#   {"context": "...", "question": "...", "answer": "..."}
# ]

ragnarok judge --file items.json --criteria faithfulness,relevance

Select Criteria¶

Evaluate specific criteria only:

# Single criterion
ragnarok judge --file items.json --criteria faithfulness

# Multiple criteria
ragnarok judge --file items.json --criteria faithfulness,relevance

# All criteria (default)
ragnarok judge --file items.json --criteria all

Medical Mode¶

Reduce false positives in healthcare RAG evaluation:

judge = LLMJudge(medical_mode=True)

result = await judge.evaluate_faithfulness(
    context="Patient diagnosed with CHF.",
    question="What condition does the patient have?",
    answer="Patient has congestive heart failure.",
)
# PASS - "CHF" and "congestive heart failure" are equivalent

Features:

• 350+ medical abbreviations (CHF, MI, COPD, DVT...)
• Context-aware disambiguation
• Multiple formats: dotted (q.d.), slash (s/p), mixed-case (SpO2)

SLM disambiguation¶

Some abbreviations are ambiguous (MS → multiple sclerosis, mitral stenosis, morphine sulfate...). By default they are resolved with context keywords, falling back to the most common meaning when the context gives no signal. For sparse or unconventional contexts, you can hand those uncertain cases to a small local model:

from ragnarok_ai.adapters import OllamaLLM

judge = LLMJudge(
    medical_mode=True,
    disambiguation_llm=OllamaLLM(model="qwen2.5:0.5b"),
)

How it works:

• Only ambiguous abbreviations with zero context-keyword hits are escalated — everything else stays free
• The SLM picks among the dictionary's own candidate meanings (closed-set classification), so it can never invent an expansion
• Decisions are cached per (abbreviation, context), so batch evaluations pay each resolution once
• If the SLM is unreachable or cannot tell, the priority fallback applies — behavior degrades to the default, never worse

The same parameter exists on FaithfulnessEvaluator.

Scoring¶

Prometheus 2 uses a 1-5 rubric, normalized to 0-1:

Verdict thresholds:

• PASS: score >= 0.7
• PARTIAL: 0.4 <= score < 0.7
• FAIL: score < 0.4

Performance¶

On Apple M2 16GB:

Configuration¶

Custom Model¶

judge = LLMJudge(
    model="llama3",  # Use different model
    base_url="http://localhost:11434",
)

CLI Options¶

ragnarok judge \
  --file items.json \
  --model llama3 \
  --ollama-url http://localhost:11434 \
  --fail-under 0.7 \
  --output results.json

Output Formats¶

Console Output¶

  RAGnarok-AI LLM-as-Judge
  ========================================

  Items to evaluate: 1
  Criteria: faithfulness, relevance

  [1/1] Evaluating: What is the capital of France?

  ----------------------------------------
  Results
  ----------------------------------------

  Item 1:
    Question: What is the capital of France?
    [+] faithfulness: 0.85 (PASS)
    [+] relevance: 0.90 (PASS)
    Average: 0.88

  ----------------------------------------
  Overall Average: 0.8750

JSON Output¶

ragnarok judge --file items.json --json

{
  "command": "judge",
  "status": "pass",
  "version": "1.4.0",
  "data": {
    "items_evaluated": 1,
    "criteria": ["faithfulness", "relevance"],
    "results": [
      {
        "question": "What is the capital of France?",
        "criteria": {
          "faithfulness": {"verdict": "PASS", "score": 0.85, "explanation": "..."},
          "relevance": {"verdict": "PASS", "score": 0.90, "explanation": "..."}
        },
        "average_score": 0.875
      }
    ],
    "overall_average": 0.875
  }
}

CI/CD Integration¶

Use --fail-under for quality gates:

ragnarok judge --file items.json --fail-under 0.7

# Exit code 0 if average >= 0.7
# Exit code 1 if average < 0.7

Next Steps¶

• Evaluation Guide — Retrieval metrics
• GitHub Action — CI/CD integration
• CLI Reference — Full command reference