RAGnarok-AI¶

Local-first RAG evaluation framework for LLM applications

Evaluate, benchmark, and monitor your RAG pipelines — 100% locally, no API keys required.

Why RAGnarok-AI?¶

Building RAG systems is easy. Knowing if they actually work is hard.

RAGnarok-AI solves this with:

• 100% Local — Runs entirely with Ollama, no data leaves your network
• Fast & Resilient — Built-in checkpointing, resume on crash
• Framework Agnostic — Works with LangChain, LangGraph, LlamaIndex
• CI/CD Ready — CLI-first design, JSON output, exit codes

Quick Example¶

from ragnarok_ai import evaluate, generate_testset

# Generate test questions from your knowledge base
testset = await generate_testset(
    knowledge_base="./docs/",
    num_questions=50,
    llm="ollama/mistral",
    checkpoint=True,
)

# Evaluate your RAG pipeline
results = await evaluate(
    rag_pipeline=my_rag,
    testset=testset,
    metrics=["retrieval", "faithfulness", "relevance"],
)

# Get actionable insights
results.summary()

Performance¶

Benchmarked on Apple M2 16GB, Python 3.10:

Retrieval Metrics: ~24,000 queries/sec

LLM-as-Judge (Prometheus 2):

Retrieval is pure computation — instant. LLM-as-Judge is the bottleneck (~25s/eval), but runs 100% local.

Key Features¶

Installation¶

pip install ragnarok-ai

With optional dependencies:

pip install ragnarok-ai[ollama,qdrant]

See Installation for details.

Next Steps¶

• Installation — Set up RAGnarok-AI
• Quick Start — Run your first evaluation
• CLI Reference — Command-line interface
• GitHub Action — CI/CD integration