Valtron

Valtron is a Python framework for evaluating different LLMs on your own tasks and data. Valtron runs the models you care about, simultaneously, and compares them on accuracy, cost, and speed. Valtron has other advanced features such as prompt manipulations, model combinations, alerting of new models, and more. It’s open sourced or you can try the hosted version. Dig in to the docs below!

New here? Start with Install.

Valtron is proudly built and backed by InferLink.

Get started by task

Classification

Compare models on a single label per document (sentiment, topic, category), string-matched against your expected answer.

Classification

Extraction

Extract nested JSON (multiple fields, lists, sub-objects) with per-field precision/recall/F1 instead of a single pass/fail.

Extraction

Summarization

Rank models on free-text summaries with no reference summary required: an LLM judge scores each one against the source document’s own must-convey facts.

Summarization

Self-Hosting

Run inference on your own infrastructure: train a local classifier for zero-cost inference, or point Valtron at any self-hosted LLM.

Self-Hosting and Local Models

Combining Models

Route between a transformer model and an LLM, escalating to the LLM only on the cases the transformer isn’t confident about.

Combining Multiple Models

How it works

Input data (documents + expected labels)
    ↓
Config (models, prompt, manipulations)
    ↓
ModelEval.run()
    ├── Generate few-shot examples (optional)
    ├── Prepare per-model prompts (apply manipulations)
    ├── Evaluate all models concurrently
    ├── Compute metrics (accuracy, cost, time, field scores)
    └── Generate report (HTML + optional PDF)
    ↓
evaluation_report.html  ·  models/*.json  ·  metadata.json

See the full User Guide, working code in Examples, or look up a class directly in the API Reference.