Open Source & Managed Service

Find the model that is
Just Right.

Valtron replaces "vibes-based" AI selection with rigorous, quantifiable data. Compare local and API-based models on Cost, Speed, and Accuracy for your specific classification and extraction tasks.

View the Open Source Engine Explore the Hosted Lab

Stop Guessing

Replace vibes-based testing with rigorous evaluation. Benchmark LLM models against your actual labeled data to prove exact accuracy for your specific tasks.

Optimize Cost & Speed

Stop overpaying for premium models if a cheaper one does the job. Find the mathematical sweet spot where maximum accuracy meets minimum cost and latency.

The Watchtower

Never miss a model drop. Get alerts for new releases to see if an upgrade is worth it. When providers sunset your current model, effortlessly test and find the perfect replacement.

Part 1: The Open Source Python Library

The Scikit-Learn of
LLM Evaluation.

A deeply rich configuration file wrapped in a very simple API. Built for developers who need extreme precision and complex comparisons.

View on GitHub Read Documentation

Core Capabilities

Rich Configuration: Control behavior, prompts, and evaluation metrics via config file.
Complex Assessments: Multiple fields per input document, supporting Exact Match, Fuzzy Match, LLM Referees, or custom extensible Python logic.
Multi-Modal Ready: Evaluate vision and text models side-by-side.
Hybrid Comparisons: Include predicted results from other legacy/local models to compare directly to LLMs.
Prompt Manipulations: Auto-optimizations to boost the performance of weaker (cheaper) models.
Wizard Setup: A built-in webpage UI to easily configure your experimental run.
Beautiful Reporting: Outputs rich, interactive JS-based histograms or professional PDF summaries for stakeholders.


                        $pip install valtron-core

from valtron_core.recipes import ModelEval

# All equivalent, choose any
experiment = ModelEval(config={"models": [...], "prompt": "..."}, data=[{"id": "1", "content": "...", "label": {...}}])
experiment = ModelEval(config="task_config.json", data="ground_truth.json")
experiment = ModelEval(config=Path("task_config.json"), data=Path("ground_truth.json"))
experiment = ModelEval(config=my_config_obj, data=data)

# run() - one shot
# Synchronous (default) - runs evaluation & saves results
report_path = experiment.run()

Generate your config automatically

Don't want to write JSON from scratch? Use our visual Configuration Wizard to select models, define metrics, and export your config file instantly.

Read about the Wizard →

Part 2: The Managed Service

Your Models,
On Autopilot.

We host your experiment and handle the compute. When the market moves, we tell you exactly how it impacts your stack.

Try the Hosted Lab

The "New Model" Sentinel

When a new model is released, the system immediately asks if you want to run it against your specific experiment. Decide if an upgrade is actually "worth it."

Sunset Insurance

Is your current model being deprecated? Effortlessly test successors against your historical baseline to ensure zero regression in production.

The Local Prover

Test locally hosted models on our infrastructure to see if the cost of self-hosting is truly "worth it" compared to managed APIs.

The Set-and-Forget Workflow

Setup

Configure your prompt, models, and upload your labeled data via our secure UI.

Baseline

We run the experiment on our infrastructure, mapping your current cost/accuracy matrix.

Monitor

Act as a CI/CD test framework. Get notified when a new model hits the market and test it.

Find the model that isJust Right.