One-Eval
An Agentic System for Automated and Traceable LLM Evaluation — from natural language to full evaluation reports.
An Agentic System for Automated and Traceable LLM Evaluation — from natural language to full evaluation reports.
Zero setup. Just paste this to Claude Code (or Codex, or any coding agent):
Use the one-eval-skill in https://github.com/OpenDCAI/One-Eval to get us started on evaluating my model.
One-Eval orchestrates evaluation through a multi-stage agentic pipeline built on DataFlow and LangGraph.
NL2Eval: From natural language requirements to complete evaluation reports, fully automated.
Input natural language evaluation goals and automatically parse intent, structure requirements, and plan personalized benchmark suites.
Automatic benchmark resolution, dataset acquisition, and schema normalization — supports both public benchmarks and private datasets.
Handles data preparation, inference, metric matching, scoring, and report generation in one traceable pipeline.
Interrupt and adjust evaluation strategies at key checkpoints. Review, edit, and rollback at any stage.
Built-in benchmark catalog covering reasoning (MATH, GSM8K, BBH, AIME), knowledge (MMLU, CEval), and instruction following (IFEval).
Built on DataFlow operators and LangGraph state management. Supports custom metrics and extensible evaluation workflows.
One-Eval is an agentic evaluation system that converts natural-language evaluation requests into executable, traceable, and customizable evaluation workflows. The system consists of three core components: NL2Bench for intent structuring and personalized benchmark planning, BenchResolve for benchmark resolution, automatic dataset acquisition, and schema normalization, and Metrics & Reporting for task-aware metric selection and decision-oriented reporting. The system includes human-in-the-loop checkpoints for review, editing, and rollback, and preserves sample evidence trails for debugging and auditability. Experiments demonstrate it can execute end-to-end evaluations from diverse natural-language requests with minimal user effort.