Item by item, toward
the open science of AI evaluation.
OpenEval is an open-source, item-level repository built on a standardized schema, spanning … benchmarks, … items, and … model responses. It aims to advance open science toward more accessible, transparent, and rigorous AI evaluation.
AI evaluation requires standardized item-level data releases
Where aggregate scores fall short
Aggregate scores can obscure low-quality items, construct misalignment, and uneven model behavior, limiting the validity evidence available for assessing evaluative claims.
What item-level data enables
Item-level content, statistics, responses, and test environments support validation, auditing, replication, and deeper analyses that cannot be recovered from benchmark-level scores alone.
Why standardization matters
A consistent data format makes evaluation results comparable and reusable across benchmarks, models, and studies, reducing one-off processing and duplicate curation efforts.
How to make it practical
OpenEval serves as a large-scale evaluation data infrastructure, providing a shared foundation for the AI evaluation community to build on, extend, and maintain collectively over time.
What makes OpenEval different
Every design decision prioritizes consistency, transparency, scalability, and community participation.
Item-Centricity
Results are archived as unique items with associated fields, facilitating customized reassembly and analysis.
Broad Coverage
A growing repository of evaluation items and model responses, covering diverse models, tasks, and constructs.
Unified Schema
One JSON schema comprehensively standardizes results across different formats and benchmarks.
Faithful Snapshots
Each item captures a full experimental context, including adaptations and environments, for reproducibility.
Traceable Archive
Every item is timestamped and linked to its original source with metadata for in-depth exploration and auditing.
Open Access
All data is publicly available and freely reusable. Anyone can conveniently share their own evaluation results.