The item-centered schema
OpenEval organizes results at the item level under a unified schema, making cross-benchmark analysis and community contribution straightforward.
Each entry in OpenEval is built around an item — the atomic unit of evaluation.
- item_id Unique identifier for the item.
- item_metadata Ingestion timestamp, contributor info, and benchmark provenance.
- item_content[] Original item content from the source benchmark, which is divided into input[] and references[].
- responses[] Response(s) to the item (see the next tab).
Given the item, each response records the test conditions and outcomes for a single request.
- response_id Unique identifier for the response, prefixed with the associated item_id.
- model Respondent model (see the next tab).
- item_adaptation How the item was adapted for the evaluation run, including the actual request_input[], few-shot demonstrations, and test environment resources.
- response_content[] The output returned by the model.
- scores[] Metric score(s) of the response (see the last tab).
- response_contributor Present when the model response and corresponding item come from different submissions/contributors.
Each model records the respondent model's identity and associated test settings.
- nameName of the respondent model.
- size Parameter count or size label.
- model_adaptation How the model was adapted for the evaluation run, including the system instruction, generation config, and available tools.
Each metric extracts capability evidence from a response and produces a score.
- nameName of the evaluation metric.
- models[] Judge model(s) used, if the metric is model-based.
- extra_artifacts[] Additional artifact(s) used for scoring, e.g., evaluation rubrics.
{
"item_id": "wildbench_20260220T224823Z_0", # AUTO
"item_metadata": {
"ingestion_time": "20260220T224823Z", # AUTO
"contributor": {
"name": "Anonymous", # OPTIONAL
"email": "", # OPTIONAL
"affiliation": "OpenEval Commons", # OPTIONAL
},
"source": {
"benchmark_name": "WildBench",
"benchmark_version": "v2",
"paper_url": "https://iclr.cc/virtual/2025/poster/29940", # OPTIONAL
"dataset_url": "https://huggingface.co/datasets/allenai/WildBench/viewer/v2", # OPTIONAL
"benchmark_tags": [
"real-world",
"Instruction Following",
"multi-turn chat"
] # OPTIONAL
}
},
"item_content": {
"input": [
{
"role": "user",
"content": "answer all of the questions you can (put the question and then the answer under it): ..."
},
{
"role": "assistant",
"content": "The function that models the data in the table is:\n\nF(x) = 200 * (1/2)^x"
},
{
"role": "user",
"content": "no answer all of the questions given"
}
],
"references": []
},
"responses": [
{ # See the next tab for a response },
...
],
"schema_version": "v0.2.0" # AUTO
}
{
"response_id": "wildbench_20260220T224823Z_0_gemini-3-pro-preview_0", # AUTO
"model": { # See the next tab for a model },
"item_adaptation": {
"request_input": [
{
"role": "user",
"content": "answer all of the questions you can (put the question and then the answer under it): ..."
},
{
"role": "assistant",
"content": "The function that models the data in the table is:\n\nF(x) = 200 * (1/2)^x"
},
{
"role": "user",
"content": "no answer all of the questions given"
}
],
"demonstrations": [], # OPTIONAL
"external_resources": []
},
"response_content": [
{
"text": "Here are the answers to the questions provided...",
"logprob": 0.0,
"tokens": [],
"finish_reason": {
"reason": "stop"
},
"thinking": {
"text": "**Evaluating the Problems**\n\nI've meticulously assessed the problems..."
}
}
],
"scores": [
{
"metric": { # See the last tab for a metric },
"value": 0.7777777777777778
},
...
],
"response_contributor": { # Same format as item_metadata.contributor } # OPTIONAL
}
{
"name": "Gemini 3 Pro Preview",
"size": "", # OPTIONAL
"model_adaptation": {
"system_instruction": "",
"generation_parameters": {
"temperature": 0.0,
"do_sample": false,
"top_k": 1,
"top_p": 1.0,
"max_tokens": 12000,
...
},
"tools": []
}
}
{
"name": "wildbench_score_rescaled",
"models": [
"gpt-4o"
],
"extra_artifacts": [
{
"type": "checklist",
"content": [
"Does the AI output provide answers to all the questions listed in the user's query?",
"Are the answers provided by the AI accurate and correctly calculated based on the information given in the questions?",
"Does the AI output maintain clarity and coherence in presenting the answers to each question?",
...
]
}
]
}
Loading dataset coverage…
Live data viewer
For practical reasons related to Parquet storage, our schema is organized into three types of tables on Hugging Face: benchmark, item, and response.