Evidence and artifacts

Every submission produces artifacts you can inspect after the run. Nothing is thrown away when the run ends.

What is saved and where

• Patch: uploaded to S3. This is the exact diff bytes the gates and build used.
• Engine image: the built engine image, pushed to GHCR and pinned by digest. This is the artifact the benchmark actually ran.
• Build log: the durable output of the hermetic build. Fetched with the build log endpoint (see below).
• Round evidence bundle: the harness output for the whole round, saved to S3. It holds correctness grades and SLA timings for every image that ran on that pod. The public API does not return this tarball (evidence_s3_url is not selected), but the scoring half of it is served per entry: see the score breakdown below.

Inspect a run

Use the campaign-scoped path (preferred). The short path returns 409 when the hash is in more than one campaign.

• Timeline and latest round entry (status, score, disqualify reason): GET /v1/campaigns/{campaign_id}/submissions/{patch_hash}
• Build log, last N lines (default 200, max 2000, plain text): GET /v1/campaigns/{campaign_id}/submissions/{patch_hash}/build-log?tail=N

Both return 404 when the submission or its log does not exist.

A rejected submission never ran, so it has no round entry. A disqualified or scored submission ran; the dashboard shows that entry's status and reason.

Check your score

A round score is one number, and the arithmetic behind it is public:

GET /v1/rounds/{round_id}/entries/{entry_id}/report

It returns one row per prompt: the speedup, the token count both engines were compared at, the baseline and candidate end-to-end times in seconds, and a reason for any prompt that scored nothing. prompt_summary counts how many prompts scored and how many fell under the tolerance bar.

Use it when a local measurement disagrees with the dashboard. A gap is usually a few prompts under tolerance, which score nothing and pull the median down. The absolute seconds are there so you can compare against your own run rather than only the ratio. The environment block in the round evidence also records the validator pod's CPU, cores and cgroup quota, which is the other common source of a gap: this workload is short generations, so CPU overhead moves the numbers as much as the GPU does.

Each entry links to its breakdown from the round page on the dashboard.

How correctness is graded

Correctness does not compare your text to the baseline's text. Your engine is free to produce different tokens.

One shared scorer, the campaign's pinned baseline image, teacher-forces the output your engine actually produced and reports a logprob for each position. Your image never grades itself, and every image in the round is graded by the same scorer.

Three campaign thresholds decide the verdict:

min_token_logprob is applied to the k-th lowest position, where k = ceil(min_token_quantile * positions), not to the outright minimum. Two instances of the same image can disagree at a single high-entropy position, and that alone must not fail an otherwise faithful output.

Failing either logprob bar is disqualified. It is terminal, and the entry carries the measured value in its reason.

Falling below min_coverage_ratio is different. The scorer could not read enough to judge, which is a harness problem rather than a wrong answer, so the entry is infra_failed and requeued once.