Dynamic workloads

This page covers how Pareton picks prompts for a round. Ranking and the correctness gate live on Pipeline stages.

Problem

If every miner sees the same public prompt list, they can overfit those prompts. We also cannot keep a secret holdout set.

The workload must be:

1. Unknown to the miner at commit time.
2. Public and checkable after the run (no secret prompts).
3. The same for every image in one round, so scores in that round are comparable.

Idea

1. Pin a HuggingFace dataset on the campaign (sampling_rule). The pin includes the dataset name, a git revision, how many rows it has, and how many prompts to draw.
2. When a round is created, take a chain block that has already finalized. That block's hash was not known at any miner's commit time.
3. Turn that hash and the campaign id into a seed. Fetch the chosen rows, build a trace, and store the exact JSON the round ran (sampled_trace_sha256 plus a sampling receipt). That receipt is checkable. It is not a published prompt deck.
4. Run the baseline, the leader, and every challenger on that same trace.

The patch hash is not in the seed. Every image in the round must see the same prompts.

Campaign setup

Set sampling_rule on a draft campaign. Type hf_rows, with dataset, revision (a git SHA, not main), config, split, n_rows, n_prompts, max_tokens, and algo_version (1). Every campaign pins this rule. There is no campaign-level fixed trace.

The sampling rule stays on the campaign. The realized trace is snapshotted onto the round (sampled_trace_sha256 plus the receipt).

Per round

1. The seed block is the chain head minus finality depth at the moment the round is created. That block is already settled. The watcher does not seed on the tip and does not wait for a later block.
2. Seed = sha256(block_hash || campaign_id) (64-char lowercase hex, no sha256: prefix). The patch hash is not in the material, so every image in the round draws the same rows.
3. Row index = sha256(seed + ":" + counter) % n_rows. Skip empty prompts and prompts longer than 8000 characters. Fetch only the kept rows.
4. Build the trace, hash it, store the hash and the receipt on the round. Every engine in the round scores those prompts.

Every round draws a different prompt set. Scores are comparable inside one round only.

Absolute p99 latency is not a ranking input. Each round draws a different trace, so raw latency would be a lottery. The score is a ratio against the baseline in that same round.

What a score means

A score is median per-prompt end-to-end speedup against the baseline on this round's trace. It is not a claim about any other prompt set. Compare two submissions only when they sat in the same round.