DeltaSelect

DeltaSelect is for developers who need to measure whether an instruction, Agent Skill, model, harness, or workflow change actually helps. Run selected tasks through coding-agent subscriptions you already pay for, instead of spending thousands of dollars on full benchmark runs through custom harnesses. Use it to plan a smaller evaluation with a clear budget, a fixed task set, and visible limits on what the result can tell you.

@article{conn2026deltaselect,
  title   = {DeltaSelect: Affordable A/B Testing for Coding Agents},
  author  = {Conn, Nicholas J.},
  journal = {arXiv preprint arXiv:2609.19607},
  year    = {2026}
}

Why select benchmark tasks?​

Running every task in a coding benchmark can be expensive, especially when each task needs several runs. Picking a few tasks at random is cheaper, but the result may be noisy or may not reflect the full benchmark.

DeltaSelect gives you a more deliberate way to choose:

• Set a budget. DeltaSelect stops adding tasks before the estimated run cost exceeds the amount you choose.
• Choose tasks backed by prior results. It prefers tasks that have tracked the full benchmark consistently, while accounting for uncertainty and room for improvement.
• Define the score before you run. Each selected task gets a fixed weight, so the scoring method cannot change after results arrive.
• See the tradeoffs. The tool shows excluded tasks, missing data, estimated cost, expected score, and uncertainty.
• Repeat the study. Export selection.json, then run al benchmark init to generate study.toml for al benchmark run.

Try DeltaSelect​

The interactive tool uses a reviewed snapshot of repeated DeepSWE runs. Choose a model, reasoning level, budget, number of runs, and optional minimum room for improvement. DeltaSelect shows the selected tasks, their weights, the estimated score and cost, and the data behind each choice.

You can also compare a second model configuration against the same tasks and weights. The comparison does not change which tasks DeltaSelect selects.

Run the study​

DeltaSelect connects directly to Agent Layer's benchmark runner:

1. Build and export selection.json from the interactive tool.
2. Run al benchmark init selection.json --directory benchmark-study to create a self-contained study.
3. Run al benchmark run benchmark-study/study.toml --dry-run to validate inputs, task environments, and provider authentication without inference calls.
4. Run al benchmark run benchmark-study/study.toml to authorize paid calls, execute missing cells, and preserve immutable evidence.
5. Compare score, cost, workflow conformance, and statistical evidence in the generated report.

Use the same declared task set to test an instruction change, a new Agent Skill, an Agent Dispatch workflow, or another coding-agent improvement.

Scope and interpretation​

DeltaSelect does not make a small task set equivalent to the full benchmark. A result describes the selected tasks, and past benchmark behavior cannot predict exactly how a new change will perform. Use the cost and uncertainty shown by the tool when deciding how much confidence to place in the result.