DeepSWE experiment design

Test plan search

Find the tasks and repetitions that historically revealed model-quality changes most clearly relative to their noise.

Configure the experiment

The published pair shows which tasks changed when model quality changed, and how noisy that evidence was.

Bare model Configuration you will test

Your fresh bare-model baseline and skill/instruction run will use this exact configuration.

Comparison Published model-quality contrast

Why it matters: its score difference from the bare model measures demonstrated task response. Both configurations’ run-to-run variance measures noise.

The future skill/instruction arm has the same plan and unknown cost.
Recommended: 3
Per task, per arm · minimum 2 to estimate observed variance
Percent in each direction
Fixed method: maximize historical model difference ÷ decision threshold · equal task weights · two-sided significance level 0.05 · deterministic branch-and-bound search, limited to 50,000 states.
Set the configurations and budget, then search.

Task controls and evidence

Optionally require or exclude tasks. Any change invalidates the current result until you search again.

Choose both configurations to inspect task evidence.
How the method works

For every candidate task and repetition plan:

  1. Calculate the signed, equal-weight historical model difference from the two published configurations’ means.
  2. Separately calculate the ± decision threshold from their variances, planned repetitions, and a two-sided 0.05 Student’s t critical value.
  3. Divide the absolute historical model difference by the threshold. This unitless detectability score says how many decision thresholds separated the published configurations.
  4. Branch-and-bound searches joint task and repetition choices for maximum detectability.
  5. The result is labeled optimal only when no remaining plan can improve detectability. Otherwise the page reports the proven ceiling and remaining gap.

When both selected variances are zero, the page visibly substitutes pooled within-configuration variance from that same task in other published configurations.

detectability = |historical model difference| ÷ decision threshold
decision threshold = Student's t × planned standard error
Data and compatibility

The exported JSON is the complete task and repetition contract consumed by al benchmark.

The operational-defect exclusion list starts empty. Only confirmed execution or evaluation defects may enter it.

This planner optimizes a benchmark suite for the selected budget. Any later result applies only to the selected tasks and does not claim representativeness across DeepSWE.

Source inspection