PatchProof: Gate-Verified Repair Search and Reproducible Benchmarks for Coding Agents

PatchProof is a reproducible benchmark harness and repair-search controller for reliable coding agents. It does not to replace the model that proposes code changes, never!. It makes that model's repair loop measurable, gate-verified, route-aware, and auditable.

The central principle is unchanged:

A coding agent should not promote a patch merely because it generated one. It should generate a patch, verify it through hard gates, receive only bounded evidence when it fails, route the next attempt deliberately, and preserve enough provenance to reproduce and inspect the result.

PatchProof extends this principle to with live LLM experiments. A single code path can run compatible endpoint. Every generated candidate passes through the same scope, application, syntax, safety, visible-test, and hidden regression gates used by the deterministic controller.

The system has two complementary evaluation modes:

• A deterministic benchmark with curated candidate pools and ablations. This remains the controlled mechanism study.
• A live LLM benchmark that generates and evaluates patches across unique task families and repetitions. This measures observed reliability, attempts, token usage, latency, and cost when available.

PatchProof does not claim that a benchmark result generalizes to every repository, model, or provider. It provides a disciplined way to produce those measurements and retain the artifacts needed to inspect them.

PatchProof is a reliability layer around a model's patch proposals.

  flowchart LR
    A["Task specification and source workspace"] --> B["LLM or curated candidate patch"]
    B --> C["Hard verification gates"]
    C --> D{"All gates pass?"}
    D -- "yes" --> E["Promote candidate"]
    D -- "no" --> F["Build bounded EvidencePacket"]
    F --> G["Route next repair"]
    G --> H["Request or select next candidate"]
    H --> C
    C --> I["Archive, trace, provenance, and metrics"]
    I --> J["Showcase, benchmark, and model-matrix reports"]
    

The same structure supports two sources of candidates:

Candidate source	Purpose	Selection behavior
Curated patch pool	Deterministic ablations and reproducible controller tests	Uses structured candidate metadata or ordered fallback.
Live LLM response	Live repair demos, benchmark runs, and model comparisons	Generates a patch from the current bounded context and then enters the same verification loop.

This separation is important. PatchProof does not present a curated result as a live-model result, and it does not let a live model bypass the controller simply because its output was generated online.

Every candidate, whether curated or generated by an LLM, is evaluated in an isolated workspace. A candidate is promoted only after each required gate passes.

  flowchart TD
    A["Candidate patch"] --> B["Scope guard"]
    B --> C["Patch applies"]
    C --> D["Python AST parse"]
    D --> E["Secret scan"]
    E --> F["Visible developer tests"]
    F --> G["Release-gate regression tests"]
    G --> H["Promote"]

    B -- "fail" --> R["Reject and build bounded evidence"]
    C -- "fail" --> R
    D -- "fail" --> R
    E -- "fail" --> R
    F -- "fail" --> R
    G -- "fail" --> R
    

The release gate is intentionally different from visible tests. A visible failure may reveal a bounded stdout or stderr tail so the next repair can be targeted. A release-gate regression exposes only limited diagnostics, such as test identifiers, and withholds assertion values and full output. This preserves a hidden-test boundary and reduces the risk of direct overfitting to withheld cases.

The PatchProof's live repair loop accepts unified diffs and structured search/replace blocks. Search/replace blocks are converted to a patch against the original task source, with matching diagnostics preserved in the trace. This gives models a robust edit format while retaining gate-verifiable patch artifacts.

  sequenceDiagram
    participant P as PatchProof
    participant M as LLM provider
    participant V as Verification gates
    participant A as Archive

    P->>M: Task context and allowed paths
    M-->>P: Patch plus optional metadata
    P->>V: Evaluate candidate in isolated workspace
    V-->>P: Gate outcomes
    P->>A: Store response, patch, evidence, route, and metrics
    alt Candidate promoted
        P-->>P: Finish with promoted status
    else Candidate rejected
        P->>M: Bounded evidence and repair route
    end
    

Live LLM evaluation is variable by nature. PatchProof makes the experimental configuration explicit rather than pretending variability does not exist.

Each LLM run records reproducibility metadata including:

• PatchProof source revision.
• Model, provider, and endpoint.
• Temperature, reasoning setting, and optional sampling seed.
• Candidate and repeated-failure budgets.
• A stable configuration fingerprint.

The optional disk cache keys each response by the model, reasoning configuration, provider, endpoint, temperature, seed, and prompt messages. This prevents a response from one server or sampling configuration from being silently reused for another experimental condition.

  flowchart LR
    A["Run configuration"] --> B["Canonical cache context"]
    B --> C["Model, provider, endpoint, temperature, seed"]
    C --> D["Prompt messages"]
    D --> E["Hashed cache key"]
    E --> F{"Cached response exists?"}
    F -- "yes" --> G["Reuse archived response"]
    F -- "no" --> H["Call provider and persist response"]
    

Caching is useful for repeatable analysis and cost control. It should not be mistaken for deterministic model behavior when cache misses occur. A seed is forwarded to the provider and recorded, but practical determinism still depends on the provider and model honoring that seed.

Controlled Deterministic Benchmark

The original deterministic benchmark remains the controller-ablation study. It replays structured candidate pools across the following baselines:

Baseline	What it tests
single_shot	One candidate, no correction loop.
linear_retry	Ordered retry under the same candidate budget.
clean_context_review	Ordered retry without route-aware selection or archive behavior.
archive_no_routing	Archive-compatible retry without route-aware selection.
evidence_aware_review	Bounded evidence plus candidate metadata without the full archive and policy machinery.
full_patchproof	Full PatchProof controller behavior.

The earlier report recorded the following controlled result shape:

Baseline	solve@budget	95% CI	Avg attempts
single_shot	0.2	0.105-0.3476	1.0
linear_retry	0.6	0.446-0.7365	1.8
clean_context_review	0.6	0.446-0.7365	1.8
archive_no_routing	0.6	0.446-0.7365	1.8
evidence_aware_review	0.9	0.7695-0.9604	1.8
full_darwinpatch	0.9	0.7695-0.9604	1.8

This result remains intentionally narrow. It demonstrates the value of bounded evidence and route-compatible metadata under a controlled candidate pool. It does not establish a live-model ranking.

Live LLM Benchmark

benchmark-llm runs the live repair loop across each unique task family in a suite. It supports repeated runs, bounded candidate budgets, provider selection, endpoint configuration, caching, temperature, seed, reasoning, and optional pricing.

For each task and repetition, PatchProof records:

• Promotion status and stop reason.
• Number of attempts and failure types.
• Token totals, prompt tokens, completion tokens, and token source.
• Cost and currency when complete provider usage and a schedule are available.
• Wall-clock duration.
• Candidate archive, response artifacts, patch artifacts, and controller trace.

The result is a live measurement harness. It reports observed solve@budget and confidence intervals, but does not fabricate a universal benchmark score before a real experiment is run.

Live-Generated Candidate Pools

benchmark --llm-generate first runs a live LLM repair search to construct a candidate pool, then replays that pool through the deterministic baselines. The generated pool preserves patch files and metadata artifacts, allowing a controlled comparison of baseline policies against the same live-generated proposals.

This mode is useful when the research question is not only "which model solved a task?" but also "how does each controller baseline behave when given the same generated candidates?"

PatchProof can compare multiple models under the same suite and budget. Supplying --llm-model more than once invokes a model-matrix experiment.

PatchProof expands beyond the earlier single-file task families with coordinated multi-file repairs:

Task	Allowed repair surface	Example of coordinated behavior
csv_report	Two source modules	Row normalization and report summary behavior.
pricing_cart	Two source modules	Cart and pricing behavior.
order_service	shop/pricing.py, shop/inventory.py, and shop/checkout.py	Rounding, validation, inventory, and checkout flow.

The order_service task is intentionally scope-restricted to these three files. A candidate that changes the catalog, specification, or tests is rejected by the scope guard before it can be promoted.

The multi-file suite demonstrates coordinated changes and scope control. It does not claim that PatchProof has solved large production repositories or arbitrary architectural changes.

The guarded policy experiment from the earlier work remains available. It splits cases into train, validation, and held-out test partitions. Policies are tuned on train data, selected using validation results, and evaluated on the held-out test split only after selection.

  flowchart LR
    A["Source benchmark suite"] --> B["Train split"]
    A --> C["Validation split"]
    A --> D["Held-out test split"]
    B --> E["Policy tuning"]
    C --> F["Policy selection"]
    F --> G["Selected policy"]
    G --> D
    D --> H["Final report only"]
    

This protects against selecting a repair budget or stopping policy based on the same held-out cases used to report its final result.

PatchProof is a rigorous technical harness, not a claim of universal autonomous software engineering. Important limitations remain:

• The deterministic benchmark still uses curated candidate pools. This is intentional for controlled ablation, but it is not a substitute for a broad live-model study.
• Live benchmark results depend on the provider, model version, endpoint behavior, prompt format, sampling configuration, and task suite. A cache or seed improves reproducibility but does not guarantee deterministic provider behavior.
• The included tasks are Python repairs and compact multi-file workflows. They do not represent long-running product development, complex build systems, security-sensitive deployment pipelines, or very large repositories.
• Release-gate tests simulate a hidden boundary within repository-controlled tasks. They are not a full replacement for independent real-world acceptance testing.
• Cost is reported only when a provider returns complete usage data and the user supplies a matching price schedule. This is deliberate, because incomplete accounting should not be presented as a precise monetary measurement.
• Generator-declared metadata is useful provenance, not proof that the model's self-description is correct. PatchProof validates route names but continues to rely on gate outcomes for promotion.
• The model matrix reports comparative observations and uncertainty. It does not by itself establish causal superiority or broad statistical generalization.