Cheapest Model for Autoresearch: Minimum to Strong (2026)

The short answer: There is no verified universal minimum model for an autonomous research loop. If the loop edits code, trains or tests it, reads results and decides what to keep, the necessary model depends on the codebase, difficulty of the modifications, tool reliability and how objective the score is. For a small, highly constrained loop, try GPT-6 Luna, Claude Haiku 5.5 or DeepSeek V4.1 Flash as low-cost qualification candidates. For a reasonable starting point on nontrivial experiments, trial Claude Sonnet 5.5 or GPT-6.1 Sol, with Gemini 3.8 Flash and DeepSeek V4 Pro as alternatives. For harder experimental design and recovery, evaluate Claude Opus 5.5, Gemini 4 Argon or GPT-6 Astra. These are test tiers, not published autoresearch pass-rate rankings.

The real question is: What is the cheapest agent that produces repeatable, valid improvements instead of merely keeping a GPU busy? The calculator supplied with this article estimates your overnight GPU + API spend for each tier, and the qualification protocol below prevents you from mistaking random metric movement for scientific progress.

What “autoresearch” actually does

Andrej Karpathy's autoresearch repository is deliberately minimal. One file (train.py) is in scope for modification; the evaluator and data preparation code in prepare.py are fixed; the agent runs a five-minute training experiment, measures validation bits per byte (val_bpb) and either keeps or discards the proposed change. Its program.md instructs the agent to commit a change, run training, inspect metrics, append a result to results.tsv, then continue until stopped.

That concrete setup matters: the agent is not the model being trained. The agent's LLM reasons about edits and chooses experiments; the training GPU evaluates those edits. A $0.10/1M-input API model does not replace the GPU that must run train.py. An eight-hour session with roughly five minutes per experiment could approach 96 trials before startup, crashes and evaluation overhead, not a guarantee of 96 successful trials. The original repo was tested on an H100; other hardware may run differently.

And a loop is only valuable if it:

,
1. Produces a valid, in-scope change without breaking the harness.
,
2. Correctly diagnoses training crashes and understands the metric direction.
,
3. Improves the held-out quality signal, not just an easily gamed score.
,
4. Remembers failed attempts and does not retry identical ideas indefinitely.
,
5. Leaves a reproducible artifact that beats the baseline when independently rerun.
,

A very cheap model can satisfy the first two on toy edits and fail spectacularly at the last three.

Three practical model tiers - and where the minimum breaks

Selection rule: A tier is qualified only when it completes your frozen trial set with acceptable syntax/test pass rate, unique valid hypotheses, repeatability and total cost. The names above are proposed models to evaluate, not evidence that each passed the original autoresearch benchmark. We did not run a head-to-head multi-model autoresearch experiment while preparing this article.

Use strong models for planning or reviewing and cheaper models for executing mechanical candidates if the toolchain can cleanly separate those roles. Do not assume a small model can be its own reliable judge.

Model API cost is usually not the whole experiment cost

Here is a fully specified illustrative budget, using the providers' public October 17, 2026 Standard/first-party rates. Suppose an overnight loop runs 100 experiments, each with exactly one 30K-input / 2K-output model call, and 5 minutes of GPU time at an assumed $2/hour. No caching, retries, failed startup, tools or extra reasoning turns are included. These are calculator assumptions, not observed traces.

Pricing references: OpenAI, Anthropic, Google, DeepSeek. The sample GPU price is hypothetical; it is not a quotation from any cloud provider. Google's Argon introductory price doubles December 1. DeepSeek has weekday peak-rate windows. The five-minute run in Karpathy's example excludes startup/evaluation overhead; a real loop's end-to-end wall time can be longer.

This table contains a deliberately optimistic assumption: one LLM call per experiment. An actual coding agent can call the model multiple times to inspect files, write patches, interpret errors, and summarize prior experiments. If your loop averages four calls of this size, multiply the API part by four, not the fixed GPU part. Don't price an eight-hour agent session from one short chat completion.

What does a cheap loop need to achieve to be worth running?

A useful decision metric is fully reproducible improvements per dollar, not total attempts. Consider 100 training experiments at the same $16.67 GPU cost:

,
• Budget agent A costs $0.40 in API usage and yields one independently reproduced improvement: about $17.07 per improvement.
,
• Mid-tier agent B costs $8.00 and yields four independently reproduced improvements: about $6.17 per improvement.
,
• Strong agent C costs $40.00 and yields five independently reproduced improvements: about $11.33 per improvement.
,

Those yields are invented sensitivity inputs, not observed model performance. The point is that the middle model wins despite paying 20 times more per attempt than the budget agent. You cannot know that ranking without performing the trial. If a model makes more ambitious but invalid changes, its raw count of generated experiments is irrelevant.

This is especially true with stochastic evaluations: an agent that sees small, noisy improvements and keeps every one can make the development metric look better while the final holdout degrades. Lock the evaluator, rerun winners and keep a separate untouched final holdout.

The 30-experiment qualification test (copy this)

You do not need to run 100 experiments on every candidate before rejecting an obviously unsuitable model. Use a constrained 10 warm-up + 20 scored experiment qualification, with the same seed plan and per-experiment budget across candidates. Thirty runs are a screening test, not a reliable estimate of long-term discovery productivity.

Phase A - 10 warm-up experiments: Ensure it edits only allowed files, runs the exact command, parses the right metric, reverts bad changes, recovers from one ordinary crash and creates the expected artifact. Reject models that persistently violate the harness or cannot finish experiments.

Phase B - 20 scored experiments: Record valid commits, unique hypotheses, crashes, repetitions, test pass rates, metric improvements, independent reruns, GPU minutes, all model tokens and human interventions. Run a known-good stronger baseline on the same protocol. Prefer repeated comparable batches over drawing conclusions from one lucky result.

Use this minimal result schema (CSV or JSONL):

{"run_id":"trial-017","model":"provider/model@version","candidate_commit":"abc1234",, "hypothesis_id":"h-12","valid_edit":true,"training_completed":true,, "metric_before":1.0031,"metric_after":1.0019,"reproduced_on_holdout":false,, "input_tokens":34000,"cache_read_tokens":8000,"output_tokens":3100,, "agent_calls":3,"gpu_seconds":329,"review_minutes":0,"failure_reason":null}

The values are fabricated to illustrate fields. Capture all agent calls; a model's billed reasoning output belongs inside output counts. Preserve the commit and execution logs so a human can explain why a candidate was kept.

Hard boundaries for a repeatable loop:

Allowed changes: train.py only,Never change: prepare.py, benchmark fixtures, evaluator, stop condition,Run command: uv run train.py > run.log 2>&1,Pass gate: process exit 0 AND parse finite val_bpb AND time within budget,Keep rule: candidate improves development metric AND does not break tests,Final gate: independent rerun + untouched holdout before reporting a win,Stop rule: cost budget, wall-time budget, repeated invalid edits, no new hypotheses

This extends the original autoresearch program with operational safety checks and a distinct final evaluation gate. It is not a claim that Karpathy's repo ships those checks as written. In production, use process isolation for executing agent-authored code, resource limits, and restricted credentials. A training script is arbitrary executable code, not a harmless text completion.

Why a model can look brilliant while doing useless research

Metric direction errors. val_bpb is lower is better. If the agent treats it as an accuracy percentage and keeps higher scores, the loop literally optimizes backward.

Evaluation tampering. If a model can edit prepare.py or result parsing, it can “win” by changing what is measured. The immutable evaluator is a core feature of the minimal design. Store the evaluator hash and report it alongside the result.

Lucky validation noise. Selecting the best of many noisy trials inflates the apparent improvement. Rerun candidates with fixed seeds and compare on a separate holdout; report run-to-run variance, not only the best run.

Idea recycling. Small models may repeatedly adjust the same learning-rate parameter or revert to tried configurations. Keep a durable hypothesis ledger, not just a growing agent context. The autoresearch community has discussed structured result history and recovery for exactly these problems.

Unpriced human work. A model that produces 50 failed edits and one good result may keep the GPU busy while forcing expensive human cleanup. Track review minutes and false winner promotions per model.

A deployment strategy that avoids burning money

Start with the cheapest model that can pass Phase A. If multiple qualify, run the scored 20-experiment screen. Compare replicated improvement yield per dollar, plus the rate of invalid changes and manual interventions. Escalate to a stronger model when a run is stuck on repeated ideas, ambiguous failures or complex architecture design; downgrade execution when the next step is mechanical. Stop the entire loop on predefined compute and spend budgets, not on model enthusiasm.

The GPU time required by the experiment and the API time required by the agent must be metered separately. Your core cost events are agent_model_call, training_gpu_seconds, experiment_attempt, validated_result and human_review_minutes. If you serve this as a product, UsageBox can record numeric quantities for each meter and preserve retry-safe evidence; it does not execute research or judge model quality. The existing LLM cost-per-task calculator helps model retry and token assumptions before traffic exists.

Related reading and primary sources

,
• Karpathy's autoresearch repository and program.md: fixed five-minute experiment, edit scope, keep/discard protocol.
,
• OpenAI model pricing; Anthropic model pricing; DeepSeek pricing; Google Gemini API pricing.
,
• Autoresearch orchestration limitations, community discussion: reproducibility and experimentation workflow.
,
• UsageBox: cheapest agentic APIs covers broader workhorse selection. The companion model-price chart answers rate comparisons, while the companion self-hosting TCO guide includes quality and electricity.
,

Pricing and repository behavior verified October 17, 2026. No original model qualification experiment was performed for this article; all illustrative yield rates are explicitly hypothetical. Do not market candidate tiers as benchmark winners.