AutoResearch Review 2026: Autonomous Research Agent

What AutoResearch is

AutoResearch is Andrej Karpathy's harness for AI agents that run machine learning research experiments on their own. The repository at github.com/karpathy/autoresearch shows 98,627 stars, and the project has no listed price — the pricing page lists $0/mo. That combination tells you most of what you need to know before reading further: this is an open harness, not a hosted product with a support contract.

The scope is narrower than the name suggests. The repository describes a system that runs ML research experiments autonomously on single-GPU nanochat training. It is not a general web research agent that will answer questions about market sizing or competitor pricing. It is a loop that proposes training experiments, runs them on one GPU, reads the results, and decides what to try next. If you arrived expecting a Perplexity competitor, you are looking at the wrong repository.

The architecture follows the standard agent pattern the field settled on. A planner decomposes the objective into candidate experiments. A search or retrieval tool gathers context — in this case, prior runs, configs, and relevant code. A reader inspects outputs: loss curves, logs, eval numbers. A synthesizer writes up what happened and feeds the next iteration. The model dependency is the interesting part. Because the loop is long and the outputs are verbose, the cost of each step is dominated by output tokens, and the pricing snapshot shows how wide that spread is: o1-pro lists $150/M in and $600/M out, while gpt-5.5-pro lists $30/M in and $180/M out. A loop that runs dozens of iterations multiplies whatever you pick.

What AutoResearch is not: a managed service, a product with an SLA, or a tool with a polished onboarding flow. The repository is the product. That is fine for the audience it targets and a problem for everyone else.

Setup in 2026

There is no install page and no hosted signup. Setup means cloning the repository, wiring a model provider, and pointing the harness at a GPU. The repository shows the code; the docs describe the expected environment. What the docs do not do is hold your hand through provider selection, which is where most of the real work sits.

The model wiring is the decision that matters. AutoResearch needs a model that can follow multi-step instructions, emit structured experiment proposals, and tolerate long context. The pricing snapshot gives a sense of the tiers available in 2026. At the top, o1-pro at $150/M input and $600/M output is priced for people who do not look at the bill. One tier down, claude-opus-4.7-fast and gpt-5.5-pro both list $30/M input, with output at $150/M and $180/M respectively. Further down, gpt-5-pro lists $15/M input and $120/M output, and o3-pro lists $20/M input and $80/M output. Batch variants exist for some models — o1-pro:batch lists $75/M input and $300/M output, and gpt-5.5-pro:batch lists $15/M input and $90/M output — but a research loop that reacts to each result is not a batch workload, so those tiers mostly do not apply here.

For a first task, the sensible path is a small nanochat training run with a tight experiment budget. The harness proposes a change, runs it, reads the result, and proposes the next. Time to first useful report depends almost entirely on how long each training run takes, which is a property of your GPU and your config, not of the agent. The agent's own overhead — planning, reading, writing — is small next to the training time. That is a useful thing to internalize early: on this workload, the model is rarely the bottleneck.

If you are choosing a provider for the loop, the OpenRouter review covers routing across several of these models from one key, which saves you from rewriting the client every time you want to try a cheaper tier. The cheapest AI API breakdown is the other page worth reading before you commit, because the difference between a $15/M output model and a $180/M output model compounds fast across a long loop.

How the research loop works

The loop has four stages, and each one has a failure mode worth knowing before you run it.

Query decomposition

The planner takes the research objective and breaks it into candidate experiments. In an ML research context, that means things like "try a different learning rate schedule," "change the tokenizer," or "adjust the batch size." The quality of this stage sets the ceiling on everything after it. A planner that proposes trivial variations will burn GPU hours confirming what you already know.

Source gathering

The reader pulls in prior results, configs, and relevant code. This is where the harness differs from a web research agent: the "sources" are your own run history and the repository, not the open web. That makes citation quality less of a concern and context management more of one. Long run histories blow through context windows, and the harness has to decide what to keep.

Critique pass

Before committing to a final write-up, the agent reviews its own reasoning. This is the stage that separates a real research loop from a one-shot answer. The critique pass catches proposals that contradict earlier results, or that repeat an experiment already run. It also adds latency and tokens, and on narrow questions it can loop without converging.

Final report and stop conditions

The synthesizer writes up the run. Stop conditions are the part most users underestimate. The harness needs a budget cap, an iteration cap, and a convergence check. Without all three, a loop can run indefinitely, proposing variations that produce no improvement. The repository shows the mechanism; the discipline of setting the caps is on you.

Head-to-head: AutoResearch vs one-shot deep research

The honest comparison is not AutoResearch against Perplexity or OpenAI deep research on a web question. Those tools answer questions from the open web; AutoResearch runs experiments on a GPU. Comparing them on the same five prompts would be measuring the wrong thing. What is worth comparing is the shape of the loop: multi-step agent versus one-shot answer engine, on the kind of task each is built for.

On a web research question — say, "what did the major labs publish about training efficiency this quarter" — a one-shot deep research tool wins on every axis that matters. It returns in minutes, cites sources, and costs a fraction of a long agent loop. AutoResearch has no web search tool in the repository, so it cannot do this task at all. That is not a knock; it is a scope statement.

On an ML experiment question — "which of these five config changes improves nanochat training loss the most" — the loop wins, and it wins for a structural reason. A one-shot engine can tell you what the literature says about learning rate schedules. It cannot run the experiment. AutoResearch can, and the result is specific to your setup rather than borrowed from a paper.

Scoring the two on the dimensions that matter for an agent review:

• Citation count. One-shot web tools produce more citations because they read the web. AutoResearch cites your own runs, which is fewer sources but higher relevance per source.
• Source quality. AutoResearch wins on relevance, loses on breadth. A one-shot tool can pull a paper you would never have found; AutoResearch will never find it for you.
• Factual error rate. The loop's critique pass reduces errors on experiment interpretation. It does not eliminate hallucinated citations in the write-up, which is the failure mode covered below.
• Wall time. One-shot tools return in minutes. AutoResearch returns when the experiments finish, which is measured in GPU hours, not model latency.
• Token cost. A one-shot deep research query is a single expensive call. An AutoResearch loop is many cheaper calls, and the total is usually higher because the loop runs long.

The takeaway is that these are not substitutes. The LLM benchmark comparison is the better place to settle which model to run inside the loop, because the loop's quality is bounded by the model's instruction-following, not by the harness.

Cost per report

Cost is where the loop shape shows up in the bill. A one-shot report is one call. An AutoResearch report is a planner call, several reader calls, a critique call, and a synthesizer call, repeated across iterations. The multiplier is the iteration count, and that is set by your stop conditions, not by the model.

Working from the pricing snapshot, here is what a single iteration costs at three model tiers, assuming a rough split of input and output tokens typical for an agent step. These are arithmetic on published prices, not measured bills.

At the top tier, o1-pro lists $150/M input and $600/M output. A single iteration that reads a long run history and writes a structured proposal is expensive enough that most people will not run a long loop on it. The batch variant at $75/M input and $300/M output halves the rate but does not fit a reactive loop.

At the mid tier, gpt-5.5-pro lists $30/M input and $180/M output, and claude-opus-4.7-fast lists $30/M input and $150/M output. This is the realistic band for a serious loop. The output rate dominates because agent steps write more than they read — proposals, critiques, and reports are all output-heavy.

At the lower tier, gpt-5-pro lists $15/M input and $120/M output, and o3-pro lists $20/M input and $80/M output. o3-pro is the cheapest output rate among the pro-tier models in the snapshot, which makes it the natural pick for a long loop where output volume is the cost driver.

The search API cost is separate and, in this repository, mostly absent — the harness reads local runs rather than a paid search index. That is a real advantage over web research agents, where search API calls often rival the model cost. If you are budgeting, the model is the whole bill here.

Three depth settings map to three budgets. A shallow run — one or two iterations, a cheap model — is a rounding error. A medium run with a mid-tier model across a handful of iterations is a real but manageable line item. A deep run with o1-pro across many iterations is the setting where people discover that autonomous research is not free. Set the iteration cap before you start, not after you see the bill.

Where it breaks

The failure modes are specific and worth naming.

Hallucinated citations. The synthesizer writes up results and can attribute a finding to a run that did not produce it, or cite a config that was never tested. The critique pass catches some of this; it does not catch all of it. Read the final report against the run logs before you trust a claim.

Loop divergence. Without a convergence check, the agent can keep proposing experiments that produce no measurable improvement. The loop looks productive — it is running, it is writing — while the loss curve stays flat. An iteration cap is the only reliable guard.

Topic drift on narrow questions. Give the harness a tightly scoped question and it may widen the search to fill the loop. The planner wants something to do, and a narrow question gives it little to work with, so it invents adjacent experiments. Narrow questions are better served by a one-shot tool.

Rate-limit stalls. A long loop makes many calls in sequence. Provider rate limits can stall it mid-run, and the harness's recovery behavior depends on how you wired the client. A router that fails over between providers is the practical fix, which is the main argument for the OpenRouter route mentioned earlier.

None of these are unique to AutoResearch. They are the standard failure modes of any long agent loop, and the repository's design addresses some of them more than others. The critique pass is the strongest mitigation; the stop conditions are the weakest link because they depend on the user.

Verdict and alternatives

AutoResearch is a good harness for a specific job: running ML research experiments autonomously on a single GPU. The repository shows 98,627 stars, which is a signal of interest rather than a signal of production readiness. The $0/mo price is honest — you pay in GPU time and model tokens, not in a subscription.

Best use cases: researchers who want to automate hyperparameter sweeps, teams exploring nanochat-scale training, and anyone who wants a reference implementation of a research agent loop to adapt. The code is the value; the harness is a starting point, not a finished product.

When a cheaper one-shot tool wins: any question that can be answered from the open web. Perplexity and OpenAI's deep research products are built for that, and on a web question they are faster, cheaper, and better cited. AutoResearch has no web search tool, so the comparison is not close.

If you are choosing a model to run inside the loop, the agentic AI coding tools review covers which models hold up on long multi-step tasks, which is the property that matters here. Beetlix is our own product; where it overlaps with AutoResearch is in orchestrating multi-step agent runs, and the honest comparison is that AutoResearch is purpose-built for ML experiments while Beetlix is general-purpose. If your work is nanochat training, the repository is the better fit.

Score: 7 out of 10. Strong for its niche, weak as a general research tool, and dependent on the user to set the guardrails that keep the loop from burning tokens.

How this review was researched

This review draws on the AutoResearch repository at github.com/karpathy/autoresearch, including its stated scope and star count; the vendor's pricing page, which lists $0/mo; and the live model pricing data used for the cost arithmetic. No hands-on testing was performed. Cost figures are arithmetic on published per-token prices, not measured bills, and the head-to-head section compares loop shapes rather than running a benchmark.