Distilly Review 2026: Model Distillation Without GPUs

What Distilly is

Distilly describes itself as a way to distill how an expert thinks into reusable skills that any agent or bot can load. That sentence is doing a lot of work, so it is worth unpacking before anything else, because the name invites a wrong assumption.

Most people hear "distillation" and picture the classical machine learning version: a large teacher model, a smaller student model, logits passed between them, a GPU cluster humming for a week, and a student that mostly holds up on a narrow eval. That is model distillation. Distilly is not that, at least not in the sense the repository and documentation describe. The repository at github.com/titanwings/distilly shows 25,663 stars and frames the project around extracting reusable skills from expert behavior rather than compressing weights from a teacher network into a student network.

So the honest answer to "data distillation, model distillation, or both" is: neither in the textbook sense, and something adjacent in practice. Distilly sits closer to a skill-extraction and packaging pipeline. You feed it examples of how an expert works, it produces a reusable artifact that an agent or bot can load, and that artifact encodes the expert's approach in a form a smaller or cheaper model can follow. The output is not a set of weights you would deploy to a serving endpoint in the usual sense. It is a skill object.

This matters for the review because the entire premise of the brief — get a small model to hold accuracy on a real eval set, measure the cost per run — assumes a weight-level distillation pipeline. Distilly does not present itself that way, and I would be inventing a product if I reviewed it as one. What I can do honestly is describe what the documentation and repository claim, what the pricing page lists, and where the gaps are. I have not installed, run, or purchased Distilly. Everything below is read from the docs, the repository, and the pricing page.

On versioning: the repository does not publish a clean semantic version string in the material I could read, so I am not going to name a version number I cannot verify. Treat this as a review of the project as it stands in 2026, not of a tagged release.

Pricing 2026 and free tier limits

The pricing page lists a starting tier at $0/mo. That is the number in the tool record and it is the only dollar figure I can state with confidence for Distilly itself. Everything past that is described by tier name rather than by number, because the numbers are not in front of me and I am not going to guess them.

What the pricing material does make clear is the shape of the model: credits, seats, and self-host are three separate axes, not three tiers of the same thing.

• Credits are the consumption unit. You spend them as you run distillation jobs. This is the axis that scales with how much you actually use the product, and it is the one that matters if your workload is bursty.
• Seats are the collaboration unit. If more than one person on a team needs to author, review, or load skills, seats are what you pay for. A single developer doing solo work can plausibly stay on the free tier longer than a team can.
• Self-host is the deployment unit. This is the option that matters if your data cannot leave your infrastructure, and it is the one that changes the cost calculus entirely, because you are paying for your own compute rather than for Distilly's.

The question the brief asks — compute billed by token or by hour — is the right question and the documentation does not answer it in a way I can quote. Credits are the unit, and how credits map to tokens or hours is not something I can state without inventing it. If you are budgeting, that mapping is the first thing to pin down with the vendor, because it determines whether a large distillation run is cheap or expensive.

For context on what the surrounding market looks like, the live pricing snapshot for frontier models runs from $15/M input and $60/M output at the low end (openai/o1) up to $150/M input and $600/M output at the top (openai/o1-pro). Batch pricing roughly halves those figures — openai/o1-pro:batch lists $75/M in and $300/M out. If Distilly's credits are denominated against calls to models in that range, the per-run cost of a distillation job is dominated by how many teacher calls the pipeline makes, not by Distilly's own fee. That is a structural point worth internalizing: the free tier is free, but the teacher model behind it is not.

The free tier's hard limits — request caps, credit grants, seat counts — are not published in the material I can read, so I am not going to state them. If you need to know whether the free tier covers a real project or just a demo, that is a question for the vendor.

Workflow: dataset in, small model out

The documentation describes a pipeline that starts with examples and ends with a loadable skill. Walking it end to end is the clearest way to explain what Distilly actually does.

Input

The repository and docs describe accepting structured example data. The formats that come up are the usual suspects for this kind of tooling: JSONL, Hugging Face datasets, and CSV, with API-based ingestion for pulling examples from a live system. If your expert behavior lives in a chat log, a ticketing system, or a set of recorded agent traces, the API path is the one you would use. If you already have a curated dataset, the file path is faster.

The important thing about the input stage is that the quality of the output is bounded by the quality of the examples. This is true of every distillation approach and it is worth saying plainly: if your expert examples are inconsistent, the extracted skill will encode the inconsistency. Distilly does not appear to have a magic step that fixes contradictory demonstrations.

Processing

This is the part the docs describe least concretely, and it is the part that determines whether the tool is doing something interesting or just wrapping a prompt. The description — distilling how an expert thinks into reusable skills — implies the pipeline is extracting patterns from the examples and encoding them in a structured form, not merely concatenating the examples into a long prompt. Whether that extraction is done by an LLM pass, by a rule-based analyzer, or by some combination is not something the documentation spells out in the material I read.

What I can say is that the output is described as a skill, which suggests a structured artifact rather than raw text. That distinction matters for the next stage.

Output

The output artifact is the crux of the review, and it is where Distilly diverges most sharply from a classical distillation pipeline. The brief asks whether the output is weights, a LoRA, or an endpoint. Based on the repository description, it is none of those. It is a skill that an agent or bot loads. That means the "student model" in the classical sense does not exist — there is no new set of weights, no LoRA adapter, no dedicated inference endpoint. There is a skill object that a general-purpose model consumes at runtime.

This is a real design choice with real consequences. The upside is that you do not need GPUs, you do not need a training run, and you do not need to manage model versions. The downside is that you are still paying for a general-purpose model at inference time, and the skill's effectiveness depends on how well that model follows the encoded instructions. A skill is not a compressed model. It is a structured prompt with a pipeline behind it, and the economics reflect that.

If you want a sense of what the alternative costs, the cheapest API tier in the live snapshot is openai/o1 at $15/M in and $60/M out, and the batch variant of the top model is $75/M in and $300/M out. A skill-based approach keeps you on those per-token rates forever. A weight-level distillation would, in principle, let you run a smaller model at lower cost — but Distilly is not selling that.

Quality: student vs teacher on the same eval set

This is the section where a classical distillation review would report accuracy deltas on a fixed eval, name where the student breaks, and publish the scoring script. I cannot do that here, and the reason is not laziness — it is that the artifact Distilly produces is not a student model, so there is no student to score against a teacher on a held-out set.

What I can describe is the evaluation question that actually applies: does a skill extracted from expert examples cause a general-purpose model to behave more like the expert? That is a measurable thing, and it is the measurement that matters for this product. The repository does not publish a benchmark, and I am not going to invent one. But the shape of the eval is clear enough to describe.

You would take a set of held-out tasks in the domain the expert works in. You would run a base model on them with no skill loaded, record the outputs. You would run the same model with the Distilly skill loaded, record the outputs. You would score both against the expert's own outputs or against a rubric derived from them. The delta is the value of the skill.

Where this tends to break, based on how skill-based approaches generally behave, is at the edges of the domain. A skill extracted from examples of a specific workflow will handle that workflow well and will not generalize to adjacent workflows the expert never demonstrated. That is not a flaw unique to Distilly — it is the nature of example-driven extraction. The failure mode is a model that confidently applies the skill to a task the skill was never meant to cover.

The other failure mode is overfitting to surface form. If the expert examples all use a particular phrasing, the extracted skill may encode the phrasing rather than the reasoning, and the model will produce outputs that look right and are wrong. This is the thing to watch for in any eval you run, and it is the reason a rubric-based score beats a similarity score.

I would not trust a vendor-published accuracy number for this class of tool without the eval set and the scoring script alongside it. Distilly does not publish either in the material I read, which is a gap, not a scandal.

Cost and time per run

The brief asks for wall-clock and dollars for a small (1-3B) and mid (7-8B) student. Distilly does not produce students in that sense, so those numbers do not exist for this product. What I can do is describe the cost structure honestly and point at the variables that determine it.

The cost of a Distilly run has two components. The first is Distilly's own credit consumption, which the pricing page denominates in credits without a published token or hour mapping. The second is the cost of whatever model calls the pipeline makes — teacher calls to extract the skill, and inference calls to use it.

The second component is the one that will surprise people. If the extraction step uses a frontier model, and the live snapshot is any guide, you are looking at somewhere between $15/M and $150/M input depending on which model, with output between $60/M and $600/M. A distillation job over a few thousand examples is not a rounding error at those rates. Batch pricing helps — openai/o1-pro:batch is $75/M in and $300/M out, roughly half the standard rate — but it does not make the problem disappear.

Wall-clock time is dominated by the same factor. A pipeline that makes one model call per example will take roughly as long as N sequential calls, minus whatever parallelism the implementation supports. The repository does not document its concurrency model in the material I read, so I cannot say whether a large job runs in minutes or hours. That is a question to ask before committing to a big run.

For a rough sense of scale: a job over 1,000 examples, if it makes one teacher call per example at an average of 2,000 input tokens and 500 output tokens, is 2M input tokens and 500K output tokens. At openai/o1 rates that is $30 in and $30 out, so roughly $60. At openai/o1-pro rates it is $300 in and $300 out, so roughly $600. Same job, tenfold cost difference, entirely determined by which teacher model you point it at. That is the single most important cost lever, and it is under your control.

If you are comparing this to running your own distillation on rented GPUs, the comparison is not apples to apples, because you are buying a different thing. A GPU-based distillation produces weights you own and can serve cheaply forever. A skill produces an artifact you load into a model you rent by the token. The break-even depends on how many inferences you expect to serve, and for low-volume use cases the skill approach wins on simplicity alone.

Limits: where distilled models still fail

The brief lists long context, tool calling, rare languages, and reasoning chains as the places distilled models fail. Those are the right categories for weight-level distillation, and they are worth translating to the skill-based approach Distilly takes, because the failure modes are different.

Long context

A skill is loaded into the context window alongside the task. That means every token the skill consumes is a token unavailable for the actual input. A large, detailed skill eats context. If your task already needs a long context — a big document, a long conversation history — the skill competes with it. This is a structural cost of the approach, not a bug, and it is the reason skill authors should keep skills tight.

Tool calling

Whether a skill improves tool-calling behavior depends entirely on whether the expert examples demonstrate tool use. If they do, the skill can encode when and how to call tools. If they do not, the skill will not conjure that ability. The model's underlying tool-calling competence is unchanged; the skill only shapes when it is exercised.

Rare languages

Same logic. A skill extracted from English examples will encode English reasoning. If the underlying model is weak in a target language, the skill does not fix that. It may even hurt, by pushing the model toward English patterns in a context where they do not fit.

Reasoning chains

This is the most interesting case. A skill can encode a reasoning procedure — a sequence of steps the expert follows — and that can genuinely improve a model's output on structured problems. But it cannot give the model reasoning capacity it does not have. If the base model cannot follow a multi-step chain reliably, a skill that describes the chain will not make it reliable. It will make it more likely to attempt the chain, which is not the same thing.

The general principle: a skill shapes behavior, it does not add capability. Every limit above follows from that. If you need capability the base model lacks, you need a different base model, not a better skill.

Data handling: your dataset, their servers

This is the section where I have the least to report and where the gap matters most.

The documentation does not state a retention policy in the material I read. It does not state whether your uploaded examples are used to train anything on the vendor side. It does not state a deletion-on-request process. These are three separate questions and none of them has a published answer I can quote.

What I can say is what the architecture implies. If you use the hosted product, your examples go to Distilly's servers and, presumably, to whatever model provider the pipeline calls. That is two parties with access to your data, not one. If your examples contain anything sensitive — customer data, proprietary process, internal code — that is a real consideration.

The self-host option is the answer to this, and it is the reason self-host exists as a separate axis on the pricing page. If your data cannot leave your infrastructure, self-host is the only configuration that satisfies that requirement, and it is worth confirming that the self-hosted version makes no outbound calls to the vendor before you rely on it.

I would flag the absence of a published retention policy as the single biggest gap in the product's public documentation. It is not evidence of bad practice. It is evidence that the practice is not documented, which for a tool that ingests your proprietary examples is a thing you should resolve before you upload anything you would not want retained.

Verdict

Distilly is a skill-extraction tool wearing a distillation name, and the name is the source of most of the confusion around it. If you came here looking for a way to compress a large model into a small one without GPUs, this is not that, and no amount of reading the docs will make it that. If you came here looking for a way to capture how a specific expert works and make a general-purpose model behave more like them, that is exactly what the repository describes, and the 25,663 stars suggest a lot of people found the idea worth a look.

The economics are the thing to think hardest about. A skill keeps you on per-token inference forever, and depending on which teacher model the pipeline uses, that ranges from $15/M in and $60/M out at the low end to $150/M in and $600/M out at the top, with batch variants roughly halving it. That is fine for low-volume, high-value work and painful for high-volume serving. If your use case is the latter, a weight-level distillation you own will beat this on cost eventually — but you will pay for it in GPUs and engineering time, which is the trade Distilly is explicitly avoiding.

Beetlix is our own product, and where it overlaps with Distilly it is on the routing and cost side rather than the skill-extraction side, so the comparison is thin and I will not stretch it. The more useful comparison for most readers is against the broader API market, which is covered in our cheapest AI API breakdown and our OpenRouter review. If you are weighing teacher models for the extraction step, the LLM benchmark comparison is the place to start.

Score: 6.5/10. The idea is sound and the adoption is real. The documentation gaps — no published retention policy, no published credit-to-token mapping, no published eval — are what keep it out of the top tier. None of them is fatal. All of them are fixable.

Use this if you have a domain expert whose approach you want to capture, your volume is low enough that per-token inference is acceptable, and you would rather not run a training pipeline.

Skip if you need actual weight-level distillation, you serve high volume and need to own your inference cost, or your data cannot leave your infrastructure and self-host is not an option for you.

How this review was researched

This review is based on the Distilly repository at github.com/titanwings/distilly, the vendor's documentation and pricing page, and the live model pricing data current in 2026. No installation, run, or purchase was performed. Where the documentation does not state something — retention policy, credit-to-token mapping, evaluation results — this review says so rather than filling the gap.