Distilling large to small, and proving it

Realistically, the only way open source AI companies will be able to build models to compete with frontier labs is via distillation.

Thinking Machines built Tinker, which made fine-tuning incredibly easy for researchers. We wanted to take their API a step further and specifically build for distillation. Distillery lets you programmatically describe your distillation process: which models to use, data formation (we provide a synthetic option as well), and actual hardware for training. We also let users define specific configs for new and experimental distillation techniques.

To prove the capabilities of this API, we ran an experiment where we attempted to distill a large Nova teacher into a much smaller student. With Distillery, we synthesized a dataset of 400 financial traces and trained the student on the teacher's outputs. We achieved a working large-to-small distillation loop end to end: teacher labeling with full provenance, two competing training runs (a standard recipe and a custom rejection-sampling recipe), and a sealed held-out evaluation where the teacher itself scores 57.5% schema validity and 25% decision-field accuracy, numbers the student has to be judged against honestly.

What Distillery is

Distillery is a distillation platform: it takes expensive repetitive calls to a large model and turns them into a cheap small model you can actually trust. The workflow is Curate, Synthesize, Train, Prove, exposed as an API with a three-line SDK happy path.

Trace synthesis covers the first two stages. Datasets are immutable, hashed resources with strict train, validation, and test splits. Labels come from a teacher model filling only missing training responses; the teacher structurally never sees test prompts. Every generated label carries provenance and must pass deterministic schema validation before acceptance. Any relabeling or curation produces a new immutable dataset, never a mutation.

Recipes are the distillation methods. Built-ins include sequence-level knowledge distillation for black-box API teachers and full-logit distillation for white-box pairs, which fails loudly with RECIPE_INCOMPATIBLE when its requirements are not met. There are no silent downgrades, and an auto resolver explains every choice it makes, including recommending do-not-distill. Users can also define their own recipes: a distillation method written as a small Python class against synthesis primitives, registered and run through the same resolver, capability gates, and API as the built-ins.

Proof is the point of the whole thing. Nothing ships on vibes. Every candidate student is evaluated on frozen held-out data the training process never touched: schema validity, decision-field accuracy, cost ratio versus the teacher, total experiment cost, and break-even request count. A candidate below target gets BLOCKED, which is a first-class outcome, not a failure. Only a passing model can be promoted behind a stable model alias, with automatic teacher fallback on invalid outputs and one-call rollback. Recipes, including user-defined ones, cannot touch or override this gate; that separation is what makes letting anyone invent a distillation method safe.

The experiment

The task is structured financial workflows: transaction review (GL account, balanced journal entry, policy action), variance analysis, and cash reconciliation, drawn from a deterministic synthetic finance world with an executable oracle that knows every ground-truth answer. The teacher is Amazon Nova Pro; the student is a 0.5B parameter model, roughly a thousand-fold parameter reduction, trained with QLoRA.

We labeled the training traces with the teacher in about two minutes of parallel calls for under a dollar. 113 labels passed schema validation. From the same labels we built two datasets: the baseline keeps all 113 valid labels, more data but noisier; the custom rejection-sampling recipe keeps only the 39 labels whose decision fields also match the oracle, three times less data with near-zero label noise. Both students train through the identical sealed pipeline and are scored on the same frozen 40-example test set as the teacher; the baseline student's sealed evaluation is complete, and the custom student is judged by the same harness. That is the comparison the proof gate exists to referee: clean-but-few versus noisy-but-many is a real methodological tradeoff, and Distillery measures the winner instead of asserting it.

Old model vs distilled model

This is the Distillery playground: pick a held-out finance task neither model trained on and run it through the student before and after distillation. Outputs are real captures from the evaluation harness; the base column is the student with the LoRA adapter disabled, the distilled column is the identical weights with the adapter enabled. Every response is validated against the executable oracle.