Which knowledge-distillation techniques are worth attempting on an open-weight, locally-served Indonesian Office assistant — read against what the programme has actually measured, and against what the literature claims. Every internal figure cited here is traceable to a file in tantular-distillation; every external claim to a linked source.
Qwen/Qwen3.5-9B meets or exceeds every threshold the programme pre-registered for its own fine-tune (calibration/WHY_TRAINING_IS_NOT_JUSTIFIED.md, 2026-08-21). A training run against those numbers can only hold or regress them. This is a finding about the eval suite as much as about the model: three model-dependent gates over 60 items and one 10-item pilot is a narrow instrument, and every corpus behind it is synthetic.calibration/INT4_WAIVER.md). Two rentals produced no treatment arm before the FP8 track was frozen. Any v2 claim inherits that hole.Three evaluations are wired, all run over the same endpoint and protocol (plain-chat, enable_thinking: false), plus a faithful-editing pilot. The thresholds were registered in train/qlora_9b.yaml before the baseline was taken. The base model passes all four.
Distillation trains a smaller student to reproduce a larger teacher’s behaviour rather than to fit the original labels. Kim & Rush (2016) established the sequence-level form now standard for generative models: run the teacher over the training inputs, keep its outputs as targets, and train the student on those. In their neural-translation setting the technique both compressed the model and, they report, largely removed the need for beam search at decode time.
Two properties of that formulation matter for this programme. First, it is off-policy: the student is trained on text the teacher produced, in contexts the teacher visits. Lu (2025) frames the resulting drawback directly — the student learns in contexts frequented by the teacher rather than the ones it will itself encounter. Second, it inherits the teacher’s errors as targets, with no mechanism to notice them.
The corrections that followed are all attempts to close those two gaps. Gu et al. (MiniLLM) replace the forward KL objective with reverse KL, on the argument that in generation the teacher’s distribution has far more modes than a small student can represent, and forward KL therefore forces the student to put mass on regions the teacher barely occupies; reverse KL is mode-seeking instead. They optimise it with a policy-gradient method and report that stabilisation tricks — teacher-mixed sampling, length normalisation — are necessary, without which the student degenerates toward short high-probability strings. Agarwal et al. (GKD) take the complementary route of training on the student’s own generations and having the teacher score them.
Recent work extends on-policy distillation to the case where teacher internals are unavailable: Ye et al. (2025) train a discriminator against teacher text as an adaptive on-policy reward and report surpassing sequence-level KD, with a 14B student reaching comparability with a much stronger proprietary teacher on one automatic chat evaluation. We flag that as a single-benchmark result on models unlike ours, not as a transfer claim.
Three separate arguments converge, and it is worth keeping them apart because they would be reopened by different evidence.
Set against that, the honest counterweight recorded in the repository is that the work was not wasted: the training path executes end to end, expanded LoRA targets attach at exactly the projected 80,216,064 parameters, a converted adapter genuinely binds in vLLM and changes output, and the gates now fail closed on a missing endpoint, a wrong model, an inert adapter, a hanging suite, and reasoning scored as an answer. The cost of establishing that the base is already sufficient was, by the programme’s own accounting, far less than a fine-tune that improved nothing and would have been hard to disprove.
The programme’s choice of open weights is not primarily an ideological one; it removes three specific dependencies at once. There is no hosted API in the path, so generation is unblocked by default and costs nothing per trace beyond electricity or rental. The teacher can be moved between an on-premises box, a rented card and a laptop by changing one argument, because the client only ever sees a URL. And the data-handling regime can be enforced mechanically: hosts declare where data goes, prompts declare what they carry, and an unclassified prompt defaults to internal so that forgetting to label real material fails closed rather than shipping it to a rented GPU.
The licence question is narrower than “is it open”. It is whether the teacher’s terms permit a commercially-shipped model trained on its outputs. Apache 2.0 settles that for the dense teacher, and its config file records the operational reason: the model card places no restriction on training other models from its outputs, with a note to re-check before a commercial run because licences get amended.
The second teacher is the open legal question, and the reason is worth stating precisely, because “NVIDIA licences are permissive” is true and insufficient. The current Nemotron Open Model License (last modified 15 December 2025) opens by confirming the works are commercially usable, that derivative works may be created and distributed, and that NVIDIA claims no ownership of outputs. But an earlier NVIDIA family of terms — the Community Model License and the AI Foundation Models Community License — defines “Derivative Models” to expressly include models created by transferring patterns of weights, activations or output, naming distillation and synthetic-data generation as covered methods. Which document governs a given checkpoint determines whether a distilled student is itself a derivative work bound by the teacher’s terms.
One further consequence of the open-weight choice is a hard constraint the roadmap in §07 has to respect: the only teacher path available without renting is an int4 Ollama endpoint behind a gateway, which is operator-visible and therefore restricted to synthetic prompts.
The repository states the principle in its hardware rules: teacher quality is the ceiling on everything downstream, and it is the one loss that cannot be recovered later. The scaling literature agrees in the aggregate — lower teacher cross-entropy generally yields lower student cross-entropy, up to the capacity gap (Busbridge et al., 2025). The practical position is therefore uncomfortable: the entire existing corpus was generated at int4.
The waiver that authorises this is unusually careful about what it does and does not buy, and the distinction is load-bearing for any published claim. It permits the statement that the student imitates the teacher behaviour users receive today — validated in that the normalized protocol reproduces the deployed endpoint 52/52 per prompt — and that generation stayed on-premises with no third-party operator. It forbids any wording implying FP8 equivalence, or that the gate passed. The correct description, in its own words, is training from an int4 teacher under a recorded waiver with the FP8 comparison not performed.
The v3 stability result is the strongest measurement in the programme and also the one most at risk of being over-read, which the file itself pre-empts: three samples would very likely miss a mode appearing five percent of the time, so the defensible statement is that no variation was observed in three observations at these settings — not that the teacher is deterministic. The reason the result is credible at all is that it was checked rather than assumed: 220 of 260 families were generated in different batch windows with different co-resident prompts across the two earliest passes and still produced byte-identical completions.
The FP8 arm that would have bounded the quantization loss is frozen. Two rentals produced no treatment arm at roughly $3.35 of compute; the infrastructure was solved (driver gate, clean-venv stack check, two vLLM protocol fixes) and the blocker turned out to be model support — no native vLLM implementation for the teacher’s multimodal causal-LM class, which generated only control tokens.
The clearest methodological lesson in the programme is about a metric, not a model. The inventory reported 260 of 260 families ready, zero blocked. That was true and uninformative: every family resolved to a source artifact, but the artifacts numbered 78, shared at a mean of 3.33 families per document and a maximum of eight. A training set built on it would have inherited each document roughly three times with an identical prompt.
Two things were right about that corpus even so, and are worth separating from the defect: no document spanned more than one split, so a train document never backed an eval family; and the unique-prompt view (n=78) was already the primary denominator in the cross-pass reporting, precisely because the family-weighted view was inflated. Source pack v3 resolved it by authoring per-family artifacts — 260 distinct documents for 260 families, zero digest reuse, zero documents crossing splits.
What no amount of that fixes is the source class. All 26 strata are synthetic. The inventory’s own note is the operative caveat: synthetic material supports pipeline generation and behavioural training, and supports no claim about performance on real Office documents. Per-family authoring buys corpus breadth and removes the repetition inflation; approved real sources are what any statement about real documents requires, and they are gated on approval, redaction and a per-host egress decision rather than on code.
Eight candidate techniques, assessed against this pipeline as it stands rather than in the abstract. “Available” means it can be run on the hosts and serving paths the repository already describes. Verdicts are ours; the constraints behind them are cited.
The pattern across the table is that the methods with the strongest published gains are the ones this pipeline is least shaped for, and the reason is architectural rather than budgetary: the programme’s central asset is a frozen, fully-provenanced corpus, and on-policy methods dissolve exactly that artifact. That is a real tension, not a defect to route around. A distillation programme whose auditability is a product feature should expect to pay for on-policy training in audit design, and should decide that consciously.
The training config carries the programme’s own observation as a comment on the gate block: distillation reliably buys reasoning while quietly costing Indonesian voice and JSON-contract adherence, and the Studio features break outright when the edit contract drifts. That is why the README instructs importing reasoning and agentic behaviour from the teachers while explicitly refusing to let either teach Indonesian — that capability is Tantular’s own and distilling over it loses ground.
The external literature supports the mitigation the pipeline has already chosen, for a reason worth being precise about. Biderman et al. (2024) compare LoRA against full fine-tuning on code and mathematics and find LoRA generally learns less in-domain — substantially so at low rank — but better preserves performance outside the target domain, with the extent of forgetting controlled by rank; they also report it maintains more diverse generations than full fine-tuning. For a model whose out-of-domain behaviour is the differentiator, learning less and forgetting less is the right side of that trade.
Two cautions follow. Their result also shows forgetting worsening with training duration and instruction fine-tuning inducing more of it than continued pre-training, which argues for short runs and for measuring voice after every epoch rather than only at the end. And rank is the dial: at r=256 they found forgetting on the mathematics task approaching full fine-tuning. The programme’s r=32 is conservative by that standard, which is consistent with its stated priority.
office_json_contract suite runs the add-in’s own JavaScript tests with fetch mocked. It never calls a model, so it returns the same result before and after training and cannot detect an adapter regression — measured 2026-08-19, and documented in place rather than quietly repointed. The model-dependent job belongs to the 20-item edit-contract gate, which pushes generated answers through the real parser and scores contract_ok: parsed, located and applied. Reporting parse_ok alone would be too weak, since valid JSON whose target string appears nowhere in the document parses cleanly and is useless.We read that episode as the most transferable methodological result in the repository, and it is not about distillation at all: a green gate that structurally cannot fail is worse than no gate, because it is spent as evidence. The same programme also recorded a before baseline of exactly 0.0000 on both model-dependent gates, produced by scoring the model’s English reasoning as its answer — and correctly refused it, on the grounds that a baseline of zero makes any output an improvement and satisfies “no regression” trivially.
Precision enters this pipeline four times, and conflating them is how a quantization claim goes wrong. It is worth listing them separately because only one of the four is currently waived.
There is a coherent position available here that is stronger than the apologetic one, and the waiver states it: if the product target is current int4 Ollama behaviour, then imitating the teacher users actually receive is a defensible objective, and privacy plus zero rental cost may reasonably outweigh maximising teacher quality. Tantular’s Indonesian-first, private, on-device positioning makes that a choice rather than a concession. What it is not, and the waiver says so, is evidence that FP8 would not have produced a better student.
The deployment budget adopted on 2026-08-22 belongs in this section for a reason that is methodological rather than technical. It was recorded before the gate was re-run under it, on the stated principle that a budget changed after seeing a result it then permits is not a policy but a rationalisation. The previous per-request ceiling was 30 s and the pulled model measured a slowest request of exactly 30.0 s — a pass only because the comparison is strictly greater-than, on an unloaded machine with the model already resident. The ceiling moved to 45 s while p95 stayed at 30 s, with three conditions kept absolute at zero: reasoning characters, empty answers, truncated answers. Any distillation that trades latency for quality is spending against that budget, and a 45 s worst case carries a product obligation — visible progress and cancellation — without which it is indistinguishable from a hang.
The sequence in the README trains the 4B last, from the improved 9B rather than from the 30B teachers directly, on the stated grounds that it then learns from something that already shares its target behaviour and voice. The literature gives that intuition a name and, partially, a formalism. Busbridge et al. (2025) report a capacity-gap regime in which a teacher too strong relative to the student produces a worse student, with the student’s loss following a power law in teacher cross-entropy that changes character between regimes. Earlier work goes further and proposes a law of capacity gap — that for an intended student scale there is an optimal teacher scale, determinable without exhaustive search.
We treat both as directional rather than prescriptive here. Those fits were established on families and objectives unlike a hybrid-attention 9B distilling into a 4B under LoRA over a 4-bit base, and the second is a proposition tested at pilot scale. What survives the transfer is the ordering claim, which is also the cheaper one: a 9B teacher for a 4B student is a smaller ratio than a 30B teacher for the same student, it is already voice-aligned, it is locally servable, and it raises no rental, egress or licence question. If any single distillation run in this programme is worth doing on efficiency grounds, it is this one.
The dependency is the awkward part and should be said plainly: v3 presupposes v2, and v2 is the step currently judged unjustified. If v2 never happens, the 4B either distils from the base 9B — which buys compression, not capability — or from a 30B at a ratio the capacity-gap literature warns about.
Distillation changes weights. There is a second family of methods that changes the decision logic around the weights, and it is worth taking seriously here precisely because the programme has concluded the weights are not the bottleneck.
Yin et al. (2024) describe Gödel Agent, a self-referential framework in which an LLM-driven agent reads, evaluates and rewrites its own logic at runtime rather than following a fixed pipeline or meta-learning routine — implemented by manipulating runtime memory through monkey-patching. They report continuous self-improvement across reasoning and agentic tasks, and are unusually direct about the framing: the experiments demonstrate feasibility, they do not compare against heavily engineered agent systems, and they note that a system growing more complex through self-optimisation may need exponentially more intelligence to understand itself — so full self-reference at the outset can be lost as it evolves.
The constrained prototype that provides this report’s design language is the sober version of that idea, and its constraint is the interesting part. The loop is the same four conceptual steps — self_inspect, interact, self_update, continue_improve — but the thing the agent may rewrite is exactly one validated function, solve(query, kb). Candidates are parsed to an AST and rejected before execution if they contain imports, class/lambda/global/nonlocal definitions, dunder attribute access, banned builtins, or anything other than that single top-level function; what survives runs against a small safe-builtins namespace with no file I/O, network or subprocess. A candidate that errors, returns the wrong type, or lowers the score is rolled back.
Three properties make this the right thing to run before a distillation run rather than after. It needs no GPU, no corpus freeze and no adapter, so the failure cost is minutes. Its rollback-on- regression rule is the same no-regression policy the training gates already implement, which means the two share an evaluation harness. And it optimises the layer the programme has already identified as the differentiator — prompts, recipes, routing — rather than the weights it has concluded are adequate.
The limits are equally clear and the prototype’s own documentation states them. Its scoring is deliberately simple keyword matching with a politeness/length bonus and a forbidden-term penalty; for serious work it recommends exact checks where possible, retrieval-grounded factuality, safety classifiers, human labels and held-out private tests. Its holdout environment is the honest configuration: the split is deterministic via a stable hash rather than Python’s salted one, category-balanced where possible, and holdout queries and missing terms never reach the feedback text, so the optimiser sees public-case signal only — with the aggregate holdout score treated as an audit signal to inspect after a run, not an optimisation target. And the sandbox is described as an experimentation boundary, not a hardened one: production use is said to require separate processes or containers, signed snapshots, a private holdout, and human review before any learned policy is deployed.
One clarification underpins everything above. Today’s shipped tantular-office:0.4-9b is a runtime profile, not a fine-tune — a base model plus a system prompt, with no trained weights. The only genuinely trained Tantular is a 3B Indonesian-support LoRA on a different base family and domain, from 153 examples. There is therefore no Office checkpoint to continue from, and teacher distillation is structurally a v2 activity: it needs a v1 to distil onto.
Two items in the remaining sequence are worth separating from the model question because they gate production regardless of any training decision: the gateway key must be rotated before production use, and no claim about real Office documents is supported until approved real source material exists. Both are people-and-process work, not GPU work.