Before fine-tuning a small language model, check why it gets things wrong. In an MIT CSAIL experiment, editing 0.016% of the weights lifted the score on a maths benchmark from 7.9% to 48.1%: editing the neural network’s weights helped it loop less often and carry its solution through to an answer more often. That’s a reason to start with diagnosis, not a ready-made universal repair recipe.
For developers this is more interesting than the promise of “free intelligence”. If a model performs badly, it isn’t always down to a lack of knowledge. Sometimes a recurring fault gets in the way of the abilities it already has, and further training isn’t the only possible way to fix it.
Editing neural network weights: a model with Python instead of an optimiser
Researchers Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL tested an approach called Weight-Space Programming: a large language model reads and changes the numerical parameters of another network according to a task stated in plain language.
In ordinary fine-tuning, the system computes the error, calculates gradients and updates the weights through an optimiser. Here the editor was GPT-6 Astra. It analysed the tensors through Python, put forward hypotheses and wrote the changes directly. Gradient descent, backpropagation and fitting the weights to a loss function were forbidden.
This isn’t prompting: what changes is the network itself, not an instruction placed before the request. Nor is it ordinary fine-tuning: the direction of the change is chosen by the editor model, not by a training algorithm.
However, “without further training” doesn’t mean “without data and compute”. Astra was given training problems with answers, access to the network’s internal states and feedback from the runs it was allowed. It never saw the final test sets, but it wasn’t working blind either.
What exactly was broken in the small Qwen
The repair targeted the text component of Qwen3.5-0.8B: about 800 million parameters, 24 layers. It was run with reasoning mode switched on.
In practice the model often repeated the same fragments and used up its entire generation budget without moving on to a final answer. This is a known quirk: the official Qwen model card warns that this version has a heightened tendency to loop in thinking mode.
Astra linked the fault to a direction written by the MLP blocks of layers 15 and 19. The MLP is the computational part of a layer that transforms the internal representation of the text. In each of the two blocks the editor chose 64 columns of the output matrix and subtracted the component associated with repetition.
The edit touched 131,072 weight values. With roughly 800 million parameters, that’s the rounded 0.016%.
The working hypothesis here isn’t “we found the maths neuron” but something more modest: the editor weakened a repetition mechanism that was preventing an already trained model from finishing its work.
The numbers are impressive, but they apply to a particular mode
In the most successful run the researchers got:
- MATH-500: accuracy rose from 7.9% to 48.1%.
- GSM8K: from 13.8% to 67.1%.
- IFEval, which tests instruction following: from 21.0% to 62.3%.
- Hitting the token limit: the share of MATH-500 generations that hit the token limit fell from 93.0% to 33.1%.
The 93% figure refers specifically to MATH-500 in this experiment, not to all requests to Qwen. Even after the repair, about a third of generations on this set still reached the limit. Hitting the limit doesn’t in itself prove looping: the model may simply be taking a long time to solve the problem. The explanation via repetition also rests on analysis of the answers themselves and on control interventions.
The same generation settings and a maximum of 32,768 new tokens were used before and after the edit. For MATH-500 the results of four generations per problem were averaged. This is mean accuracy, not the probability of finding at least one correct answer in four attempts.
We checked the main figures against the published evaluation files. We didn’t rerun the model ourselves: checking the reporting is not the same as independently reproducing the experiment.
The fact that the improvement carried over to IFEval supports the explanation that a general generation fault was removed. But these numbers don’t mean the small Qwen became universally stronger at every task or beat the original model under any settings.
“50 runs” isn’t fifty questions or an instant repair
In different runs the editor was allowed 10, 25 or 50 calls to the forward-pass tool. A single such call could include generating answers to hundreds of training problems or reading internal states while the model processed text.
So the popular “just 50 tests” distorts the scale. These are up to 50 batch calls to the model, not 50 individual examples. Separately, the editor could run Python code to analyse the weights.
The successful variant took 158 turns of interaction and 80 Python runs. The log records 20.3 hours from start to finish, including 4.7 hours of waiting because the editor model was unavailable. That’s the time of one specific research run, not a yardstick for a future tool.
The small size of the final edit says nothing by itself about the cost of finding it. Changing a few lines in a complex program can sometimes take a whole day of debugging too.
Variants with smaller budgets gave considerably more modest results: the MATH-500 score was 12.3% with 10 calls and 14.1% with 25. These are different search runs, not a guaranteed ladder of improvements to the same edit.
What to check in your own agent before new training
Our practical conclusion: first find out at which stage the result gets lost. An agent may know the solution but not finish generating. It may finish the answer but break the format. Or it may genuinely lack the skill required. The same “failed” mark hides different faults.
For an applied project it’s worth starting with what’s available:
- Count wrong answers, truncated generations, repetitions and format violations separately.
- Check the chat template and settings against the model developer’s recommendations.
- Compare reasoning modes and limits on one fixed set of tasks.
- Check not only the target metric but also ordinary requests, cost and latency.
This order is useful even without access to the weights. Why quality depends on the whole system and not just the model’s name is covered in more detail in our write-up of the CMU course on agent harnesses and evaluation. And training through language self-play (in Russian) shows a different path to improvement, which shouldn’t be confused with targeted repair of parameters.
On Gemma the repair achieved almost nothing, and that’s an important result
The researchers tried to improve Gemma 3 270M, where looping was much less pronounced. On MATH-500 the baseline score was 6.8%; after the variant with 50 calls it was 7.0%. No convincing improvement emerged.
The errors there looked different: wrong arithmetic, incorrect algebraic steps. No single local mechanism could be found whose removal would eliminate the main problem.
You can’t directly compare the absolute abilities of Gemma and Qwen on these numbers: the model sizes, settings, generation limits and answer-extraction rules used in evaluation all differed. But the negative result does limit the main claim. Removing an obstacle and acquiring a new skill are different tasks. The Qwen results are consistent above all with removing an obstacle; a universal way to make any model smarter doesn’t follow from them.
In a separate experiment Astra couldn’t reliably tell, from the weights alone, a model prone to sycophancy (agreeing with the user) from an ordinary one. It only succeeded once it was given a tool that showed the distribution of attention. Access to the numbers alone doesn’t turn a network into an open book.
The novelty isn’t the idea of changing individual weights
Targeted model editing existed before Weight-Space Programming. For example, the ROME method, presented at NeurIPS 2022, locates factual associations and changes them with a small edit to an MLP matrix.
What sets the new approach apart is the role of a general-purpose LLM. It doesn’t just explain a neuron that has been found or run a pre-written editing algorithm; it takes part in the diagnosis and in designing the edit itself for a task stated in words.
That makes the work interesting for mechanistic interpretability: research into which internal computations give rise to a network’s behaviour. An explanation becomes more convincing when it’s followed by a targeted intervention and a measurable result. In the successful run the researchers tested individual layers and the strength of the intervention, and used a random edit of comparable size as a control.
But a small edit doesn’t guarantee there are no side effects. A UC Berkeley study of applying ROME and MEMIT sequentially showed a gradual and then abrupt deterioration in stored knowledge and other abilities. That doesn’t prove Weight-Space Programming behaves the same way; it’s a reason not to replace regression testing with a neat percentage of changed parameters.
The final patch is reproducible; the search process only partly
If you do experiment with weights, you need a separate copy of the model, a reproducible edit and the ability to roll back. The researchers published the logs and the exact weight changes, along with a tool for applying them. However, the code that drove the whole Qwen experiment isn’t included in the published package: restoring the final patch and repeating the entire process of finding it are different levels of reproducibility.
For now it’s more sensible to see Weight-Space Programming as a research debugging method rather than a replacement for training. The first step doesn’t require access to the weights: take a fixed set of your own tasks and separately count wrong answers, repetitions and truncated generations. That will show what exactly you are about to improve.
