A frontier LLM named GPT-6 Astra was able to improve the MATH-500 accuracy of the Qwen3.5-0.8B model from 7.9% to 48.1% by editing only 0.016% of its weights. This feat was achieved through 'weight-space programming,' a method where an LLM reasons about what specific weights compute and modifies them directly without using traditional training loops or gradient descent.
The Problem of Thinking Loops
Small reasoning models often suffer from 'thinking loops,' where they repeat the same few lines of reasoning indefinitely. In the case of Qwen3.5-0.8B, 93% of its answers on the MATH-500 benchmark hit the 32,768-token limit without ever reaching a conclusion.
This figure depicts the 'thinking loops' problem where models repeat reasoning indefinitely.
Weight-space programming addresses this by identifying the specific weights responsible for these behaviors. By editing these weights, developers can remove obstacles to a model's existing abilities rather than trying to train new ones into a model that lacks the necessary capacity.
This figure illustrates the practical outcome of weight-space programming by showing a successful navigation task.
“Astra changed only 0.016% of the weights, and MATH-500 accuracy jumped from 7.9% to 48.1%.”
— Can a Linear Program Do Weight-Space Programming?
Mechanism of the Weight Edit
Transformers use residual connections where each layer's output is added to a 'residual stream.' When a model loops, this stream shifts in a consistent direction, which researchers call the 'loop direction' (u).
The edit involves modifying the weights of the MLP layers so that they push the residual stream in the opposite direction of the loop. Astra achieved this by applying a rank-1 change to layers 15 and 19, effectively flipping the direction of the model's output for looping tokens.
This figure explains the mechanism of modifying MLP weights to counteract the loop direction.
Astra's method involved a trial-and-error search, spending roughly 8 hours and 28 forward passes to determine which weights to change and by how much.
From Trial-and-Error to Linear Programming
The search for the optimal weight change can be framed as a linear program. By labeling tokens as 'looping' or 'productive' and recording their activations, the problem becomes finding a vector (v) that pushes looping tokens away from the loop direction while barely affecting productive ones.
Solving this as an LP with an L1 norm minimization naturally produces a sparse solution, selecting only the most critical weight columns to modify. While Astra's search took hours, a GPU-accelerated LP solver can find a comparable solution in about one minute.
Testing showed that the LP-derived edit matched Astra's performance on benchmarks like MATH-500 and GSM8K while modifying significantly fewer weights (16,291 vs. Astra's 130,188).
Chart drawn by HN Digest from figures reported by nathan.rs; values are shown as written in the source.
Limitations and Steering Vectors
The current LP method still relies on Astra to identify the specific layers and the loop direction vector. It does not yet autonomously select the optimal layers for the intervention.
Interestingly, the LP's push was found to be nearly constant across tokens, meaning the fix could also be implemented as a 'steering vector'—a fixed value added to the residual stream at every token. This provides a trade-off between a permanent weight edit and a runtime inference modification.