@adlrocha - Forget “Fat” Models. The Future is Narrow
How to REAP your experts to fight the upcoming open-weight crunch
A quick confession before we start. I told you across the last two posts that I was working towards building my own local inference rig, and I’ve decided to hold off on the hardware for now. After my analysis, it was clear that the market is moving too fast to freeze a build, we may be at an all-time-high of hardware prices, and I’d rather keep my money in my pocket a bit longer. But I haven’t dropped the thread (I can be pretty obsessive). I’m still deep in the design space, still trying to work out what the right box actually looks like before I spend a penny on it.
And then this week a piece of news dragged me right back in. China’s Ministry of Commerce has reportedly spent the past month in meetings with Alibaba, ByteDance and Z.ai about restricting overseas access to their best AI models. And not only the closed ones. The report says it covers open-weight models too, the freely downloadable kind, with Qwen and GLM-5.2 named directly. The idea being floated is a tiered system: basic models wave through, high-performance ones get a security review, and the most sensitive frontier ones either stay inside China or don’t get released at all. Nothing is law yet, so take it with the appropriate salt, but the direction of travel is hard to miss.
A big chunk of the open-weight models the local community runs are Chinese. DeepSeek, Qwen, GLM, they’re the reason a person can run something close to the frontier at home at all. Even more, they are the reason we can get lots of serverless LLM providers offering cheap and performant tokens able to compete with the big labs demand. A ban or freeze on the flow of open-weights models from China into the West could be catastrophic for our access to intelligence.
Which brings to the original inspiration of this post. I’ve been following for a while how the local inference community is trying to compress models so they fit in the kind of hardware that you can afford at home. Some people have been able to run DeepSeek on a single DGX, and I’ve already talked about in this newsletter about how Antirez managed to get DeepSeek to run on an M5 with DS4. These are all models that in theory would only fit in expensive compute infrastructures, but can be adapted to replace many use cases at home.
To no one’s surprise, this is part 3 of the topic that has been obsessing me for a couple of months, i.e. how to run LLMs at home. Part 1 was the hardware, and why memory bandwidth is the number that actually decides what you can run. Part 2 was the software, the flags that decide how much of that hardware you actually get to use. This one is about the model itself, and something the China news sharpened for me. Every big lab is building fat models, huge generalists good at a hundred benchmarks with nothing to do with each other, and then almost nobody uses them like that. You use one for coding. Or for pulling fields out of documents. Or for one industry’s very specific paperwork. The model hauls around a mountain of capability you’ll never touch, and that really adds up when you are trying to run these models in constrained hardware.
So this post is about the techniques that can allow you to throw what you don’t need away, so you compress the model to your needs. If the supply of new open weights really is about to get choppier, doing that well matters more than ever if we want to be able to run these models at home. And my take is that this will not only affect individuals, but also organisations that increasingly rely on LLMs for their day-to-day.
Throwing away the experts that you don’t need
Quick recap, because it matters here. As we’ve discussed before in this newsletter, most of the interesting open-weight models you have the chance to run at home are Mixture-of-Experts (MoE). I went into this in part 2, but the short version: instead of one big dense network where every parameter fires on every token, a MoE model is chopped into a lot of smaller “expert” sub-networks, and a little router picks a handful to switch on for each token. DeepSeek-V4-Flash activates 6 experts plus a shared one per token, out of a pool of hundreds.
The reason everyone builds them this way is efficiency: you get the knowledge of a giant model but only pay to run a slice at a time. The key thing here is that the slices you never use don’t go anywhere. They still sit in memory, still take up disk, still set the floor on how much hardware you have to buy. Your coding model carries experts for languages you don’t speak and domains you’ll never ask about, but as you don’t know the specific experts and network layers that activate to access that knowledge, you are stuck with the full model.
In order to compress models, the obvious technique is to use quantization, which I wrote about earlier this year, where you make every weight a bit blurrier to save room. But you can also remove the experts that you don’t need. Inference engines for MoE models already avoid loading in memory the experts they don’t need. But what if we could track the experts and layers of a model that we will never use for our use case, and completely “reap” them from the model?
How REAP decides what to cut
The method to achieve this expert pruning that the local crowd is leveraging to make models affordable is REAP, a research from Cerebras Research. Router-weighted Expert Activation Pruning. (Annoying naming footnote: there’s a completely unrelated prompting technique also called REAP. This is not that one. This one cuts experts out of MoE models.)
The whole thing hangs on one number per expert, a “saliency” score, answering a pretty narrow question: when the router actually bothers to call this expert, how much does it change the answer? There are two ingredients to this technique. One is the router gate value, how strongly the router picks that expert. The other is the size of the expert’s output, how much it shoves the model’s internal state around when it fires. Multiply the two and average.
The bit I like is what it averages over. REAP only looks at the tokens where that expert was actually selected. So an expert that almost never fires but does something real every time it does gets a high score and stays, and one that fires constantly but barely moves anything gets cut. Firing often and mattering aren’t the same thing, and cruder methods mix the two up.
None of this needs retraining, which is what makes it usable for people who aren’t a big lab. You run a calibration corpus through the model once, record the gates and the output sizes, score everyone, delete the cold ones. Done. The “only” thing that you need is traces from your use cases that you can use for the pruning.
One thing worth noting about this technique is that the calibration corpus is everything. Feed it code and REAP keeps the experts that matter for code. It may seem obvious, but the paper shows the importance of data by calibrating a coding model on generic web text instead of code that basically wrecked its coding accuracy. So it is really important to choose your calibration set wisely to achieve the model’s specialisation, for which one needs to hold relevant data for their use case (this is why keeping all of your coding sessions with your agents can be so valuable in the future, and why you shouldn’t lose them. I am more than happy to follow-up on how I am ensuring I am keeping this corpus of agent interactions).
With REAP you’re not compressing a model in the abstract, you’re compressing it towards one job, and picking that job is the whole thing. If you get it right and the numbers are honestly a bit ridiculous. Cut half the experts out of Qwen3-Coder-480B and it keeps 97.6% of its coding ability, 96.7% on SWE-Bench. Half the model gone, a couple of percent lost on the task you cared about. Of course, there are no “free lunches”, that same pruned model gets noticeably worse at general multiple-choice knowledge, roughly 8% down on Qwen3-Coder and worse on a couple of others. It’s near-lossless on the thing you calibrated for and it starts showing seams off it. Which is what you’d want. You asked for a coding model and you get it, but don’t expect it to be good at everything (because it would require those experts you pruned).
Another way of shrinking MoE models is by merging experts instead of reaping them out. Merging replaces two experts with a fixed average. The issue with this approach is that the router network that decides the experts to activate for each token wasn’t trained on these merged experts, damaging the performance and the accuracy (which is the fundamental part of a MoE model). This is why removing half of the experts doesn’t impact performance significantly, but modifying the experts (by for instance merging) completely kills its accuracy because the router no longer knows the right experts to activate. (The paper has a proof for why merging is worse than cutting, functional subspace collapse, if you want the maths.)
How to see what to remove
One thing that I was really curious about when I was reading about REAPis how one can visualise the experts that activate for specific tasks. Think of a grid, with layers down one axis, experts across the other, and a saliency score in every cell. The cells with low saliency scores represent the parts of the model that you can cut without impacting the overall performance for the task/
The interesting thing is that when you draw it out, the cutting isn’t even. The REAP paper plots the expert activations (PCA plots, if you want to go read it) and the early layers turn out to be crowded together, the experts all doing broadly similar low-level work, not much you can safely pull. The late layers are the opposite. The experts fan out into all sorts of specialised behaviours, so for any given task a lot of them are just cold, so that’s where most of the cutting ends up happening. The dead weight isn’t spread evenly. It collects in the deep layers (but if you’ve worked with CNNs for image recognition, this same pattern may look familiar).
One note about this, REAP’s code computes all these numbers for you but won’t draw you the picture. If you want to see the map rather than read a table of scores, you’re building that yourself with the usual interpretability kit, TransformerLens and CircuitsVis. I highly recommend everyone interested in ML and LLMs, at least once in their lifetime, to visualise the internal layers of a transformer or a neural network, it can be creepy and eye-opening.
People are actually using it
The part I find genuinely exciting, and why I started researching this technique, is that it is already showing real results. That DeepSeek-on-a-Spark I opened with is real, and it’s 0xSero, who a few of you will remember from the hardware breakdowns in part 1. He took DeepSeek-V4-Flash, ran REAP down to 160 experts a layer, quantised what was left down hard, and got the whole thing to 103GB on disk, small enough to sit on one DGX Spark with room for a 200K context, at around 24 tokens a second with a bit of speculative decoding helping.
And this can be treated as a general-purpose technique for MoE models. There’s a 162B version too, pruned harder to 144 experts a layer, for when you want it to sit even more comfortably on the box and you’ll trade a little capability for the room. Same base model, same method, you just choose how deep the cut goes. He’s done the same to GLM-5.2, keeping 168 of its 256 experts. You can find the GGUF files for these models on 0xSero’s Hugging Face in case you want to try giving them a run with llama.cpp or your inference engine of choice..
All that being said, one-shot pruning isn’t quite free, and the GLM work is the perfect illustration of this. Straight after the cut the model started looping, repeating itself on about 7% of prompts against the original’s 3.6%. The fix is a tiny recovery step he calls Router-KD: retrain 0.016% of the parameters so the banged-up router remembers how the original one used to behave. Basically a rounding error of a training run, and it works. But it means “one-shot lossless compression” is not a thing. You cut the experts for free, and then every so often you pay a little to remind the router how to drive what’s left. Still, I think that a combination of REAP and quantisation, and the right toolings to adapt to specific tasks, can start enabling small narrow models to run in consumer hardware.
I am one of those convinced that fat models are meant for data centres, and narrow models for specific use cases running in consumer and mid-tier hardware.
One more piece of the puzzle
Compression gets a frontier-grade model, trimmed to the one job you need, onto a box you already own. That’s the model half of the plug-and-play appliance I’ve been circling since part 1: something you buy for a specific use (and corresponding model sizes), running locally, that doesn’t leave you renting a lab’s API forever or losing your life understanding how to build hardware and fine-tune the software for your needs (been there, still doing that).
The other half is the engine that runs it, hand-fitted to your model on your chip, the way antirez did with ds4. Compression decides what you run. Vertical integration decides how fast it goes once it’s there. That’s a whole post on its own (already added to my backlog).
The crazy thing about this is that if China really does start rationing the frontier open weights, this stops being a hobbyist’s optimisation and starts looking like a small insurance policy (even more for Europeans, as mentioned last week). The model on your disk today might be a better one than you can freely download in a year. Learning to cut it down to exactly what you need, so it runs on hardware you own and answers only to you, is a reasonable thing to know how to do before you find out you needed it. That, honestly, is why I haven’t put the rig idea down even though I’ve decided to delay the compute purchases for a bit (I hope I am not being a fool waiting or I may regret it:) ).
I really think there is a business opportunity here, and especially companies and small teams will start feeling the pain of relying on fat models. I may not have the time to explore this full-time for now, but if you or your team have a specific use case or need that you want to start co-designing around, that would be the push that I need to make this more tangible.
And if you’re already running a REAP’d model, I’d genuinely like to know what you calibrated it for and where it fell over. That gap, between being near-lossless on the job and clearly worse off, is the whole thing nobody’s mapped properly yet, and I keep thinking the interesting product hides somewhere in there. I am also interested in tooling. If we get tooling where REAP’ing models for specific use cases is easy, I could see more and more teams (and individuals) REAP’ing at night a model for their use case. Would love to know your thoughts.
Until next week!