A 120B model that costs less to run than a 27B one? How does that work?
There's a belief I hear in almost every conversation about running models locally: once you've bought the machine, the tokens are free. You bought a Mac, you installed a model, and that's it, the electricity is a rounding error.
This morning I came across an article in Towards Data Science that puts that belief to the test, and I have to share it with you, because one of the findings is genuinely counterintuitive. Justin Stewart took a Mac Studio with an M3 Ultra chip and 96GB of unified memory, ran five local models on it, and measured how much electricity each one actually drew. Not an estimate, not a simulation, a calibrated measurement against a real power meter at the wall.
Let's look at what he found:
Why these numbers are worth taking seriously
This is the part that made me stop and read to the end. You could simply read the power counters that Apple's chip reports about itself, the same ones behind the Energy tab in Activity Monitor. The problem is that internal counters tend to drift, and a reader has no way to know whether to trust them.
Stewart solved this nicely: he plugged the machine into a smart plug that measures real energy use, and calibrated the internal counters against it. Every number in the article comes with a real margin of error, roughly between 2.6% and 4.5%. He also subtracted the machine's idle draw, so what he measured is the marginal cost of the request itself, not of a computer simply being switched on.
I tend to be skeptical of home measurements, but the methodology here is solid enough to draw conclusions from. That's exactly what makes the surprise we're about to get to worth paying attention to.
The results
Note that they're sorted by cost, not by model size
The electricity rate in the experiment was $0.31/kWh, and cost is calculated per million output tokens:
| Model | Parameters | Type | Throughput (tok/s) | Cost per 1M tokens |
|---|---|---|---|---|
| Qwen3.5-4B | 4B | dense | 133.8 | $0.063 |
| Qwen3.6-35B-A3B | 35B | MoE | 76.0 | $0.087 |
| Qwen3-Coder-Next | ~80B | MoE | 65.0 | $0.103 |
| gpt-oss-120b | 120B | MoE | 74.0 | $0.109 |
| Qwen3.6-27B | 27B | dense | 21.5 | $0.554 |
Stop on that last row for a moment, because that's where the story is.
The most expensive model in the table, by a factor of five to nine, is the 27B dense one. Every model larger than it came out cheaper. The 120B model runs a token at about a fifth of the price of a model that's a quarter of its size, and it does it faster, too.
In other words, if you'd picked a model by the obvious logic, "27 billion parameters is less than 120 billion, so it must cost me less", you'd have landed on the single most expensive option!
So what's actually going on here
The formula is simple, and worth remembering: cost per token = watts divided by tokens per second.
Two variables, and the dense model loses on both. It draws the highest power in the table (about 138 watts) and produces the fewest tokens per second.
The explanation, and I think this is the most elegant part of the article, lies in memory bandwidth, not in compute power. To generate a single token, the machine has to read all the active weights out of memory. A dense 27B model at 8 bits activates every parameter for every token, meaning it pulls about 28 gigabytes of weights on every step. That's the largest data movement in the table, and it's exactly what makes it both slow and power-hungry.
An MoE model breaks that link. In gpt-oss-120b, all 120 billion parameters sit in memory, but an internal router activates only about five billion of them per token. Fewer bytes in motion, more speed (74 tokens per second), less power (94 watts). The more aggressive quantization used by the MoE models widens the gap a little further.
If there's one sentence I'd want you to take from this post, it's this: you pay for bytes moving through memory, not for parameters sitting in it. Parameter count only determines the cost when the model activates all of them.
Does it hold up under real workloads?
That was the first question that came to my mind, and I'm glad Stewart asked it too. All the numbers in the table were measured under lab conditions: the machine generating tokens continuously, with no pauses and no idle time.
Day-to-day work looks nothing like that, a short request here, waiting on a user there, and every token ends up costing more.
So he checked that as well, across roughly 6,300 real requests that accumulated over a month. The gap didn't close; it grew. The dense model cost him about ten times more than the MoE models.
Three caveats worth knowing
- This is electricity cost only - The machine itself isn't in the equation, and it's the real expense here. Divide the price of the Mac across a few thousand requests and you get a number that dwarfs a tenth of a cent of electricity. This measurement shows you what a token costs after you've already paid for the hardware, not what the whole operation costs you.
- Unified memory enables a lot, but it also constrains you - An MoE model has to keep all of its experts in memory, even if it only activates two of them per token. The 120B model takes up about 59GB of RAM, and after the operating system and the context cache there isn't much headroom left on a 96GB machine. One more detail that surprised me: simply holding a large model in memory costs about 21 watts constantly, even when it isn't generating anything.
- The comparison to the cloud is less flattering than it looks - True, a tenth of a cent versus a few dollars per million tokens sounds like a knockout. But in the cloud you're also paying for the hardware, the hardware you bought out of pocket here, and you're getting far larger models. So the conclusion isn't "local is ten times cheaper," but something more precise: once the machine is already yours, each additional token costs very little, and your choice of model is what determines whether that's "very little" or "ten times that."
What I'm taking from this, and suggest you do too
- Stop picking models by size. Measure throughput on your own hardware and choose based on that.
- Take MoE seriously if you have enough memory to hold one. The physics works in your favor.
- A model running in the background that isn't working, shut it down. A model sitting in memory "just in case" costs you electricity and resources.
- Clear the quality bar first. Cost is a tiebreaker between models that can do the job, not a substitute for checking whether they can do it at all.
And one last thing, maybe the most important: all these numbers were measured on one machine, at one electricity rate, under one workload. All three will be different for you.
The tool used for the measurement, TokenWatt, is open source and connects to the server you're already running, so anyone who wants to can repeat the test on their own hardware. And that's the right approach in my view: don't adopt someone else's numbers, learn from them what's worth measuring in your own environment.
📎 The original article: How Much Does a Local LLM Actually Cost to Run? by Justin Stewart, in Towards Data Science. Worth reading in full.
How about you, are you running models locally? Did you choose by size or by speed? I'm curious to hear which model actually held up for you in practice, and not just in a benchmark.
Feel free to email me at dolev@ravid.email or reach out on LinkedIn.