Back to Articles AI Critique

Why Local AI Can't Keep Its Promises

Every token still costs billions of multiplications. Quantization makes them cheaper, never fewer. That floor is why "private, on-device, real-time" rarely means all three at once, and what honest local AI can actually do.

25 August 2026 20 min read
Edge AI is not a linear algebra problem: a dense weight matrix does not fit on an edge device, and shrinking it lowers the cost, not the wall

A personal note before I start. I am writing this in response to the technical claims that stream through my LinkedIn feed every day. This article deals in real truths and real data, not marketing hype, deflection, or spin. Everything in it is my personal opinion, grounded in technical fact.

Every few weeks a new product promises the same thing. Artificial intelligence that runs on your own device, keeps your data private, and does real work for you. No cloud, no privacy tradeoff, no compromise. It is a genuinely appealing pitch, because it is the thing a lot of us actually want. It is also, in almost every case, not what the product does.

Before taking the pitch apart, it is worth being exact about what these products are, because the marketing rarely is. Underneath almost all of them is a transformer: a large language model, or its smaller cousin the SLM, a small language model trimmed to fit on consumer hardware. Around that core sits a familiar toolkit. Retrieval, or RAG, fetches relevant text and pastes it into the prompt. Quantization compresses the model's weights so it loads in less memory. Memory injection, sometimes called context stuffing, feeds your personal history into the input so the model appears to know you. Products mix these differently, but they all rest on one computational base: linear algebra, dense matrix multiplication over arrays of token values. That is the architecture under review. When a product says "private on-device AI," this stack is almost always what it means.

This is not aimed at any one company. It is a pattern, and the pattern is worth understanding, because the gap between what these products claim and what they deliver is not a rounding error. It is the architecture I just named.

Four Properties Sold as One

When someone says "private on-device AI that does your work," they are quietly claiming four separate things at once:

  • Local: the computation happens on your hardware.
  • Private: your data does not leave your device.
  • Real-time: it responds when you need it, not hours later.
  • Capable: the model is smart enough to do useful, general work.

Buyers hear these as a single feature. They are four, and on today's consumer hardware you can hold at most two or three of them at the same time. The marketing works by implying all four. The engineering picks a subset and stays quiet about which one it dropped.

The Prefill Tell

Start with the hardest combination to fake: local, real-time, and capable, all together, over a large amount of your personal context.

The bottleneck people forget is not how fast the model writes its answer. It is how long the model takes to read the input first. That step, prefill, scales with how much context you feed it. Ask a local model to understand your entire life, your email, your messages, your files, and you are asking a memory-bound consumer chip to process an enormous context window before it produces a single useful token. On an 8GB laptop that is not a snappy interaction. It is minutes.

So watch what the careful ones do with the timing. When a product says it works overnight, while you sleep, that is not a lifestyle feature. It is an admission. It runs at night because it cannot run in real time. The batch schedule is the confession. There is nothing wrong with batch processing, it is a sensible design, but it is the opposite of the responsive assistant the headline implies.

The Offload Tell

The other way to make a small local model feel capable is to stop doing the hard part locally. Many products described as on-device do the light work on your machine and then send the heavy reasoning to a cloud model, often through your own subscription to a large provider.

Read that slowly, because this is where the privacy claim quietly dies. If your personal context is sent to a cloud model to finish the job, your data left the device. The usual defence, "our servers store nothing," is technically true and beside the point. The exposure did not disappear. It moved to someone else's servers, under your account. A product can be perfectly honest that it stores nothing and still ship your whole life to a third party the moment it needs a real answer. On-device became cloud AI with a local cache, and private became private from us but not from them.

A Smarter Client Still Phones Home

There is a more sophisticated version of this worth addressing head on, because it is where the industry is heading. Instead of a thin app, you run a capable local agent: an interface layer on the device, a desktop agent or a local node, that holds context, plans, calls tools, routes requests, and decides what to do next. It feels like intelligence living on your machine, and in a real sense some of it does.

But look at where the thinking actually happens. The local agent is orchestration. It manages the conversation, keeps state, chooses which tool or model to invoke, and formats what comes back. The reasoning it cannot do itself, the part that needs the large model, it sends to the cloud as an API call. This is moving intelligence to the edge in precisely the way that does not change the privacy or the dependency. A smarter client is still a client. It still phones home for the hard part, and your context still travels with the request when it does.

The distinction matters because the two are so easy to conflate. An agent that runs locally is not the same thing as intelligence that runs locally. The first is a control loop on your device wrapped around a model that lives somewhere else. The second is a model that actually runs where you are. Most of what is sold as on-device agentic AI is the first. It moves the interface to the edge, not the intelligence. That is not true AI on the edge. True AI on the edge would mean the reasoning itself runs on your hardware, with nothing to phone home to.

The Bill a Smarter Client Hides

Treating the cloud model as the brain and the device as a smart client is not just a privacy problem. It carries costs the pitch never mentions, and they get worse the more you rely on it.

Start with context. The model on the other end of the call is stateless. It remembers nothing between requests, so the agent has to bring it up to speed every single time: your history, the retrieved documents, the tool outputs, the running conversation, all resent on every turn. That is the memory injection and the RAG from earlier, and it is not free. Each call pays to process all of that context again, the prefill cost from before, now metered. Longer memory means longer prompts, means more tokens, means more money and more latency, on every request. Context does not accumulate cheaply. It is re-billed from scratch each time you use it.

That is where the "marginal cost is basically zero" claim falls apart. It is zero to the vendor, because the meter is running on your subscription, not theirs. Someone pays per token, per call, for as long as the product runs, and the bill grows with exactly the thing that makes it useful: more context, more history, more autonomy. An agent that takes ten steps to finish a task makes ten billed calls, each carrying the full context along with it. The economics do not improve as you lean on it. They compound against you.

And none of the orchestration fixes the thing underneath. The cloud model still guesses. It hallucinates, confidently, and wrapping a local agent around it does not make it more truthful, it makes it more autonomous. That is the dangerous pairing. When a chatbot hallucinates you tend to catch it, because you are reading every answer. When an agent hallucinates in the middle of a task, it acts on the false step and keeps going, and the error propagates through everything downstream. More autonomy on top of an unreliable core is not more intelligence. It is less oversight of the same unreliability.

So the smarter client is not merely still a client. It is a client that re-bills your context on every call, depends on a network to think at all, and inherits every failure of the model it phones home to while removing the human who used to catch them. The obvious question is why the reasoning cannot simply run on the device, so the loop closes locally and none of this applies. That comes down to a single cost, and it is the floor under everything else.

The Floor Nobody Can Quantize Away

Credit where it is due, because the engineering to shrink these models is real and genuinely clever. Quantization drops each weight from sixteen bits to four, sometimes fewer, and a good k-quant scheme keeps most of the quality. Distillation trains a small model to imitate a large one. Pruning strips out the weights that were barely contributing. Speculative decoding lets a small model draft while a larger one checks. Mixture-of-experts activates only a slice of the network per token. Every one of these is a real advance, and together they are why a capable model fits on your laptop at all. None of it is spin.

But notice what they all have in common. They make the same operation cheaper. They do not remove it. And the operation is the problem.

Underneath every one of these models is dense matrix multiplication. The model represents meaning as long arrays of numbers, the token values, and it computes by multiplying those arrays through layer after layer of learned weight matrices. Producing a single token means pushing your context through that entire stack of multiplications. Quantization shrinks the numbers being multiplied. Pruning removes some of them. Sparsity skips some of them. But the shape of the work never changes: you are still multiplying enormous arrays of values by enormous arrays of weights, billions of multiply-and-add operations for one token, and the total grows with both the size of the model and the length of what you ask it to read.

That is the floor, and it is not a flaw in any particular product. It is the cost of the representation itself. This is the deepest reason local generative AI strains against the hardware: the entire paradigm computes by brute-force linear algebra over massive numerical arrays, and you can make each multiplication smaller and faster for years without ever changing the fact that you must perform an astronomical number of them. Every optimization in the field, real as it is, buys a better constant in front of the same term. The term stays.

The Race to the Edge

The floor explains why the whole industry is now racing to get AI onto the device, and it is worth laying out the actual playbook, because it shows both what is being attempted and what stays out of reach. The approaches fall into a handful of buckets. Almost all of them make the matmul cheaper. Only a few try to remove it.

Shrink the trained model. Quantization stores the weights at lower precision: GPTQ and AWQ for GPUs, QLoRA for fine-tuning, and the k-quant formats in llama.cpp and GGUF that put usable models on a laptop. Pruning deletes the weights that contribute least, one-shot and without retraining, as in SparseGPT and Wanda. Distillation, the old idea from Hinton and colleagues, trains a small student to imitate a large teacher. All three take a big model and make it smaller. None changes what the smaller model is doing.

Build small from the start. Rather than shrink a giant, train a compact one: Microsoft's Phi-3, Google's Gemma, Meta's Llama 3.2 at 1B and 3B, Alibaba's Qwen, Hugging Face's SmolLM, Apple's OpenELM, Meta's MobileLLM. These small language models are genuinely useful inside a narrow band, and they are still transformers doing matmul. Capability falls away as the parameter count drops.

And they are not fast. This is the part the benchmarks hide. Run even a three-billion-parameter model on an ordinary phone, or a mid-range laptop with no discrete GPU, and you wait: a handful of tokens per second, a long pause before the first word on any real prompt, the battery draining and the fans spinning up. Small enough to load is not the same as fast enough to use. For anything past a short, bounded task, the experience is closer to unusable than to the instant assistant the demo implies. The models that fit comfortably are too weak to matter, and the models strong enough to matter are too slow to bear, which is the same tradeoff in a different costume.

Make the hardware and runtime faster. A whole layer exists to run these models efficiently on the device: llama.cpp and Ollama on the CPU, Apple's MLX on Apple silicon, Google's LiteRT, Meta's ExecuTorch, MLC-LLM, ONNX Runtime, Qualcomm's QNN. Underneath sits dedicated silicon, the Apple Neural Engine, Qualcomm's Hexagon NPU, the 40-plus-TOPS NPUs that now define a Copilot+ PC. This is real and it matters, but it raises the floor by making each multiplication faster and cheaper in power. It does not remove the multiplication.

The vendor systems, and the hybrid hedge. The most advanced shipping on-device AI comes from the platform owners: Apple Intelligence, which runs a roughly three-billion-parameter model on the phone; Google's Gemini Nano through Android's AICore; Microsoft's Phi Silica on Copilot+ PCs. Look closely at what even they do. They keep the small, fast tasks on the device and route the hard ones to the cloud. Apple's Private Cloud Compute is the most honest and best-engineered version of this, a verifiable private server tier for exactly the requests the phone cannot handle. It is a genuine privacy advance. It is also a concession: the best-resourced teams on earth still split the work, because the on-device model alone cannot carry it.

Compute less of the model per token. Mixture-of-experts, from the Switch Transformer to Mixtral, activates only a slice of the network for each token, so a large model runs at closer to the cost of a small one. It is a clever saving, but the whole model must still be resident, and each active slice is still matmul.

Replace quadratic attention. A serious line of work attacks the part of the cost that grows with context length. State-space models like Mamba, recurrent-style architectures like RWKV, and recent zero-order RNN training all trade the transformer's quadratic attention for linear recurrence with constant memory. This is the most structurally significant shift in the mainstream, and it is real progress. It is also still linear algebra, still learned by gradient descent, still a large model of trained weights.

Attack the multiplication itself. The frontier worth watching, because it names the real enemy, is the 1-bit and matmul-free work. BitNet trains networks whose weights are ternary, so the core operation becomes addition instead of multiplication. MatMul-free language modeling goes further and removes dense matrix multiplication from the network entirely, replacing it with additive and element-wise operations. That the field's own leading edge is trying to delete the matmul is the strongest possible confirmation that the matmul is the floor. But notice what these still are: trained networks, learned by gradient descent over enormous parameter counts, approximating a process rather than capturing its structure. They lower the floor dramatically. They do not step off it.

Why It Crawls, and Why Only Money Fixes It

The slowness is not an implementation detail that better code will erase. It is architectural, and it has a name: the memory wall. Generating text with a transformer is autoregressive, one token at a time, and producing each token requires reading the model's entire set of weights out of memory. Not computing with them cleverly, reading them, all of them, for every single token. The useful arithmetic per byte read is tiny, close to one operation per byte, so the processor spends almost all its time waiting on memory rather than calculating. Engineers call this being memory-bandwidth bound, and roofline analyses of LLM inference put the decode phase hundreds of times below the hardware's compute ceiling for exactly this reason (see the roofline survey of Yuan and colleagues, and on-device roofline work such as RooflineBench).

That single fact explains the whole experience. A model's token rate on a given device is capped, roughly, by memory bandwidth divided by model size. Shrinking the model with quantization helps because there are fewer bytes to stream, which is why a 4-bit model feels faster than a 16-bit one, but you are still bottlenecked on moving weights, not on thinking. The public benchmarks bear it out. MLPerf Client, the industry's on-device suite, reports the two numbers that actually matter, time to first token and tokens per second, and on real consumer hardware both are humbling for anything past a small model. Independent mobile benchmarks show even a sub-billion-parameter model managing only a couple of dozen tokens per second on a Raspberry Pi class device, and that is the easy case.

Here is why this points straight back at cost. The way you beat the memory wall is with memory bandwidth, and memory bandwidth is exactly what you are buying when you pay for a datacenter GPU. The high-bandwidth memory on an H100-class card moves data more than an order of magnitude faster than the memory in a phone or a laptop. That is the real product of the expensive stack: not more intelligence, just a wider pipe to stream the same giant array of weights through, fast enough to feel instant. A workstation humming in the corner drawing hundreds of watts can do it. The phone in your pocket and the sensor on a machine cannot buy that pipe. So the architecture leaves three doors, and only three: run a model small enough to be weak, wait through a model slow enough to be unusable, or spend thousands on hardware to brute-force the bandwidth. There is no fourth door that keeps this architecture and delivers fast, capable, and cheap at the same time. The wall is load-bearing.

Why This Is Not Pedantry

You could call all of this hair-splitting. It is not, and here is why it matters beyond the spec sheet.

Most of the people making buying decisions, and most of the journalists shaping the story, cannot see the architecture. They hear private local AI and they believe, reasonably, that their data never leaves their laptop. When that belief is wrong, it is wrong in the one direction that hurts people: the direction where sensitive data went somewhere they did not expect. And every time one of these claims is walked back, or surfaces in an incident, it spends trust that the whole field has to borrow against.

That is the real cost of the hype. Not that the products are bad, many are genuinely useful, but that the language has been inflated until it means nothing. When the correction comes, it lands on everyone, including the developers who described their own systems accurately. Overclaiming is not a victimless marketing choice. It taxes every honest builder in the same space, because it teaches the market that private and on-device are advertising words rather than technical ones.

What Honest Local AI Looks Like

None of this means local AI is a fiction. It means the honest version is scoped.

Small models running fully on-device are real and useful for narrow, bounded tasks: autocomplete, dictation and transcription, wake-word detection, summarizing a single document, sorting an inbox. These run locally, privately, and fast, precisely because the task is small and the context is short. The claim is modest and it holds.

The overreach begins the moment the pitch expands to "understands everything about you and acts on your behalf, locally and privately and instantly." That specific combination is where the physics runs out and the marketing takes over.

A Test You Can Apply

So here is a simple test for any product that calls itself on-device or private. Ask two questions. Does it call a cloud model to finish the job? And does my data leave the device to do so? If the answer to either is yes, it is cloud AI with a local component. That is a fine thing to be. It is not what the headline said.

Hold the words to their meaning. Local should mean local. Private should mean private. Real-time should mean now. When a product delivers two of the three, respect the engineering and correct the sentence. The technology is impressive enough without the spin.

A Different Machine

Every approach in that playbook keeps the same kind of machine and makes it smaller, faster, or cheaper to run. Even the matmul-free frontier is still a trained network approximating a process. It is worth saying plainly what a different kind of machine would be, because that is the part I work on. Senua AI does not run a transformer. It does not run a language model at all, large or small. There are no arrays of token values, no stack of weight matrices, and so no matrix multiplication to perform, quantize, or offload. There is nothing to phone home to.

Instead of approximating a process with billions of parameters trained by gradient descent, it induces the causal structure of a signal directly from the signal. The result is the minimal machine that reproduces the process, and its memory is intrinsic: a causal state is, by construction, a sufficient summary of everything in the past that matters for what comes next. The crutches the transformer stack needs fall away with it. There is no RAG, because the model is not retrieving text to paste into a prompt; the structure is the model. There is no quantization, because there is no giant array of weights to compress. There is no memory injection, because state is not stuffed into a context window, it is what the machine is made of. This is not a smarter client. It is the intelligence itself, running where the signal is.

I am not claiming this replaces the chatbot on your phone. It is not a general language model and does not pretend to be one. Its domain is reading and reacting to the signal of a real process: telemetry, protocol bytes, audio, video, the streams a machine actually produces. But that domain is exactly the one the "on-device assistant" pitch keeps reaching for and cannot hold, because it is trying to do it with the wrong machine.

Why the Difference Matters

Return to the floor. The reason local AI cannot keep its promises, the reason it batches overnight and offloads to the cloud and ships you a smarter client instead of real intelligence, is the matmul: an astronomical number of multiplications for every token, growing with model size and with the length of what you ask it to read. That single cost is what forces the tradeoff between local, private, real-time, and capable.

Remove the matmul and the tradeoff goes with it. When there is no giant computation to run, there is nothing to offload, so your data has no reason to leave the device and private stays private. When the state updates once per observation instead of multiplying your whole history through every layer, real-time is not a stretch, it is the default. When the machine is sized by the complexity of the process rather than a parameter count chosen in advance, it fits on hardware the transformer stack cannot touch. The edge agent no longer has to phone home, because the reasoning it was outsourcing now runs beside it. The four properties stop competing because the thing that made them compete is gone.

That is the whole argument in one line. The promises are not broken because engineers are dishonest or lazy. They are broken because the architecture underneath cannot keep them, and no amount of quantization, retrieval, or memory injection changes the architecture. If you want local, private, and real-time to be true at the same time, and you want the intelligence itself on the edge rather than a better client in front of the cloud, you do not tune the model. You change the machine.

References and Further Reading

Making the model smaller: quantization, pruning, distillation

Running it faster: inference tricks, sparsity, runtimes

Small models and shipping on-device systems

The speed bottleneck: benchmarks and the memory wall

Attacking the matmul, and post-transformer architectures

The mathematics of causal states

This piece continues the argument from "Zero-Order Training, Zero-Gradient Learning" and "The Physical-World Turn": the honest frontier of edge intelligence is not a smaller copy of a cloud model, it is a different architecture that runs where the signal lives. At Senua AI that is the layer we build.