Why We Ditched Local LLMs in PhraseKit
When we started building PhraseKit, our AI writing assistant for international remote workers, we did the thing that sounds most impressive on a landing page: we ran the language model on the user’s own machine. No servers. No API bills. Your words never leave your laptop. It felt like the principled choice.
A few months later, we tore it out. This is the story of why — and why we think it’s the right call for almost anyone shipping AI to real people.
The dream of local
On paper, local inference is beautiful. You bundle a small open model — in our case a quantized
Llama 3.2 3B — and run it with node-llama-cpp right inside the desktop app. Privacy becomes a
non-issue because there’s nothing to send anywhere. Costs go to zero because there’s no provider
to pay. You can even work offline.
For a writing tool aimed at people who are, by definition, privacy-conscious about their English, this looked like a perfect fit. So we built it, and it worked. In a demo.
Where it fell apart
The trouble is that a demo runs on our machines. Our users don’t all have a 32GB developer laptop. And the moment PhraseKit met the real world, the cracks showed.
RAM was the killer. A 3B model, even quantized, wants somewhere north of 8GB of system RAM to run comfortably alongside everything else a remote worker keeps open — a browser with forty tabs, Slack, a video call, an IDE. On an 8GB machine, loading the model didn’t make writing faster; it made the whole computer crawl. The tool meant to reduce friction became the heaviest thing running.
The download was a wall. The model weights were roughly 2GB. That’s a first-launch experience where a new user installs your app, opens it, and then waits — on hotel Wi-Fi, on a capped mobile hotspot, on the kind of connection a lot of our target users actually have — for a 2GB download before they can fix a single sentence. A meaningful number of people never made it to the other side of that progress bar.
It didn’t work for everyone. This was the part that bothered us most. PhraseKit exists for people writing English as a second language on whatever hardware their job handed them. Gating the product behind “you need a recent machine with 8GB of free RAM” excluded exactly the users we were building for. An AI tool that only works well on expensive computers is not an inclusive tool.
And underneath all of that sat a quieter tax: maintenance. Model versioning, download resumption, disk cleanup when users uninstalled, handling the machine that ran out of space halfway through — every one of these was code we had to write and support instead of making the writing better.
What we replaced it with
We moved to cloud inference, and we did it without giving up the privacy promise that made local attractive in the first place.
PhraseKit now runs in one of two modes:
- Managed inference. You sign in and we handle the model call for you. Our primary model is a small, fast, multilingual one that returns a first token in under a second — quicker in practice than spinning up a local model, and dramatically lighter on your machine. We deliberately chose a provider tier that does not train on user inputs, with an independent fallback provider so a single outage never takes the app down.
- Bring your own key. Advanced users can plug in their own provider API key, and the desktop app talks directly to that provider. Nothing routes through our servers at all.
In both modes, your text isn’t stored on PhraseKit servers, and the app now installs in seconds and runs happily on modest hardware. The privacy story survived. The RAM problem didn’t.
The lesson
“Runs locally” is a great sentence in a pitch deck and a hard promise to keep in someone else’s RAM. Local inference isn’t wrong — for the right model, the right hardware, and the right user, it can be exactly right. But it is a serious product constraint dressed up as a feature, and you should choose it with your actual users’ machines in front of you, not your own.
We build our own products precisely so we learn these lessons on ourselves first. If you’re weighing local versus cloud for an AI product of your own, we’ve now made the expensive version of this mistake so you don’t have to — come talk to us.