Why on-device AI is the right default for a prompt tool

A prompt improver reads everything you’re about to ask an AI. Think about what passes through that pipe in a normal week: fragments of work documents, half-formed product ideas, personal questions, code you haven’t shipped, names of clients and colleagues. Drafts are the rawest text you produce — less filtered than anything you actually send.

Most tools in this category treat that stream as an asset. The text goes to their server, gets processed, and you take the privacy policy on faith. Even with the best intentions, a server is a liability: it can log, it can be breached, it can be subpoenaed, and its policy can change after you’ve built the habit.

The alternative: no server at all

RefineIQ takes the structural way out. There is no RefineIQ backend — not a minimal one, not an anonymized one, none. The default model (SmolLM2-360M-Instruct, with a lighter 135M tier for modest hardware) is downloaded once from a public CDN, cached in your browser, and runs locally via WebGPU with a WebAssembly fallback. After that download, the extension works fully offline. Privacy isn’t a policy that could drift; it’s an architecture that can’t.

This wasn’t practical until recently. Two things changed: small instruction-tuned models became genuinely good at narrow tasks, and WebGPU gave browsers real access to the GPU. Rewriting a prompt is exactly the kind of narrow task a 360M-parameter model handles well — it doesn’t need world knowledge, it needs to restructure the words you already gave it. A quality gate strips any markdown or preamble the model adds, enforces the length limit, and falls back to your original text if the rewrite adds nothing.

What the trade-off costs

Honesty matters here: on-device is not free. The one-time download is ~380 MB. On older hardware without WebGPU, refines run slower through WebAssembly. A 360M model won’t match a frontier model’s prose. For a prompt refiner, that’s the right trade — the task is bounded, the output is capped at 500 characters, and the gap in quality is small where it matters.

But some people want the bigger model anyway, and the answer to that shouldn’t be “so we’ll route your text through our server.” RefineIQ’s cloud mode is bring-your-own-key: you pick OpenRouter or Hugging Face, paste your own API key, and grant explicit consent. Each refine then goes directly from your browser to your provider. The maintainer never sees it — there’s nothing in the middle to see it.

How to evaluate any tool that reads your drafts

Three questions cut through most privacy theater:

1. Where does inference run? If the answer is “our API”, your text is on someone else’s disk.
2. Whose key is used? A tool proxying through its own key sees every request. Your key, direct to the provider, means the tool’s operator is out of the loop.
3. What breaks if the company disappears? An on-device tool keeps working. A server-backed one becomes a brick — which tells you where it really lived.

RefineIQ’s answers: in your browser; yours, only if you opt in; and nothing — it runs on your device and keeps working offline. That’s what “private by default” should mean. The full accounting of what is (and isn’t) stored lives in the privacy policy.