Sandro Turriate

How I made yap fast enough for a local coding agent

The slowest thing in yap was how much I made the model read before it could speak. So I cut the tool surface, trimmed the prompt, and rebuilt compression until a 27B Ollama model could actually drive the agent.

I started yap for a specific, slightly obsessive reason: I wanted a coding agent that could run on a local model. On my hardware. No cloud, no key, no per-minute meter — a brain that lives in my desk.

Six months ago, that was a joke

I tried driving Claude Code with local Ollama models a bunch of times, and it never worked — it would spin its wheels and hammer my GPU without ever finishing anything. I downloaded dozens of models. Most didn't even run. A discouraging chunk didn't even support tool calling, which is the whole point of a coding agent. The ones that did work just… weren't very good: looping, editing the wrong lines, and hallucinating. So I shelved the local dream and built yap properly for the models that could handle it. I've toiled at it constantly since, tweaking and improving it every which way.

Local had worked once

But I should be honest about the history here: yap did run a local model successfully, months ago. I got qwen3.5-coding — then the best local option — to grind through the full yap eval suite. It didn't score like a frontier model. But it passed all the tests. Local works in yap. It had, once.

Then, I now believe, the prompt exploded on me. yap grew. A whole roster of tools. MCP. A memory store, a knowledge base, plan tools, the meta-toolbag (yap_query, api_log, update_settings, summarizing). It became a coding agent for models that can afford a long conversation. And quietly, that killed the local experience.

Five minutes to answer “Hi.”

Then qwen3.8 dropped, and I decided to find out what the local model experience actually felt like in yap today.

The answer was sad. yap, pointed at qwen3.8 on my GPU, took five minutes to answer “Hi.” Five minutes. For a greeting. The chat model chewed on a single “hi” for three hundred seconds. My GPU, a furnace.

Not because 27B is slow at generating — it isn't. The killer is prefill. Before a model writes a single word, it has to read your entire context. And yap had ballooned that context. The first cold turn measured 3 minutes 10 seconds just to pay the prefill on ~19k tokens. Steady-state turns sat at 77–118 seconds each. The ~11,000 tokens of tool schema JSON alone, re-read and re-prefilled (at ~100 tokens/sec) on every turn, because one changed byte in the middle invalidated the whole cache. A local model, reading my agent's favorite dissertation about itself — before it even knew I said “hi.”

Debug yourself

So I did what I love doing these days. I told yap to launch itself in an interactive tmux session with a local-model flag, pointed it at its own codebase and said debug yourself.

And it did. Turn after turn, step after step. I watched the numbers come down. The 3-minute cold turn fell to the 2-minute range, then settled into steady-state turns of 77–118 seconds.

First came the solo-mode tool prune — a core of 11 tools with one-line descriptions instead of the full surface — which took a warm turn from 1m19s down to 51s.

Then came the timing fix: moving the per-turn volatile context (retrieved memory and knowledge slow-changes) out of the system prompt and into a trailing transient message, so the stable prompt plus tools stayed byte-identical across turns. Steady-state turns dropped to ~26 seconds. By then the prefix cache was holding — per-turn prefill had collapsed to near nothing, and most of those 26 seconds was the model yapping: answer length at ~25 tokens a second, not reading time.

The cuts were real

To get to these latencies I had to cut a lot out of yap. Memory. Knowledge base. AGENTS.md auto-injection. The MCPs. The meta-toolbag. The capacity to summarize. In solo mode, compaction doesn't summarize at all anymore — when the context gets long it keeps the verbatim request plus a short tail and drops everything older. No model call, and it's marked [solo truncation]. Because in solo, the human is sitting right there, watching, restating the task as it happens. The two anchors a fresh model needs are the live request and the newest exchange — not a compressed history written by a slow, reluctant narrator.

What remains does real work: bash, file read / batch read, the edit trio, file write, grep, glob, and web search/fetch. Everything else — workers, MCP tools, the knowledge store — is intercepted by name: solo marks any tool the agent tries to call as dead surface, because a small local model can't drive a subagent or a protocol bridge anyway. And the AST/LSP “search” tools you'd normally reach for? A one-liner in the shell does the job with ast-grep or ctags.

Tool descriptions collapsed to one line each. They were tuned for frontier models that actually attend to routing prose; a 27B model just pays for them, every byte. The JSON schema already names the parameter — the paragraph was dead weight. And even the “thinking about thinking” system prompt — Thinking Discipline, risky-action rules, slash-command ads, the skills catalog — is gone. Solo mode is lean to the bone.

Restoring was half the work: early on I only pruned the MCP tools and left the core registry half-restored. The fix is that solo snapshots the full tool registry on turn-on and restores everything non-MCP on turn-off. That's the difference between a feature and a trap.

Each cut, I asked the same question: is this worth the toll to a model that re-reads everything before it blinks? And one by one, the answer was no.

The honest numbers

After the cuts I stopped guessing and measured the real wire. The whole /solo prompt — system prompt, the 11-tool roster with one-line descriptions, a capped project map, a one-line user turn — comes to about 2,700 characters, roughly 700 tokens: one twenty-fifth of what yap used to feed the model every turn. Every tool description you prune is prefill that never happens.

Measured against qwen3.8:27b-nvfp4 on my M1 Max (64GB), model warm:

• Cold prefill of the whole 700-token wire: ~6.6s (≈107 tok/s) — paid only when a genuinely new prefix appears. Ollama's prefix cache lives server-side, so a fresh app with a warm server still hits it.
• Cached prefill (byte-identical repeat): ~0.5s — first token in half a second once the cache holds.
• Generation: ~25 tok/s — a 100-token answer lands ~4s after the first token, 500 tokens ~20s.
• Realistic solo turn (cached prefill + a 100-token answer): ~4.2s end to end. The bookend to “five minutes to say hi”: a minimal warm turn clocks ~1.0s against Ollama (833ms prefill + 115ms eval) — deliberately flattered, no agent loop, but it shows where the floor lives.

That's the whole game: when the cache holds, prefill collapses to ~0.5s and the time is the model talking; when it doesn't, you pay ~6.6s before the first token. The volatile-context fix keeps the stable prompt and tool registry byte-identical turn after turn, so the second case almost never happens — a full ~14k-token re-prefill would be ~150 seconds. (And the 26-second steady-state turns from the debugging session don't contradict the 4s picture: those were heavier, tool-driving turns with longer answers — the gap is output tokens, not cache misses.)

Then I stopped benchmarking and just used it — recording the session, fittingly, by running yap's CLI inside yap's own embedded terminal, the agent at work inside its own skin.

Here's every turn as it actually happened:

Every sub-3s turn decomposes exactly into those two numbers: ~0.5s of cached prefill, then ~25 tokens a second. Even a fresh app with no history came in at 0.7s rather than the ~6.6s uncached figure, because the fix pays rent twice: the prefix cache lives in the server, which stayed up across restarts, so the byte-identical wire kept hitting it. Real work costs what it costs — the Go turn's 10.7s is a file write, a bash run, and the model narrating its own output, each round-trip a fresh prefill. And the lone outlier — Erlang at 29.5s against Node's 1.9s and Java's 2.8s for the same shape of request — wasn't model speed: 29s at 25 tok/s is ~700 tokens, not a ten-line answer. Either the model sat on a long reasoning block (solo keeps thinking on by default, adaptive), or — more likely, right after the tool-heavy Go turn — the cache had evicted and yap paid a fresh prefill. Cache state and thinking time, not model speed. If you want the answer to start right after the prefill, /effort off turns thinking off entirely — a deliberate knob.

It works

And now, as I write this, that same model — a 27B qwen running in my box — is happily coding. Not as a demo, not chewing on “Hi.” It's making small, real improvements to a new project in my repo, using tokens that cost my GPU attention and not my wallet.

It's liberating. It's a little scary — my own coding agent, self-hosted for its model, spending free tokens to improve my projects. But mostly? It's exciting as all hell. Solo mode made the local model usable : a 27B brain in my desk that answers in four seconds and says hello in under one. And when the cloud budget runs dry — it always does, usually mid-project — the local model is still there, still fast, still working. Same agent, no meter. That's the win.