Do token-saving plugins make AI coding agents worse?

AI coding agents have started acquiring personalities. Install Caveman and your agent drops articles and speaks in fragments. Install I Have ADHD and it leads with the next action, caps lists at five items, and suppresses tangents. Add RTK or Context Mode and the agent sees a compressed version of command output instead of the raw result.

The appeal is obvious. Agent sessions are expensive, context windows still fill up, and nobody enjoys scrolling past another paragraph that begins, “You’re absolutely right.”

But the numbers used to sell these tools are usually variations of the same number: tokens removed. That is not the same as work preserved.

A coding agent is not a document compressor. Its job is to inspect evidence, form a plan, make changes, and catch its own mistakes. A plugin that removes 90% of an agent’s context has not demonstrated a 90% improvement. It has demonstrated that 90% of something disappeared.

A removed token may be filler. It may also be the clue that explains the failure.

Four tools, four different interventions

Grouping every token-saving plugin under “context optimisation” hides where each one intervenes in the agent loop.

Caveman is primarily a presentation layer. Its skill tells the model to drop filler, articles, hedging, long error logs, tool-call narration, and unnecessary explanation. The default full mode permits sentence fragments and compressed grammar while promising to preserve technical substance.

I Have ADHD goes further by combining an output style with an interaction policy. The agent must lead with an action, number multi-step work, suppress tangents, restate state, provide specific time estimates, and finish with one concrete next step. Those rules can make an agent much easier to use. They can also affect which options it considers worth mentioning and how quickly it moves from diagnosis to action.

RTK, short for Rust Token Killer, sits on the other side of the model. It intercepts shell commands and filters their output before the agent reads it. Passing tests collapse to a count. Git logs lose metadata. Search results are grouped and truncated. Repeated log lines collapse into one line with a count.

Context Mode changes the workflow more broadly. It asks the agent to process large outputs inside a sandbox, store material in SQLite with FTS5 search, and return only derived answers or selected snippets to the conversation. Instead of reading 50 files and reasoning over their contents, the agent is encouraged to write a script that computes the answer.

These are not equivalent changes:

Layer	Example	What can change
Final presentation	Caveman	Explanations, caveats, uncertainty and warnings
Interaction policy	I Have ADHD	Planning, prioritisation and when the agent acts
Tool observations	RTK	The evidence available to the model
Context architecture	Context Mode	How evidence is processed, stored and retrieved

The closer a plugin gets to the agent’s observations, the more directly it can affect reasoning. If RTK removes the line that explains why a test failed, the model cannot reason from that line. It does not matter how capable the underlying model is.

Shorter can be better, or much worse

There is no simple relationship between answer length and model quality.

A 2024 study of personas in system prompts tested 162 roles across nine models from four open-weight model collections and 2,410 MMLU questions. Adding a persona did not improve average performance over using no persona. Individual personas changed accuracy, but the effects were inconsistent, and automatically selecting a helpful persona proved difficult. The study did not cover closed models or open-ended agent work, but it does show that persona instructions are not reliably neutral.

A persistent style skill changes more than presentation. It adds another system-level instruction for the model to follow alongside the task.

Output constraints can have a clearer cost. The EMNLP 2024 paper Let Me Speak Freely? tested models under structured output requirements such as JSON and XML. Reasoning performance declined under format restrictions, with stricter constraints generally causing larger drops.

Brevity has a more mixed record. The Benefits of a Concise Chain of Thought cut the average response length of GPT-3.5 and GPT-4 by 48.7% on a multiple-choice benchmark with little overall accuracy loss. On the two maths sets, however, concise prompting reduced GPT-3.5’s accuracy by 27.69%.

A 2026 preprint reached almost the opposite headline. Brevity Constraints Reverse Performance Hierarchies in Language Models identified 115 inverse-scaling problems, 7.7% of its 1,485-problem set. In a follow-up intervention using seven selected models, brief prompts improved the four designated large models by 26.3 percentage points on that subset. The authors describe the result as an upper-bound estimate under greedy decoding, not an average effect across all models or tasks.

Both findings can be true. Extra text can introduce distraction, contradiction, and confident improvisation. Remove too much, though, and the model loses the space to state assumptions, compare alternatives, or notice that its first answer is wrong.

The useful amount of output depends on the model and the task. “Explain git rebase” can survive aggressive compression. Diagnosing a race condition from partial logs may not.

A coding-specific warning came from an issue filed on behalf of Moonshot AI. The company reported that OpenCode’s generic prompt scored 54.1 ± 3.8 and 60.0 ± 2.4 on two private coding and reasoning benchmarks, compared with 58.0 ± 2.4 and 67.1 ± 1.0 using a Kimi-specific prompt. Moonshot cited aggressive brevity, trivial few-shot examples, and conflicting instructions as possible causes. Several prompt features changed together, so the comparison does not isolate brevity.

The datasets and reproduction procedure were not published. OpenCode also merged the dedicated Kimi prompt that day, so this describes a resolved configuration rather than current default behaviour. It remains a useful example of a model vendor finding that a generic agent prompt was not behaviourally neutral.

The visible answer is not the reasoning trace

The explanation we see is not necessarily the reasoning the model used.

OpenAI’s guidance for reasoning models says models such as its o-series reason internally and generally do not benefit from prompts asking them to “think step by step.” Anthropic’s current thinking documentation likewise separates thinking blocks from final text and accounts for thinking tokens separately, with controls that depend on the model.

So Caveman’s basic claim, “brain still big, mouth small”, is architecturally plausible. A short final answer does not mechanically imply a short internal reasoning process.

But that separation is not a guarantee of behavioural neutrality. The style instruction still enters the prompt. It can affect when the model stops, what tools it calls, which uncertainty it exposes, and whether it reports a concern to the user. In an agent loop, plans, tool calls, summaries, and final prose also feed later turns. Today’s compressed explanation can become tomorrow’s incomplete context.

Three separate layers matter here: hidden reasoning, visible communication, and tool observations. Plugins can alter one, two, or all three.

Context compression changes the evidence

Raw context is not automatically good context. Long transcripts contain repeated logs, irrelevant search matches, stale plans, and thousands of lines of successful test output. Removing that material can help.

The Lost in the Middle study showed that long-context models often perform best when relevant information appears near the beginning or end of the prompt. Performance can fall when the same information sits in the middle, even for models designed to accept long inputs.

Microsoft’s LongLLMLingua went further and deliberately compressed long prompts. On NaturalQuestions, it reported improvements of up to 21.4% while using roughly four times fewer tokens with GPT-3.5 Turbo. Removing low-value context increased the density and visibility of useful information.

Context Mode and RTK are built around a valid premise. A terminal can produce a hundred lines when the agent only needs the exit code and one failing assertion. Keeping every byte is not caution; sometimes it is noise preservation.

The risk is that relevance is often obvious only after the bug is understood. A warning in a passing test, an unchanged line around a diff hunk, or the fifth repeated log entry may be the clue that changes the diagnosis. Compression asks a filter, retrieval system, script, or model to decide what matters before the investigation is complete.

RTK has encountered this boundary in practice. Issue #827 described a large diff being truncated without a sufficiently clear warning. The user said Claude summarised the partial output as if it were complete, and they made a decision based on it. RTK fixed the direct rtk diff path so it no longer truncates changes or long lines. Its separate compact rtk git diff path can still truncate large hunks, but now marks the omission and provides rtk git diff --no-compact as a recovery path. That is a good maintainer response, but it also demonstrates the failure mode: invisible compression turns missing evidence into false confidence.

Context Mode reduces this risk by indexing large outputs and allowing the model to retrieve exact chunks later. That is better than silently throwing data away. It still introduces a retrieval dependency. The model has to recognise that it needs more information, formulate the right query, and inspect the right result.

Compression can improve reasoning when it removes noise. It can damage reasoning when it hides the clue that would have made the model ask a different question.

The benchmarks measure the wrong outcome

The projects are often more honest about this than their headline percentages suggest.

Caveman advertises an average 65% reduction in output tokens across ten prompts, compared with an unconstrained “helpful assistant” baseline. The raw responses behind that table are not committed. The repository’s newer three-arm snapshot, which includes a plain Answer concisely control, shows roughly a 50% median reduction against that terse control.

Its own evaluation documentation explicitly says the harness does not measure fidelity. As the documentation puts it, a skill that replied k to everything would score −99% and “win.” The Honest Numbers page also estimates that the skill instructions add roughly 1,000 to 1,500 input tokens per turn. For already-terse coding conversations, the plugin can consume more tokens than it saves.

I Have ADHD includes a more promising evaluation harness with scoring for correctness, autonomy, actionability, safety, and concision. At the time of writing, however, the repository does not publish completed comparison results. The existence of a good rubric is not yet evidence that the skill passes it.

Context Mode’s benchmark processes 376 KB of real tool output into 16.5 KB of context, a 96% reduction. It checks retrieval, code-block preservation, execution, and integration behaviour. It does not compare whether an agent with Context Mode fixes more or fewer bugs than the same agent with raw tools.

RTK makes the scope of its claim unusually clear. Its documentation says “up to 90%” refers to shell-output bytes, not the total context window or the bill. Its token count is an approximation based on four bytes per token.

These are useful engineering measurements. None answers the user’s actual question:

Did the agent complete the work correctly, without creating a regression, and was it faster or cheaper overall?

Token count is easy to collect on every command. Correctness often requires hidden tests, human review, and waiting to see whether the bug returns. The easy metric is winning.

How to test your own plugin stack

The absence of a universal benchmark does not mean guessing. A small controlled test on your own work will tell you more than another project’s token dashboard.

Choose 20 to 30 representative tasks from one codebase: straightforward edits, test failures, ambiguous bugs, code review, and at least a few tasks where the first plausible fix is wrong. Run each under four conditions:

1. Baseline agent with no optimisation plugin
2. Output-style skill only
3. Tool or context compression only
4. Style and compression together

Keep the model version, reasoning effort, harness, permissions, and task prompt fixed. Repeat each condition more than once because agent runs are stochastic.

Then measure outcomes rather than just traffic:

• Tests passed, including tests the agent did not write
• Regressions or incorrect files changed
• Number of turns, retries, and tool calls
• Input, output, and reasoning tokens separately
• Wall-clock time and actual provider cost
• Warnings, assumptions, or failure modes omitted from the final report
• Cases where the agent had to bypass compression and request raw output

For filtering tools, save both the raw and transformed observation. When a run fails, you need to know whether the model reasoned badly from good evidence or reasoned reasonably from incomplete evidence.

The combined condition matters. Two plugins that look harmless alone can conflict when stacked. A context filter may hide detail while a brevity skill discourages the agent from explaining uncertainty or asking for more. Their effects are not necessarily additive.

Treat plugins as part of the runtime

These tools can be useful, but they are more consequential than their presentation suggests. Caveman can make a chatty model tolerable. I Have ADHD can turn a wall of prose into an actionable sequence. RTK can collapse thousands of irrelevant test lines. Context Mode’s idea of computing over data instead of pouring it into the prompt is sound.

But none of them is merely cosmetic. A persistent skill changes the agent’s policy. A shell proxy changes its observations. A context manager changes what it can retrieve and remember. Install any of them and you have changed the system you are evaluating.

Use aggressive compression on repetitive, recoverable output. Keep raw-output escape hatches for debugging, migrations, security reviews, and large diffs. Prefer visible truncation over silent omission. Most importantly, distrust any benchmark that reports tokens saved without reporting tasks completed.

A small transcript is useful only when it leaves enough evidence to verify the work.