If you're evaluating miru for your coding agent, the question you actually care about is simple: does semantic search save money and time, or is it just a nicer interface on the same work? We ran a controlled benchmark to find out — 41 real questions about a large, well-known codebase, answered by Claude Code once using only its built-in file search (grep/glob) and once using only miru, three times each, 246 test runs in total.

The short version: on everyday code questions, miru is meaningfully cheaper and faster, at accuracy that's a wash. On one specific type of task — brute-force "find every instance of X" sweeps — the built-in search still wins, and we're not going to pretend otherwise.

• Up to 50% cheaper, 20% cheaper on average, per question answered

• Up to 60% faster, 11% faster on average, wall-clock per question

• Up to 45% fewer steps, 17% fewer on average, for the agent to reach an answer

• Zero hallucinated file citations from miru across 123 answers, versus 2 from the built-in search

We asked both tools 41 real questions about the VS Code codebase — a large, well-known open-source project — using the same underlying AI model (Claude Sonnet 5) both times, so the only variable was which search capability was available. Each question was asked 3 separate times per tool, for 246 test runs. The codebase was frozen at one fixed snapshot in time, so every run was looking at exactly the same code.

To keep the comparison clean, we forced Claude Code to rely exclusively on one search method or the other for each run: the baseline could only use standard grep-style file search, and miru could only use its own semantic search — no falling back to grep if it came up short. We verified this isolation directly through the AI provider's own logs, not just the model's self-report.

Most of the 41 questions (30 of them) are the kind of everyday question a developer actually asks — "where does X happen?", "how does Y work?" The rest are deliberately harder edge cases: 6 "find every instance of X" sweeps, and 5 "does this even exist" checks. The headline numbers above are based on the 30 everyday questions, since that's what real usage looks like day to day.

Every time Claude Code answers a question, it costs real money — the AI provider charges for what gets read and processed along the way. On the 30 everyday questions, miru brought the average cost down from $0.416 to $0.332 per question — a 20% reduction.

That average holds up when you look question by question, not just in aggregate: miru was cheaper on 23 of the 30 questions (77%). It's not a clean sweep — on 7 questions the built-in search actually came out cheaper, by a few cents — but the pattern is broad, not a mean propped up by a couple of outliers.

On the single best question in the set, miru came in 52% cheaper than the built-in search. That's the ceiling, not the typical case — the median saving across all 30 questions is closer to 14% — but it shows what's possible when miru's search lines up well with the question.

We measured two different things here, because they tell different stories. Wall-clock time is how long you'd actually wait for an answer: miru brought that down from 72.3 to 64.1 seconds on average, an 11% improvement, winning outright on 19 of the 30 questions (63%) and reaching 60% faster on the best single question.

The more consistent story is in the number of steps the agent needs to reach an answer — a proxy for how much it has to poke around before it finds what it's looking for. miru cut that from 14.9 to 12.3 steps on average (17% fewer), and won on 24 of the 30 questions (80%) — the most consistent advantage of any metric we measured, with a best case of 45% fewer steps on a single question.

Because accuracy is essentially tied between the two (more on that below), it's fair to ask how much each tool costs *per unit of correct answer* — not just which one is cheaper in isolation. On that combined measure, miru comes out ahead on cost, speed, and step-count alike, which is the clearest way we can show "same quality, less spent getting there."

We want to be upfront about this one: miru does not win on accuracy. Baseline scored 74.7% correct on the 30 everyday questions, miru scored 71.9% — a small gap that's within the noise of running each question only 3 times (we didn't run a formal statistical test at this sample size). Read this as roughly even, not a miru advantage.

One thing that does favour miru: across all 123 answers each tool produced, the built-in search fabricated a file citation that doesn't exist in the codebase twice; miru fabricated zero. That's a reliability signal worth knowing about even though it's separate from the raw accuracy score.

On the 6 "find every instance of X" questions — sweeps across 500+ matches in some cases — the built-in search is clearly more thorough than miru. We're showing this plainly rather than burying it:

Here's the thing, though: in practice, this either/or setup is stricter than how the tool will actually be used. We forced Claude Code to pick one search method exclusively, to keep this benchmark clean — but in real-world use, Claude Code is free to use miru and grep side by side, choosing whichever is best suited to the question at hand. So this result isn't a case against miru; it's simply the query type where Claude Code would reach for grep instead. Used this way, the two are complementary: miru for fast, targeted answers to "where/how does X work," and grep for exhaustive sweeps, giving you the benefit of both without giving up either one.

We'd rather you trust these numbers than be impressed by them, so here's what we're not claiming:

• This is one model (Claude Sonnet 5), one codebase (VS Code), and each question was only run 3 times — no formal significance testing. Treat the percentages here as a strong early signal, not a peer-reviewed result.

• Accuracy is a wash, not a miru win — every cost and speed number above should be read alongside "at comparable quality."

• The exhaustive-search result above is a real limitation for that specific task type, not spin. We believe that we can adapt miru to perform faster than grep on this use case and are actively implementing a specialized locate tool.

• 4 of the 246 test runs hit their step budget before finishing and were recorded as failures rather than scores — 2 shared between both tools on the hardest exhaustive-search questions, 2 one-off. Their effect on the cost numbers above is small and accounted for.

• This is one benchmark run, and we're planning a broader evaluation across more models and codebases next. Treat this as an early finding, not a final verdict.

Want the full technical writeup, including per-question data and methodology detail? Contact Us.