An evaluation of MCP tool choice · 12 experiments run · 3 models
Which tool does the agent call?
We trained an instrument on 82 tools from real MCP servers and recorded where each decision landed. Every point in this sky is a real tool. Every beam is a recorded call.
Before the data · how an agent picks a tool
An agent never sees your tools. It sees their descriptions, all at once.
browser_click: Perform click on a web page
list_pages: Get a list of pages open in the browser.
browser_snapshot: Capture accessibility snapshot…
… 28 more, from two serversI · The request
Before it reads your message, the model reads every tool you installed.
II · What didn't matter
When each tool has one clear job, almost nothing moves the choice.
Six interventions, each preregistered. A dot is the estimate, the bar its 95% interval, counted over families of cases. Each tile has its own scale.
What does “95% interval, clustered over families” mean?
A family is one request and all its variants (orderings, wordings). Variants of one request are not independent: if the model is confused by a request, it tends to be confused in every ordering. So we resample whole families, recompute the number thousands of times, and report the middle 95% of those results. Chapter VIII lets you watch it happen.
These surfaces were written by us, with one distinct job per tool. Chapter IV asks what happens on real servers.
III · Ties
When four tools do the same job, position breaks the tie.
The rack: four near-synonym tools, re-ordered through all 24 recorded orderings. Each beam is one of 8 requests' recorded calls. The strip under the rack fills in how often the right tool was picked, by the slot it sat in.
results/exp001_siblings_t0_analysis.json · exp014_analysis.jsonThese four tools are synonyms, so "right" is the keyword gold of each request; this measures tie-breaking, not an error rate.
IV · Real servers
On real servers, misses land on look-alikes.
Threads join a wrong call to the right tool; brighter means more trials made that mistake. Most join near neighbours: tools whose descriptions read alike.
results/exp013_analysis.json · exp013_h4.json48 families, three open models, one provider, first call only.
V · Who gets the call
When servers overlap, the model's first move follows habit.
Two waves of beams: the same 12 requests, once per ordering of the servers. The bars count where every first call landed, by server.
results/exp013_120b.jsonCounts of recorded first calls. A multi-step agent might still finish the job; this measures the first action, not the outcome.
VI · The lab
Point the instrument yourself.
Pick a model, a set of servers and a request. The beam is the recorded first call. Switch which server is listed first to compare orderings. Drag the open sky to orbit.
VII · The plate archive
Every experiment, including the ones that found nothing.
Fourteen preregistered experiments. Each plate has its question, sample, verdict, key number, source file and what it taught us. Nulls and experiments we chose not to run are kept beside the effects.
VIII · How sure we are
Where the intervals come from, and why one run is never enough.
The bootstrap, live simulated · resampling recorded families in your browser
The synonym test has 8 request families. We draw 8 families with replacement, recompute the first − last third gap, and repeat. The spread of those gaps is the interval. Repeated trials of one family are never counted as independent.
Seeing conditions
Astronomers call a steady sky good seeing. We asked the same question 5 times at temperature 0. Each row is one request and ordering; each dot one answer, coloured by tool.
results/determinism_probe.jsonWhy we fix temperature, record every answer and count over families.
IX · Your server
Point the instrument at your own tools.
Drop a tools/list JSON (or a surfacelab snapshot). Your tools join the sky, placed by how alike their descriptions read, with the token cost and the pairs most likely to be confused. The file never leaves your browser.
Pairs that read most alike
Load a file to see them.
Similarity here is lexical (TF-IDF over description words), computed in your browser; the sky above uses an embedding model. To measure real confusion, run the CLI below on your server.
X · Test this page · EXP-008
Does the 3D sky actually help? Help us measure it.
Two short tasks, about a minute. In one you use the sky, in the other a plain table; which comes first is decided for you. We record whether you answered right and how long it took. Nothing else is stored, and the result is published whatever it shows.
Preregistered in docs/experiments/EXP-008.md. Exploratory: a small sample of peers, not generalisable. Answers are stored per person; only the page owner can read them.
Coda · What it means
Tools fail where their descriptions blur together.
results/exp013_analysis.json · docs/FINDINGS.mdRun it yourself
Source, data and methods on GitHub. Every number on this page traces to a file in results/ (see docs/FINDINGS.md). Star positions are a 3D projection (pca) of description embeddings (text-embedding-3-small): similarity, not confusion. The sky, horizon and dome are scenery and carry no data.