Does the prompt match the tools?
I build LLM agents. Every request an agent sends has the same parts: instructions, a list of tools and the conversation. The instructions often name the tools: “use mem_ssh to connect to Tower.”
It works until a tool gets a new name.
The bug no test catches
This one is from a post by Pete. The agent’s instructions said “use mem_ssh to connect to Tower.” The tool came from an MCP server called mem, and in the schema the model actually sees it was mcp__mem__mem_ssh. The model called mem_ssh, got “Error: unknown tool”, and told the user that SSH was “currently unavailable”.
Nothing crashed. The instructions are a string, the tools are a list, and each one is fine on its own. What breaks is the link between them, and no unit test looks at it.
This is not rare. OpenHands once shipped a tool description that said “prefer using the webpage_read tool”. The tool was called web_read (fixed in #5902). I collected 43 fixed cases like these from the history of 26 projects.
What I built
agent-handshake reads a request as it is sent to the model, in OpenAI Chat Completions, OpenAI Responses or Anthropic Messages format, and checks that the text matches the tools:
pip install --pre agent-handshake
handshake check request.json
3 rules enabled. Checked request.json (Anthropic Messages): 1 tools, 2 text segments.
✗ HS002 request.json (system): Text names `mem_ssh`; no tool has that exact name. Did you mean `mcp__mem__mem_ssh`?
1 findings (1 errors, 0 warnings).
Three rules are on by default:
- HS001 (warning): the text names a tool in a form that can only be a tool, such as
functions.xor a tool-call example, and the request has no such tool. - HS002 (error): a tool name that is almost right: different case,
-instead of_, a missing prefix or suffix. - HS003 (error): an argument that the tool’s schema does not have.
It makes no LLM calls and needs no network. Errors make the command exit with code 1, so it can fail a CI job. Warnings do not, unless you pass --strict. Rules can be turned off and names ignored in handshake.yaml or pyproject.toml.
How well it works
I wrote the expected results down before writing any rule, and I measured the rules on projects I did not use to build them. The configuration was frozen first, and the benchmark was run once:
| projects used to build the rules | holdout | holdout 2 | |
|---|---|---|---|
| real cases caught | 8 of 11 | 4 of 11 | 2 of 2 |
| HS002 findings that were right | 11 of 11 | none found | 5 of 5 |
| HS001 findings that were right | 13 of 14 | 0 of 24 | none found |
HS002 holds up. HS001 does not yet: on the holdout all 24 of its findings came from one project that declares functions as tools.x({ ... }), which HS001 reads as calls. That is why HS001 is only a warning.
What it does not do yet
- It checks a request file. You have to get that file yourself, for example by logging the request your agent sends.
- HS001 can warn about names that are not tools: function declarations like
tools.x({ ... }), negative forms (“never callapply_patch, it does not exist”) and placeholders liketool_name.
Try it
pip install --pre agent-handshake
--pre is needed while 1.0 is in beta. The README has the rules and the full benchmark. If your agent’s instructions and tools have drifted apart, I would like to hear about it.