Does the prompt match the tools?

3 min read

I build LLM agents. Every request an agent sends has the same parts: instructions, a list of tools and the conversation. The instructions often name the tools: “use mem_ssh to connect to Tower.”

It works until a tool gets a new name.

The bug no test catches

This one is from a post by Pete. The agent’s instructions said “use mem_ssh to connect to Tower.” The tool came from an MCP server called mem, and in the schema the model actually sees it was mcp__mem__mem_ssh. The model called mem_ssh, got “Error: unknown tool”, and told the user that SSH was “currently unavailable”.

Nothing crashed. The instructions are a string, the tools are a list, and each one is fine on its own. What breaks is the link between them, and no unit test looks at it.

This is not rare. OpenHands once shipped a tool description that said “prefer using the webpage_read tool”. The tool was called web_read (fixed in #5902). I collected 43 fixed cases like these from the history of 26 projects.

What I built

agent-handshake reads a request as it is sent to the model, in OpenAI Chat Completions, OpenAI Responses or Anthropic Messages format, and checks that the text matches the tools:

pip install --pre agent-handshake
handshake check request.json
3 rules enabled. Checked request.json (Anthropic Messages): 1 tools, 2 text segments.
✗ HS002 request.json (system): Text names `mem_ssh`; no tool has that exact name. Did you mean `mcp__mem__mem_ssh`?
1 findings (1 errors, 0 warnings).

Three rules are on by default:

  • HS001 (warning): the text names a tool in a form that can only be a tool, such as functions.x or a tool-call example, and the request has no such tool.
  • HS002 (error): a tool name that is almost right: different case, - instead of _, a missing prefix or suffix.
  • HS003 (error): an argument that the tool’s schema does not have.

It makes no LLM calls and needs no network. Errors make the command exit with code 1, so it can fail a CI job. Warnings do not, unless you pass --strict. Rules can be turned off and names ignored in handshake.yaml or pyproject.toml.

How well it works

I wrote the expected results down before writing any rule, and I measured the rules on projects I did not use to build them. The configuration was frozen first, and the benchmark was run once:

projects used to build the rules holdout holdout 2
real cases caught 8 of 11 4 of 11 2 of 2
HS002 findings that were right 11 of 11 none found 5 of 5
HS001 findings that were right 13 of 14 0 of 24 none found

HS002 holds up. HS001 does not yet: on the holdout all 24 of its findings came from one project that declares functions as tools.x({ ... }), which HS001 reads as calls. That is why HS001 is only a warning.

What it does not do yet

  • It checks a request file. You have to get that file yourself, for example by logging the request your agent sends.
  • HS001 can warn about names that are not tools: function declarations like tools.x({ ... }), negative forms (“never call apply_patch, it does not exist”) and placeholders like tool_name.

Try it

pip install --pre agent-handshake

--pre is needed while 1.0 is in beta. The README has the rules and the full benchmark. If your agent’s instructions and tools have drifted apart, I would like to hear about it.