In short. Replay labelled negative prompts and compare expected routing with the selected tool, arguments, and rendered component.
Definition: A negative prompt is a labelled test case where a ChatGPT app tool should not handle the request. An agency tests unwanted plugin activations by replaying these prompts and comparing the observed tool selection with the expected action.
Key facts
- OpenAI says ChatGPT and Codex decide when to call a tool from the metadata supplied for that tool, including its name, description, and parameter documentation, in its metadata guidance.
- The guidance places negative prompts in a labelled dataset alongside direct prompts and indirect prompts.
- Each prompt gets an expected behaviour: call the tool, do nothing, or use an alternative.
- During a test run, OpenAI says to record the selected tool, the arguments passed, and whether the component rendered.
- OpenAI's evaluation guidance identifies tool selection and data precision as separate evaluation areas for an agent that can choose tools.
What belongs in a negative prompt set?
A negative prompt is a request for which a built in tool or another tool should handle the task. It is one part of a golden prompt set. OpenAI's metadata guidance also names direct prompts, where a user explicitly names a product or data source, and indirect prompts, where the user describes an outcome without naming a tool.
For each record, label the expected behaviour before running the test. The documented choices are to call your tool, do nothing, or use an alternative. The label describes the intended routing outcome, so an observed call on a do nothing case is an unwanted activation.
{
"prompt": "<labelled user prompt>",
"expected_behavior": "call your tool | do nothing | use an alternative",
"selected_tool": "<observed tool>",
"arguments": "<observed arguments>",
"component_rendered": "<observed result>"
}
The field names in this compact record mirror the observations OpenAI lists for a golden prompt run. The prompt wording and label come from the agency's own reviewed test set.
How should the agency score unwanted activations?
OpenAI describes evals as structured tests for measuring model performance. Its evaluation process names an objective, a dataset, metrics, a comparison run, and continuous evaluation. For an agent, the guidance specifically maps the tools chosen by the model to tool selection tests, and maps the arguments sent to a tool to data precision tests.
The metadata guidance uses two measures for each prompt: precision asks whether the right tool ran, while recall asks whether the tool ran when it should. For negative prompts, review precision against the expected alternative or no call, then inspect the recorded arguments when a call was expected. OpenAI also recommends typical cases, edge cases, and adversarial cases in test data, with human expert labellers.
What should change after a failure?
Start with the tool description. OpenAI recommends beginning it with “Use this when…” and naming disallowed cases. It also recommends names that pair the domain with the action, parameter documentation with examples and allowed values, and annotations such as readOnlyHint, destructiveHint, and openWorldHint where they describe the tool's behaviour.
Change one metadata field at a time, keep a revision log with timestamps and test results, and repeat the evaluation after each revision. The guidance says to aim for high precision on negative prompts before pursuing marginal recall improvements. Weekly review of tool call analytics and periodic prompt replays are also part of its production monitoring advice.
What counts as an activation in the recorded run?
OpenAI's function calling documentation defines a tool call as a special response in which the model examines a prompt and determines that it needs one of the available tools. The documented flow starts with a request containing tools the model could call, then receives a tool call, executes application code, sends the tool output back, and receives a final response or more tool calls.
For a ChatGPT app test, the agency can therefore treat the selected tool as the primary activation result, the arguments as a separate correctness check, and component rendering as an additional observed outcome. The metadata guidance says to connect the custom MCP server in ChatGPT, install the resulting plugin, run the golden prompt set, and record these results.
Related terms
FAQ
What is the simplest negative test?
Use a reviewed user request whose expected behaviour is “do nothing” or “use an alternative”. Replay it with the plugin available, then record whether the tool was selected and what arguments were passed.
Should a negative prompt name the competing tool?
The label should identify the expected behaviour. OpenAI describes negative prompts as cases where built in tools or other tools should handle the request, so the record can name the alternative when the test requires that distinction.
How often should an agency replay the set?
OpenAI recommends periodic prompt replays, especially after adding tools or changing structured fields. It also recommends reviewing tool call analytics weekly after the MCP server is live.
Sources
- OpenAI, "Optimize Metadata", undated
- OpenAI, "Evaluation best practices", undated
- OpenAI, "Function calling", undated
Check how a page reads to answer engines. Open the GEO score analyzer