Benchmarking Guardrails for AI Agent Safety

Introduction

AI Agents extend large language models beyond text generation. They can call functions, access internal and external resources, perform deterministic operations, and even communicate with other agents. Yet, most existing guardrails weren’t built to protect these operations. OWASP and security researchers have been investigating the increased risk that agentic systems afford, which range from tool poisoning attacks to amplifying malfunctions in reasoning capabilities required for tool calling to indirect prompt injection attacks on the web or in vector databases , and many more.

This is very much related to the lethal trifecta , a way to think about how the risk surface evolves when giving LLMs access to tools.

Credit: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

That tool access can expose private data by linking to a vector database, potentially leaking company secrets or, worse, sensitive information on individuals. Allowing people to communicate with the system through natural language gives an avenue for new styles of attacks. And allowing information to come from external sources creates a great amount of untrusted content for your agentic system to handle. This trifecta showcases why it is so important to not only do whole system evaluations, but also to stress test each component on as many threats as possible.

Most guardrail models today are specifically trained to operate on inputs and outputs of agentic systems, and not their inner workings. This runs counter to the construction of agents. The LLMs governing agents may be black boxes, but that doesn’t mean the data flowing throughout an agentic system is a black box. To address this, we built any-guardrail to quickly test off-the-shelf, open-source guardrail models on some of the threats within agentic systems. Our findings: Some models show promise for prompt injection detection, but critical gaps remain in protecting function-calling operations.

In this post, we walk through how we evaluated certain guardrail models on out-of-distribution tasks, specifically indirect prompt injection attacks and detecting function-call malfunctions. We will discuss the guardrails selected, the benchmarks used, our setup and the results.

TL;DR

Our experiments found that PIGuard was an effective indirect prompt injection detection model against the email and tabular portions of the BIPIA dataset . Moreover, we found that detecting function call malfunctions is still a hard task for customizable judges to tackle, a gap in the field that we hope closes in the coming months. To run any of our experiments or analyze our data and results, please go to our GitHub Repo . And remember to try out any-guardrail !

Guardrails

We selected guardrails that could plausibly handle out-of-distribution tasks: detecting indirect prompt injection attacks and customizable judge models for function calling malfunctions. Although none of these models were purpose-built for these agent-specific risks, we selected specific guardrails supported by any-guardrail (as of the writing of this blog post) that are either purpose-built for adjacent tasks or are built to support a wide variety of judgement tasks.

The prompt injection detection models were all encoder-only models. This is most probably due to the fact that encoder-only models cannot be jailbroken, whereas decoder-only models are susceptible to jailbreaks and prompt injection attacks. Of course, these models still have potential privacy issues and may be susceptible to traditional adversarial attacks.

The customizable judge models (FlowJudge and GLIDER) are decoder-only models that have been fine-tuned on diverse judgement tasks with a rubric-based evaluation, making them promising candidates for judging function call correctness.

Benchmarks

Since we are benchmarking guardrails, not general LLMs or agents, our data has to be formatted differently. Instead of input-output pairs or an environment to check how an LLM uses tools, we need datasets that have complete conversations or examples with labels.

For indirect prompt injection attacks, this is relatively easier, as most prompt injection datasets contain labeled data points. However, for function calling, evaluation is typically done in an environment with known tools, where success is measured by whether an LLM executed the right tool tasks. Our challenge was to find datasets that could proxy both indirect prompt injection and function-call malfunctions, while still providing enough detail for evaluating guardrails.

Two datasets stood out. BIPIA is a benchmark for indirect prompt injection attacks embedded in realistic formats like emails, tables, news articles, etc. to simulate how such attacks appear in external resources. Since BIPIA only contains attacks, we augmented it with benign data points from WildGuardMix to test whether guardrails could distinguish between malicious and harmless prompts. You can see the data here . Details of the data we used are below:

Note, BIPIA requires you to download the data yourself. For email and tabular, this is quite easy and only requires some editing of the BIPIA scripts. However, for TextQA and Summarization it required wrangling some docker images and older code with dependencies on other dataset loaders. To make sure we were using the correct datasets for benchmarking, we only utilized the email and tabular test sets, since we had limited issues in generating them. To maintain balance, our WildGuardMix sample was drawn to roughly match the size of the BIPIA email subset.

For function calling, HammerBench best met our needs. It contains user, agent conversations, the full tool list available to the agent, the tool selected by the agent, and a label mapping each function call as either working or faulty, based on their taxonomy. The taxonomy is expressed below:

We specifically use single turn because the labels were explicitly expressive of the types of issues that can occur in their taxonomy. This resulted in 13054 data points, evenly split amongst:

Any data point not labeled Perfect (and relevant) was labeled as a malfunction.

Experiment Results

For each experiment, we used our any-guardrail labeling schema, where a data point is valid if a guardrail deems it benign, and invalid otherwise. As a result, each task became a binary classification task. This approach allowed us to do an apples-to-apples comparison across different datasets.

Indirect Prompt Injection Attack

We break this section into two: one assesses guardrails on the BIPIA email dataset, whereas the other is assessed on the BIPIA table dataset. Note that both use the same benign data points from WildGuardMix.

Email Results

We notice three broad patterns:

Table Results

We see a similar pattern as we do in the email dataset: some models overtriggered, some struggled, and some models achieved balanced performance, and only one performed consistently well on both datasets: PIGuard. This suggests that PIGuard is a strong guardrail against indirect prompt injection attacks, while also being permissive enough to allow benign prompts through.

Function Calling

We evaluated GLIDER and FlowJudge, highly customizable judge models that have been fine-tuned to judge a variety of tasks effectively. Each requires passing criteria and a rubric in order to accomplish their tasks. Here is the exact prompt used for both models:

For FlowJudge, we also incorporated 3 examples of how to judge the user conversations and associated function calls. When doing so with GLIDER it only judged the example prompts, not the user conversations and function calls to be judged. The results are below:

We notice that both models struggled extremely on this task. This did not change when we looked at FlowJudge with few-shot prompting either. However, we did notice that on multiple runs, we were receiving quite different results when using the default parameters for FlowJudge. Thus, we also wanted to see the agreement between the FlowJudge few-shot results.

The few shots did increase the capabilities of the FlowJudge model, which is promising, but the scores are still fairly low and wouldn’t provide much confidence in judging production function calling. Even more concerning are the Cohen Kappa scores, which measure the pairwise agreement between the judgements from each FlowJudge run. A score of 0.26 is considered a fair agreement , which is not ideal when considering the reliability of the FlowJudge results.

When looking qualitatively, the results become even more interesting. Each FlowJudge judgement contained an explanation and a score. The score was from the rubric, but the explanation is a detailed analysis of why the judge chose the score.

Below are three explanations and scores from FlowJudge on the same data point. Note that the data point should have been given a score of 2 (i.e. the function call being made was Perfect).

Let’s look at each sample. In the first sample, we see that the explanation says that all criteria have been met, but then hallucinates that not enough parameters were given for the function call, thus resulting in a score of 1. In the second sample, the explanation identifies that the tool exists in the tool list, but then says it doesn’t exist in its conclusion. Even so, the second sample gives a score of 2. In the third sample, the explanation says that not enough parameters have been provided. But we know from the prior two runs, and the fact that this is a sample that should be awarded a 2, that there are enough parameters.

In each case, we see that FlowJudge provided inconsistent explanations and corresponding scores. This inconsistency makes it an untrustworthy judge for this use case. Note, however, that the results we find here are not absolute judgements on these judge models. As was discussed at the beginning of this post, the task of judging function call malfunctions is out of distribution of these judges. In other words, these judges were not meant to judge function calls. But that is exactly why we wanted to perform this assessment. We wanted to see whether out-of-the-box judges, such as these, could perform out-of-distribution tasks.

Conclusion

This exploration examined whether off-the-shelf, open source guardrails could defend against internal agentic safety issues, specifically indirect prompt injection attacks or function-calling malfunctions.

However, this does not mean that no models exist or that the field isn’t catching up. Models like Granite Guardian ’s latest iteration and Sentinel v2 (released after our evaluations) show signs of progress. The space is continuing to move and change at a rapid speed, which is why a framework like any-guardrail is important for standardized, reproducible benchmarking.

Moreover, guardrail models are only one part of a greater ecosystem that should be employed to create safe agents. Each component within an agentic system presents a new risk surface that needs detection and mitigation mechanisms. By creating a framework that allows researchers and practitioners to quickly switch between guardrails, assessing those components, with and without mitigations, becomes easier. That is what any-guardrail attempts to do. Try out any-guardrail today, and help us make AI agents safer for everyone.

What did you think?

If you liked this project, feel free to check out the GitHub repo to replicate our work!

This is a product of our Builders in Residence program. If interested, check it out here and apply!

Content version: 699776cfff0f