Adversarial Attacks on Agents - Bring Your Own Ideas for Red-Teaming AI Agents

What is needed to make an AI agent act against its user's intentions? Bring your own idea for attacking agentic systems, and work with us to turn it into a research project that puts their security to the test.

An AI agent can read a repository, edit code, run tools, and pass work to another agent. Along the way, it encounters material the user may never have checked: a comment in a file, a tool's response, or a message from a supposed collaborator. What happens if an attacker controls part of that material? Can they redirect one action, influence a whole team, or make the attack travel further on its own?

This open topic invites you to bring your own ideas for attacking and red-teaming systems built around LLM agents. The focus is on how agents act, use tools, retain information, and interact. Possible starting points include: 1) Attacks on coding agents. Investigate whether material in a repository, issue, or tool response can steer an agent into making an insecure change or taking an action outside the user's request. How do the surrounding workflow and permissions affect the outcome. 2) Social contagion between agents. Explore whether an attacker can seed a false claim or unsafe instruction that other agents repeat, reinforce, and eventually act on. Does repetition make it seem more credible? Do particular roles or communication patterns amplify the effect? 3) Worms in agentic systems. Study when an injected instruction can cause an agent to reproduce and forward it, exposing further agents through messages or shared artifacts. What lets propagation continue, and what breaks the chain?

These are suggestions, not a checklist. We welcome your own ideas on helping us understand or improve the security of agentic systems. You will choose one manageable problem, with systems, tasks, and experiments suited to the time and computing resources available. You do not need a finished proposal before getting in touch. Experiments will use controlled environments, with propagation studies confined to isolated test systems.

Goals

The goal is to produce a reproducible study of a security weakness in an agentic system, showing how it arises, what it enables, and where its limits lie. The exact question is yours to develop with your supervisors.

a) Explore and scope. Read relevant work and choose a system or interaction that interests you. Define the legitimate task, the attacker's objective, what they can access or alter, and which boundaries the attack would need to cross. Your own idea can be the starting point.

b) Build a small test setting. Reproduce a relevant attack or establish a baseline in a controlled workflow. Record messages, tool calls, and changes to the environment so that you can trace what actually happened. Verify that the system can complete the legitimate task without the attack.

c) Pursue your question. Develop an attack, investigate a propagation mechanism, or test an assumption about how agents trust information. For example, vary how a coding agent encounters a malicious instruction, whether agents check claims from their peers, or how shared memory affects the persistence of an attack. Choose the experiments that answer your question; covering all these examples is not expected.

d) Evaluate the consequences. Measure whether the attack causes the intended action, how reliably it works, and how much control the attacker needs. For contagion or worms, also examine how far the effect spreads and whether it persists. Separate an agent repeating an instruction from actually following it, and distinguish attack-induced failures from ordinary task errors. Use suitable comparisons to explain which parts of the system matter.

The main outcome is a focused empirical contribution supported by a prototype or evaluation setup. Testing a mitigation is a possible extension. A clear account of why an attack fails, or why a proposed risk only appears under narrow conditions, can also make a valuable thesis.

Learning outcomes

• hands-on experience with AI agent security and offensive-security research
• understanding of how tools, memory, permissions, and communication shape an agent's attack surface
• experience turning your own idea into a focused research question and experimental study
• proficiency in building controlled agent workflows and analyzing execution traces
• experience evaluating both attack outcomes and the system's ability to complete legitimate tasks
• experience working in an exciting and active research environment, with opportunities to develop your results into a scientific paper

Qualifications

• interested in AI security, autonomous agents, and how software systems fail
• comfortable programming in Python; experience with APIs, development tools, and debugging helps
• willing to build experimental systems and investigate behavior across several interacting components
• familiarity with LLM tools or agent frameworks is useful, but can be developed during the project
• experience with software security helps for coding-agent attacks; interest in distributed systems or information propagation helps for the multi-agent directions

Supervisors

• Leon Moonen
• Birk Torpmann-Hagen
• Fernando Ruiz
• Annika Olstad
• Finn Schwall

References

Possible starting points; the reading list will follow your chosen direction:

• CodeIPI: Indirect Prompt Injection for Coding Agents. An example evaluation setup for attacks embedded in software engineering artifacts.
• Papadopoulos et al. (2026). Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems. Studies how ideas and goals spread by inducing agents to pass them on, including in collaborating coding teams, and which factors and defenses limit their propagation.
• Xu et al. (2026). When Agents "Misremember" Collectively: Exploring the Mandela Effect in LLM-based Multi-Agent Systems. Background on how false information can be reinforced through agent interaction.
• Cohen, Bitton, and Nassi (2024). Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications. A starting point for studying self-replicating prompt injections.