Adversarial Attacks on Models - Bring Your Own Ideas for Red-Teaming LLMs

What is needed to make a language model do something it shouldn't? Bring your own idea for attacking LLMs, and work with us to turn it into a research project that puts their security to the test.

A language model's behavior depends on more than the question you ask it. It also depends on the examples it learned from and the information it retrieves. Each gives an attacker a different place to interfere: change the request, tamper with the training data, or influence what the model finds when it looks for an answer.

This is an open topic for students who want to explore how LLMs can be manipulated and what those failures tell us about their security. You bring the curiosity and, if you have one, an initial idea. Together with your supervisors, you turn it into a focused research question. Possible starting points include: 1) Jailbreaking and indirect prompt injection. Explore new ways to bypass a model's safeguards or redirect it through instructions hidden in material it reads. Which properties make an attack effective, and does it transfer to other models or settings? 2) Poisoning preference training. Investigate what happens when an attacker tampers with the response comparisons used in Direct Preference Optimization (DPO). Can small changes produce a targeted behavior without noticeably reducing the model's ordinary usefulness? 3) Attacking retrieval through SEO. Explore whether search engine optimization can make attacker-controlled pages more likely to reach an LLM and influence its answers. Getting a page retrieved and getting its content trusted are separate problems; either could be the focus.

These are suggestions, not a checklist. We welcome your own ideas on helping us understand or improve the security of LLMs. You will choose one manageable problem, with models, data, and experiments suited to the time and computing resources available. You do not need a finished proposal before getting in touch.

Goals

The goal is to develop and investigate your own research question about attacking LLMs, supported by reproducible experiments. The project takes shape in four stages:

a) Explore and choose. Read the work closest to your interests and discuss possible gaps with your supervisors. Choose a direction and define what the attacker can control, what they want to achieve, and what would count as success. Your idea can guide the literature review from the start.

b) Establish a starting point. Reproduce a relevant attack or build a small baseline experiment. Check that you can distinguish an attack's effect from ordinary model errors. Use this stage to test feasibility and refine the question before committing to a larger study.

c) Develop your contribution. Follow the question where the evidence leads. You might design a new attack, test an overlooked assumption in an existing one, explain why a technique works, or develop a more realistic way to evaluate it. The contribution is agreed with your supervisors and can adapt as you learn more.

d) Evaluate and explain. Compare against relevant baselines and test the conditions under which your findings hold. Depending on the direction, measure attack success, attacker effort, transfer between models, or the effect on legitimate tasks. For retrieval attacks, distinguish whether the content is found from whether it changes the answer. Explain the limitations and what your results imply for defenses; implementing a defense is optional.

A carefully explained failure of an attack can be a useful result too. The expected outcome is a focused empirical contribution, with code and experimental materials that make the findings reproducible.

Learning outcomes

• hands-on experience with LLM red-teaming and adversarial machine learning
• experience turning an open-ended idea into a clear, feasible research question
• proficiency in implementing attacks and designing controlled, reproducible experiments
• understanding of the security assumptions behind the models, training methods, or retrieval systems you investigate
• experience with responsible offensive-security research and communicating its findings
• experience working in an exciting and active research environment, with opportunities to develop your results into a scientific paper

Qualifications

• interested in AI security, LLMs, and adversarial machine learning
• comfortable programming in Python and willing to learn the relevant LLM tools
• curious about unexpected model behavior and willing to investigate why an idea does or does not work
• some familiarity with machine learning is useful; experience with deep learning and model training is particularly helpful for the DPO direction
• experience with information retrieval or web search helps for the SEO direction, but is not required for other directions

Supervisors

• Leon Moonen
• Birk Torpmann-Hagen
• Fernando Ruiz
• Annika Olstad
• Finn Schwall

References

Possible starting points; the reading list will follow your chosen direction:

• Greshake et al. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.
• Pathmanathan et al. (2025). Is Poisoning a Real Threat to DPO? Maybe More So Than You Think.
• Zou et al. (2025). PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. A starting point for retrieval poisoning; web search and SEO introduce additional questions.