A Payload in Plain Sight - Stegomalware in LLMs and AI Workflows

A model checkpoint can hide more than learned knowledge: its weights can also carry a concealed payload. Investigate when that hidden data becomes a practical security threat, and whether an AI assistant can be persuaded to unpack it.

Download a model, load its weights, and start working. Those weights look like numbers, but they can also encode a hidden payload. This is neural network stegomalware: malicious content concealed in model parameters, potentially with little effect on the model's usual performance. Some embedding methods can survive further training or reduced numerical precision, making persistence another concern.

Safe loading formats such as safetensors prevent the code execution associated with pickle deserialization. They do not, by themselves, remove information hidden in tensor values. For that information to become an active threat, something still needs to extract it and use it. Understanding that path is central to this project.

You will investigate practical exploitation scenarios, starting with those described in the literature and then examining modern AI pipelines. Could a coding agent or an assistant helping a developer inspect a model be persuaded to decode a supposed "watermark" and treat its contents as instructions or executable material? Which steps would require attacker control, unsafe tooling, or explicit user approval?

The project combines a reproducible baseline with one deeper investigation: detecting concealed payloads, studying whether payloads can be conveyed through model outputs in a way analogous to watermarking, or examining agent-assisted extraction. Recovering data directly from weights and eliciting it through generated text are different mechanisms; the latter is an exploratory direction whose feasibility must be established.

The outcome should explain when stegomalware creates a realistic risk and which assumptions make an attack possible. Demonstrations will use harmless substitute payloads in an isolated environment, allowing the full sequence of actions to be studied without deploying malware.

Goals

a) Map the threat. Review stegomalware attacks and defenses, separating payload storage, extraction, and execution. Catalogue exploitation scenarios from the literature, then identify plausible opportunities in model conversion, inspection, fine-tuning, and coding-assistant workflows. Record the access and user actions each scenario requires.

b) Reproduce a baseline. Use an existing embedding method with a harmless payload and a manageable model. Measure recovery and model utility before and after selected transformations, such as fine-tuning or quantization. Test a permutation-based defense where the architecture permits it. Distinguish detecting a payload from corrupting it without detection.

c) Choose one investigation. Agree a focused direction with your supervisors:

• Detection. Test whether weight statistics or other signals distinguish embedded payloads from ordinary model variation, including benign watermarks. Measure false alarms as well as detection rates.
• Output-space stegomalware. Test whether a model can emit a recoverable encoded payload through its outputs, and assess capacity, reliability, and effects on normal generation.

• Agent-assisted extraction. Test a realistic inspection or development task in which an agent encounters attacker-controlled material, for example a claimed watermark. Trace whether it merely decodes data or proceeds to interpret or execute it.

d) Assess practical impact. Connect the chosen investigation to at least one concrete workflow. Compare poisoned and clean versions, document each necessary step, and test where a suitable safeguard interrupts the sequence. Explain whether the attacker needs capabilities that would already permit a simpler attack, and what hiding the payload actually adds.

The outcome is a reproducible prototype and an account of feasibility and limitations. Universal detection or successful exploitation of every scenario is not required.

Learning outcomes

• hands-on experience with neural network steganography and AI supply-chain security
• understanding of the distinction between hidden data, unsafe model loading, and payload execution
• proficiency in designing experiments on payload recovery, model utility, and detection reliability
• experience assessing realistic attacker capabilities and tracing agent and developer workflows
• experience with responsible offensive-security research and reproducible demonstrations
• experience working in an exciting and active research environment, with opportunities to develop your results into a scientific paper

Qualifications

• interested in AI security, adversarial machine learning, and software supply-chain security
• comfortable programming in Python; experience with PyTorch and model checkpoints is useful
• familiar with basic deep learning concepts and willing to investigate numerical model representations
• experience with LLM tools or coding agents helps for the agent-assisted direction
• an interest in steganography, statistical analysis, or model watermarking is useful for the detection and output-space directions

Supervisors

• Leon Moonen
• Birk Torpmann-Hagen

References

• Hitaj et al. (2024; revised 2025). Do You Trust Your Model? Emerging Malware Threats in the Deep Learning Ecosystem.
  Introduces MaleficNet 2.0, which embeds payloads in network parameters, and demonstrates an extraction and execution mechanism in a machine learning framework.
• Torpmann-Hagen et al. (2025). Defending against Stegomalware in Deep Neural Networks with Permutation Symmetry.
  Studies how function-preserving permutations can disrupt embedded payloads without reducing network accuracy.
• Gilkarov and Dubin (2025). NeuPerm: Disrupting Malware Hidden in Neural Network Parameters by Leveraging Permutation Symmetry.
  Evaluates permutation-based disruption of stegomalware, including experiments on LLMs.
• Hugging Face. Safetensors.
  Documentation for a tensor storage format designed for safe loading; useful for distinguishing serialization risks from payloads concealed in tensor data.