Safety evaluation of language models for Norwegian deployment
Published:
Researchers from Simula and SimulaMet have published a new report assessing the safety of large language models when used in Norwegian societal and public-sector contexts.
The report is authored by Michael A. Riegler, Sushant Gautam, and Klas H. Pettersen. It presents a comparative evaluation of five language models, including the Norwegian-language model Borealis 4B developed by the National Library of Norway, several international open-source models, and OpenAI’s GPT-5 as a reference model. The evaluation is based on 36 Norwegian-specific scenarios covering areas such as healthcare, legal and regulatory questions, public services, and institutional information.
Using the researchers’ SimpleAudit evaluation framework, the study focuses on safety-related behavior that is not captured by standard language benchmarks, including how models handle harmful requests, maintain appropriate boundaries, and provide accurate institutional information. The report also examines the impact of retrieval-augmented generation (RAG) and different model precision settings on safety outcomes. A key finding in the report is the 'Understanding Paradox': researchers observed an inverse relationship between standard benchmark rankings and safety outcomes. Surprisingly, some smaller models with high language understanding scores exhibited the most critical safety failures, challenging the assumption that 'smarter' models are inherently safer.
The full report, including methodology, results, and recommendations, is publicly available and intended as a contribution to ongoing discussions about responsible deployment of large language models in Norwegian and international settings.
References
Safety Evaluation of Language Models for Norwegian Deployment
SimpleAudit evaluation framework on Github
Image: Michael Riegler presenting on AI security at a Digital Norway networking event at Simula (photo: Simula).