Masters-projects
Local AI Desktop Agents for the Visually Impaired

Local AI Desktop Agents for the Visually Impaired

Are you keen on using your technical background for helping the visually impaired? Then this project is for you. This Master’s thesis project develops a privacy-preserving, locally deployed AI agent that integrates open-source web automation with Norwegian Speech-to-Text models to enable voice-driven digital interaction. Conducted in collaboration with Blindeforbundet, the study will empirically evaluate system performance, non-visual audio feedback loops, and real-world task success rates across Norwegian web applications.

Being a master’s student
List of projects
Register as a master’s student

This Master project addresses the following research question: can locally deployed AI desktop agents be employed to assist visually impaired individuals in Norwegian to accomplish web navigation and interaction tasks?

Background

Navigating digital interfaces presents significant accessibility challenges for visually impaired users. Traditional screen-reader approaches can be cumbersome and unintuitive when synthesizing complex application layouts (Stragier et al., 2023). However, recent advancements in Computer-Using Agents (CUAs) offer new interaction paradigms. Open-source frameworks like 'Browser Use' (Müller and Žunič, 2024) now empower AI agents to navigate and interact with web browsers, effectively bridging the gap between natural language prompts and autonomous DOM-based web automation. Yet, creating a fully accessible experience requires high-quality localized voice inputs/outputs. For the Norwegian context, open-source Norwegian Speech-to-Text (STT) models (e.g. by Nasjonalbiblioteket), provide the essential foundation for robust, voice-driven AI assistance in Norwegian.

Limitations

Current AI desktop agents exhibit significant technical and architectural limitations. Notably, heavy reliance on large proprietary models (e.g., GPT-4o, Claude) raises substantial privacy concerns. Furthermore, cloud-based vision processing incurs high latency and significant energy consumption compared to parsing localized accessibility trees on-device.

Beyond infrastructure issues, current open-source agents suffer from key algorithmic and accessibility bottlenecks. Models frequently drop components of multi-step instructions mid-task and lose context during transient subwindow or pop-up transitions (Ram et al., 2026). Semantic parsing also breaks down when encountering noisy accessibility trees, unlabelled DOM elements, or custom web canvases (Liu, Y., et al., 2026). Crucially, existing platforms lack low-latency non-visual verification mechanisms, preventing visually impaired users from auditing or interrupting intended agent actions before destructive commands are executed (Mohanbabu et al. 2026).

Knowledge Gap

While AI computer-use agents are increasingly researched in English, there is a lack of empirical evidence regarding their performance in Norwegian. Specifically, the integration of Norwegian STT models with open-source DOM-parsing web agents remains unexplored. Furthermore, it is unknown how accurately locally deployed, smaller-parameter models can handle complex web applications and website tasks (e.g., navigating Norwegian public service portals) while maintaining non-visual verification constraints for visually impaired users.

To address these gaps, this project aims to:

  1. Understand the state of the art: Conduct a comprehensive literature and technical review of current open-source AI computer-use agents and conversational web-accessibility architectures.
  2. Map limitations: Evaluate the constraints of local web-interaction frameworks (e.g., Browser Use) regarding task accuracy, privacy preservation, and local latency.
  3. Build a prototype: Develop a locally deployed desktop assistant featuring a Norwegian voice interface (leveraging the National Library Lab's STT models) that executes web navigation tasks securely on-device.
  4. Test the prototype on users: Conduct early user testing in collaboration with Blindeforbundet (the Norwegian Association of the Blind and Partially Sighted) to evaluate real-world usability, task success rates, and the effectiveness of non-visual audio feedback loops.

The technical focus of this Master’s thesis project will conform to the academic specialization of the Master student. Specifically:

  • Data Science  -  Development of an accessibility benchmark of web navigation and interaction for the visually impaired.    
  • Computer Science - Integration of low-latency DOM-tree scraping methods with high-latency browser action models to reduce the time of a predefined set of tasks for the visually impaired.
  • Social Science - Study of human-agent interaction patterns and modalities to present information to the visually impaired.

Goal

The primary goal of this thesis is to evaluate the viability of locally deployed, voice-driven AI desktop agents in advancing digital accessibility for visually impaired individuals in Norway. By constructing a privacy-preserving prototype that integrates local Norwegian speech recognition with open-source browser automation, the research empirically measures agent execution performance (accuracy, latency, and non-visual verification loops). Through real-world user testing with Blindeforbundet the project will identify human-agent interaction patterns and modalities to present information to the visually impaired.

Learning outcomes

  • Deployment of open-source local LLMs on-device using inference engines (e.g., Ollama, vLLM, llama.cpp).
  • Experience implementing Speech-to-Text (STT) pipelines using Hugging Face transformers or PyTorch (specifically deploying localized models like NB-Whisper).
  • Hands-on experience with open-source agent integration (e.g., Browser Use).
  • Knowledge of operating system accessibility APIs (e.g., HTML DOM structure,).

Qualifications

  • Proficiency in Python 
  • Fluency in Norwegian (Bokmål/Nynorsk) to evaluate speech recognition accuracy, assess localized prompt responses, and directly facilitate user testing sessions with Blindeforbundet.
  • Preferred, but can be picked up during the thesis:
    • Knowledge on Human-Computer-Interaction Design

Supervisors

  • Claudia Battistin
  • Omar Richardson
  • Thomas Roehr

Collaboration partners

  • Blindeforbundet

References

  1. Kodandaram, S. R., et al. (2025). Screen Reader AI: A Conversational Web-Accessibility Assistant for Enabling Uniform Computer Interaction. DOI: 10.1145/3706598.3713634
  2. Müller, M. and Žunič, G. (2024). Browser Use: Enable AI to control your browser. GitHub. https://github.com/browser-use/browser-use. 
  1. Stragier, V., Seddati, O., & Dutoit, T. (2023). Developing an Interactive Agent for Blind and Visually Impaired People.
  2. Liu, Y., et al. (2026). Tactile: Giving Computer-Using Agents Hands and Feet. arXiv.org. https://arxiv.org/abs/2607.14443
  3. Ram, K.S., et al. (2026). Are We There Yet? Assessing Computer-Use Agents for Blind Users” Accessible Interaction with Desktop Applications. [online] arXiv.org. Available at: https://arxiv.org/abs/2609.00524.
  4. Mohanbabu, A. G., et al. (2026). A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents. (arXiv:2609.13540).

Associated contacts