Local AI models running through tools like Ollama can handle real-world research data preparation tasks with surprising effectiveness, according to new research from University College London. Researchers tested consumer-grade, locally run AI models on sensitive data within secure research environments, finding that the strongest performers completed complex data preparation tasks up to 87.9% of the time.

Why Can't Researchers Use Cloud AI Tools with Sensitive Data?

Many researchers work within Trusted Research Environments (TREs), which are highly secure digital spaces designed to protect sensitive information like health records or personal survey data. These environments are intentionally isolated from the internet to prevent data breaches. This isolation creates a major problem: most popular AI tools like ChatGPT or Claude run on cloud servers operated by external companies, which means sending data outside the secure environment.

That's where local AI models come in. Open-weight models, which can be freely downloaded and run on a researcher's own computer or server, offer a potential solution. Tools like Ollama and LM Studio allow researchers to run AI systems entirely within their secure environments, keeping sensitive data completely private.

What Did the Research Actually Test?

Researchers from University College London conducted one of the first systematic evaluations of whether smaller, consumer-grade local AI models could handle real research work. They focused on data preparation, one of the most time-consuming parts of any research project. Before researchers can analyze data, they typically need to clean it, organize it, and write code to create new variables from raw information. This process often requires thousands of lines of code and can take many hours.

The team used data from Next Steps, a long-running study that has followed the same individuals from age 14 to age 32. The original task involved transforming raw survey data into a research-ready dataset, which required thousands of lines of code and the creation of 102 different variables. The researchers created a standardized benchmark called RRBench to measure how well AI systems could complete these tasks, and they made both the benchmark and evaluation tools freely available for other researchers to use.

How Well Did Local Models Actually Perform?

The results were better than expected. The strongest locally run AI models successfully completed research data preparation tasks up to 87.9% of the time, suggesting they could provide a practical route to AI-assisted research in fields like population health and social science. However, performance varied considerably between different models.

One critical finding emerged: smaller models were much more likely to make "silent" errors. These are mistakes where the code appears correct and runs successfully, but actually contains errors that produce incorrect results. This could lead researchers to draw wrong conclusions without realizing it. This highlights the importance of choosing AI tools carefully and thoroughly checking their outputs.

What Factors Determined Success or Failure?

The research identified several key factors that predicted whether local AI models would succeed or fail on research tasks:

• Information Complexity: Tasks with many variables or complicated descriptions were significantly more likely to fail. For each task, the AI was given information describing the data and variables involved, and simpler documentation led to better performance.
• Model Size and Power: The highest-performing local AI models generally require large amounts of memory (RAM) and more powerful computers than many researchers currently have access to, creating a practical barrier for individual researchers.
• Documentation Quality: Well-organized, searchable documentation could help AI systems perform better, suggesting that how research data is described matters as much as the AI model itself.

The researchers also noted that performance might improve by using multiple AI agents rather than a single system. Different agents could be given specialized roles, such as refining tasks, searching documentation, reviewing code, and checking results.

How Can Research Institutions Use These Findings?

The research suggests several practical ways that research infrastructure providers could implement local AI models to benefit researchers:

• Deploy Within Secure Systems: Providers of Trusted Research Environments could offer local AI models like Ollama directly within their secure systems, allowing researchers to use AI-assisted tools without sending data outside the environment.
• Create Research-Ready Datasets: Data providers could use local AI systems to help create standardized datasets and the code needed to generate them, potentially saving researchers thousands of hours by reducing duplicated work across different projects.
• Invest in Computing Infrastructure: For researchers working on personal computers, hardware remains a significant challenge, suggesting that further investment in computing infrastructure is needed if researchers are to make full use of local AI tools.

The researchers emphasized that implementing these approaches carefully could help maximize the scientific and policy impact of valuable research datasets.

What Are the Remaining Challenges?

Despite the promising results, significant barriers remain. The most powerful AI models often require extremely expensive computing systems that most researchers do not have access to. As a result, many researchers are limited to smaller models that can run on ordinary desktop computers or laptops. While smaller models continue to improve, the hardware requirements of the strongest performers remain a practical limitation.

Additionally, researchers will continue to play an important role in assessing the accuracy, reliability, and potential biases of AI systems. This evidence will help determine which tasks can safely be accelerated or delegated to AI, allowing researchers to spend more time on interpretation, insight, and meaningful analyses rather than routine data preparation.

The RRBench benchmark remains available for testing future AI models as they are released. The framework is open source and can be adapted to different research areas, programming languages, and tasks, meaning any task that produces structured output can potentially be evaluated using it.