A new study led by the University of Bristol and co-authored by the Health Research Authority (HRA) on using Artificial Intelligence (AI) for routine research approval checks has been published in the Nature Scientific Reports journal today (Monday 16 November).
The peer reviewed early stage proof of concept study, was carried out in 2024 on 6 historical research applications and involved assessing 3 Large Language Models (LLM), Claude 3.5 Sonnet, ChatGPT and Microsoft Copilot, on their ability to evaluate research participant information materials for the corresponding applications against specified quality standards.
The standards set out the criteria that information for research participants need to meet. These are routine checks that take place before a study reaches a Research Ethics Committee or HRA staff for HRA and HCRW Approval. Depending on the study, it can take more than a day to process the checks.
All three models were able to perform the checks, but their performance differed substantially. When reviewing their performance, the study assessed that Claude answered 87.9% of checks fully correctly, compared with 66.5% for ChatGPT and 62.8% for Copilot. Claude was also the fastest, taking an average of 7.17 minutes per application, compared with 13.58 minutes for ChatGPT and 23.25 minutes for Copilot, and it returned an answer to every question.
Failure of logical reasoning was the commonest source of error across the three models. Claude was able to recognise and correct its errors when challenged, whereas ChatGPT and Copilot sometimes defended incorrect answers by citing page numbers that did not exist.
The performance of the LLM was assessed by two HRA researchers with expertise in assessing against these standards, blinded to the specific LLM, and evaluating the answers against their own knowledge of the quality standards.
The study was the first of its kind to explore elements of how AI could be used in this way to streamline routine research approval checks. It is part of wider work the HRA is doing to establish an evidence base and proof of concept on how AI could be used responsibly to drive improvements in speed, quality and consistency.
We collaborated on this study to explore and develop an evidence base on how AI could potentially be used to support the streamlining of research approval processes, and to identify the potential risks, limitations, and opportunities.
The study demonstrated how AI, at the time of the research, could be used for specific routine checks before a study is reviewed by a Research Ethics Committee or HRA staff. The potential benefit of using AI in this way is that it could release staff capacity to focus on more complex regulatory issues, making our processes more effective and potentially also helping reduce the timeline for regulatory review while increasing the consistency of how assessment is approached, a common theme in research and sponsor feedback.
Excerpt from the official page. The original is authoritative. Continue reading at UK Health Research Authority ↗