Quality comparison of traditional and LLM PILs
Research type
Research Study
Full title
Comparative Quality Analysis of Traditional and Large Language Model Derived Patient Information Leaflets in Vascular Surgery
IRAS ID
337080
Contact name
Iain Roy
Contact email
Sponsor organisation
St George's University Hospitals NHS Foundation Trust
Duration of Study in the UK
0 years, 6 months, 7 days
Research summary
This study aims to compare patient information leaflets (PILs) for vascular surgical conditions, including peripheral artery disease and abdominal aortic aneurysm (AAA) with patient information produced by large language models, a form of generative artificial intelligence.
PILs from the Circulation Foundation and Society for Vascular Surgery will be compared with patient information text generated by large language models. Patient information will be generated by developing questions to pose to the following generative AI platforms from PIL subheadings: ChatGPT, the NHS Large Language Model and Google Bard.
The LLM patient information will be compared with the gold standard of traditional PILs: 1) for accuracy, comprehensiveness, readability and tone through surveys completed by vascular surgery clinicians, 2) comprehensiveness, readability, tone and overall utility through surveys completed by patients, carers and relatives, 3) for readability through automated techniques and 4) for overall impression of quality through a comparative rating task completed by vascular clinicians.
REC name
London - Queen Square Research Ethics Committee
REC reference
24/PR/0304
Date of REC Opinion
2 May 2024
REC opinion
Further Information Favourable Opinion