DYNAMIC-AI
Research type
Research Study
Full title
Digital Innovation with Remote Management and Predictive Modelling to Integrate COPD Care with Artificial Intelligence-based Insights: An Acceptability, Feasibility and Safety Study
IRAS ID
287942
Contact name
Chris Carlin
Contact email
Sponsor organisation
NHS GG&C
Clinicaltrials.gov Identifier
Clinicaltrials.gov Identifier
N/A, N/A
Duration of Study in the UK
0 years, 8 months, 31 days
Research summary
Summary of Research
We are proposing undertaking this DYNAMIC-AI study to establish the acceptability to patients with COPD and technical feasibility of presenting AI model-based risk prediction scores to the COPD MDT.
We have established in the 'RECEIVER' trial that the 'LenusCOPD' based COPD digital service has shown sustained patient usage and improved COPD outcomes (significantly increased time to hospital admission or death, reduced respiratory-related hospital admissions and occupied bed days) in patients who are setup in the service. Information on the service and 'how it works' is at https://support.nhscopd.scot
In parallel with this digital transformation, we have undertaken machine-learning based risk prediction model development and validation using a large de-identified dataset of COPD patients from NHS GG&C SafeHaven. With routine healthcare data these models can predict a patient with COPD's risk of 12-month mortality, 3-month hospital admission and 3-day COPD exacerbation. The accuracy, performance, explainability and fairness metrics of these models are ready for adoption within an implementation-effectiveness evaluation framework. We therefore propose to undertake a 12-month observational cohort study, with planned 3-monthly interim analyses of the co-primary endpoints. This study design ensures scope to adapt the implementation strategy (eg adapt patient information, resolve unexpected issues with technical architecture) if these interim evaluations demonstrate suboptimal effectiveness. The secondary objective analyses will establish preliminary data including clinician interactions and actions based on model risk scores in a live clinical environment. These will help evaluate impact of risk-score triggered patient or clinician actions which may influence COPD events and model accuracy. This study is therefore an essential preliminary investigation, which will inform future implementation and design of subsequent pivotal clinical investigation(s) of COPD AI-insights, with the appropriate governance and safety scrutiny ensured by this feasibility clinical investigation.
Summary of Results
Background & Purpose
Chronic Obstructive Pulmonary Disease (COPD) is a long-term lung condition that often causes sudden flare-ups (exacerbations) and hospital admissions. Currently, healthcare is often "reactive"—treating patients only after they become unwell.
The DYNAMIC-AI study aimed to test a "proactive" approach. We developed Artificial Intelligence (AI) models that look at a patient’s health records to predict who is at high risk of adverse events (mortality, hospital re-admission, hospital admission, exacerbations) over the following time period. This study explored if providing these AI "risk scores" and the explainablility risk score features (components of the health record that were increasing or decreasing the risk score) to the investigator's respiratory clinical team meetings was acceptable to patients, technically possible, and safe.
What We Did
We invited patients already using the LenusCOPD digital service in NHS Greater Glasgow & Clyde to join the study. If they agreed, our AI system securely analysed their routine health data (such as medication history and previous hospital visits) to create risk scores. These scores were shown to doctors and nurses during study multidisciplinary team (MDT) meetings. The scores and the model features (which health data points were increasing or decreasing risk) to help them decide which patients needed extra support or a change in treatment.
Key Results
Patient Support (Acceptability): Patients were very supportive of the technology. 130 patients joined the study, while only 14 declined. This suggests that patients are comfortable with AI being used to help their clinical team provide better care.
System Performance (Feasibility): The system worked well in a real-world NHS setting. We successfully generated 12-month mortality AI risk scores for 121 out of 130 participants. The few failures were due to missing data in electronic records rather than an error in the AI system.
Safety: The study was safe. There were no device-related adverse events or safety issues related to the AI tool during the study recruitment or follow up period.
Model performance (accuracy): The model risk scores were compared with the patient outcomes during the follow up period. The model's prospective performance matched that in the retrospective training data, which is reassuring. Clinical correlation of the model risk scores was also reassuring: there were no inappropriately high or low risk scores identified.
Clinical Value (Utility): The clinical team found the AI insights helpful. A number of patients with high or rising risk scores and proactive clinical review undertaken. In some cases the review or care intervention triggered was directly related to addressable features highlighted by the model. The clinical team were also reassured that the "low-risk" patients were stable, allowing them to focus their time where it was needed most.
Conclusion & Impact
The study demonstrates that it is safe and technically possible to use live machine-learning model AI based on electronic health record data to help manage COPD in a secondary care NHS setting. Patients were generally happy for their data to be used in this way. These results provide a blueprint for adoption of predictive risk stratification AI insights within healthcare systems, and provide essential foundation data for subsequent prospective clinical trials which would further explore the utility, clinical effectiveness and cost effectiveness of these AI tools. Importantly the results suggest prospects of these tools being used for patients with COPD and other long-term conditions to reduce disease impact, improve access to personalised interventions, shift from reactive to proactive care and support workforce capacity and experience in future.
REC name
West of Scotland REC 4
REC reference
22/WS/0152
Date of REC Opinion
21 Dec 2022
REC opinion
Further Information Favourable Opinion