Testing an AI appointment management system
Research type
Research Study
Full title
A pragmatic trial of an Artificial intelligence DRiven appOInTment maNagEment SyStem
IRAS ID
301143
Contact name
Nicola Thomas
Contact email
Sponsor organisation
London South Bank University
ISRCTN Number
ISRCTN16329124
Clinicaltrials.gov Identifier
NA, NA
Duration of Study in the UK
1 years, 10 months, 1 days
Research summary
Managing out-patient appointments is a challenge to healthcare providers and patients, with the possibility that missed appointments can result in missed treatments and wasted capacity. The study is evaluating a medical technology called DrDoctor which aims to improve and make best use of appointment attendance through Artificial Intelligence (AI). The technology has three main elements (1) a Did-Not-Attend (DNA) system which predicts how likely it is that patients will attend (2) a linked appointments system, which monitors the impact of appointment changes in one care pathway on other pathways, eg. blood tests with appointment changes and (3) a decision support tool which recommends the appointment type (eg. face to face vs. remote) and urgency of appointment to clinical staff, based on patient provided information.
The lead Trust (hospital) site involved in the evaluation is Nottingham University Hospitals NHS Trust. We are focussing on the specialities of kidney, cancer and ophthalmology (eyes). One part of our evaluation tests how effective and value-for-money the technology is. To test this, the research team will evaluate patient outcomes from hospital data at three time points: pre-COVID, three months prior to the start of the technology being implemented (pre) and six months after the technology has been implemented (post). At the second two time points (pre and post), we will collect mental and physical health outcomes and cost and service satisfaction via a questionnaire directly to patients. We will also compare pre-and post- outcomes across the sites and we will compare the outcomes across the sites with the technology, and also against the sites that do not have the technology (control). This ethics application mostly concerns the patient questionnaires described above, and will be supplemented by a Confidentiality Advisory Group (CAG) application covering the data components.
Lay summary of study results: Background During 2021/2022, nearly 7.5 million outpatient appointments in England were missed. These missed appointments are often termed ‘Did not Attends’ (DNAs). In 2023, NHS England highlighted the need to reduce DNAs, in order to improve patient experience, free up capacity to tackle the large numbers of people waiting for an appointment or operation. One possible way to reduce DNAs is the use of digital technology. DrDoctor is a digital solution that has already helped hospitals by informing people about their appointments by text alerts. The new DrDoctor technology aims to further improve the best use of appointments and attendance through artificial intelligence (AI) modelling.
Phase 1 is where the AI model predicts people who might not attend their appointment. Phase 2a/2b involves linked appointments to ensure that if one appointment relies on a diagnostic test, the result is available before the appointment.
The purpose of the evaluation was to assess whether the Dr Doctor AI technology is accurate, safe, effective, good value for money, and is acceptable to patients and NHS staff. Based on the evaluation, recommendations are to be made on the possible extension of the technology more widely throughout the NHS.
Methods
The evaluation was carried out in two hospitals where they had the technology and two hospitals where they did not, in the areas of kidney, cancer and eye care. The evaluation looked at the process to develop the AI model; administration and patient records to look at how safe and effective the technology was; a patient survey to understand the experiences of patients; and interviews with patients and staff to explore any concerns they might have about the DrDoctor technology. The cost of the technology and possible cost savings to hospitals was also looked at.
Results
The evaluation has found that the Phase 1 technology is accurate in terms of predicting who might not attend their appointment, as well as being safe and mostly acceptable to patients. Phase 1 technology did not appear to have a negative effect on patients’ mental health and physical wellbeing, including self-reported adverse events. However, it did not appear to improve patients’ satisfaction of appointment management. There was no detectable change in the number of people who attended their appointment across three specialities in two NHS Trusts. However, only 80 people were called, so more work is needed to see if calling can make a difference. Evaluation of Phase 2 technology was limited due to its recent implementation in only one Trust.
Conclusions
There was no significant change in the number of people who attended their appointment following the use of the Phase 1 technology. Patients who were identified as not likely to attend their appointments were either called or sent an additional SMS. However very few people were called so it was difficult to see if calling made a difference. On the basis of the current results, we cannot recommend the Phase 1 technology in these two hospitals. However other hospitals might get different results, so more work is needed. Phase 2 technology could be useful, but this also needs more work to see if it makes any difference to appointment management.REC name
London - Fulham Research Ethics Committee
REC reference
22/LO/0130
Date of REC Opinion
1 Apr 2022
REC opinion
Further Information Favourable Opinion