DOTS clinical evaluation in the A&E
Research type
Research Study
Full title
Clinical evaluation and implementation of DemDx AI-driven Ophthalmology Triage System (DOTS) in an adult eye casualty accident and emergency (A&E)
IRAS ID
310792
Contact name
Alex Day
Contact email
Sponsor organisation
Moorfield's Eye Hospital NHS trust
ISRCTN Number
ISRCTN25862260
Duration of Study in the UK
0 years, 3 months, 31 days
Research summary
Research Summary
Eye emergencies are becoming increasingly common, leading to a rise in demand for eye emergency care. Patients attending eye casualty services are first assessed by a trained doctor or nurse who decides whether their condition requires urgent medical care or if their condition is not urgent. However, evidence suggests that many patients with non-urgent conditions are given urgent attention, causing delays and incurring unnecessary costs.
To improve emergency eye care, a computer system that uses Artificial Intelligence (AI) to help nurses decide which patients need urgent attention is being tested in a new research study. The system, called DemDX AI-driven Ophthalmology Triage System (DOTS), has already been tested in a previous study and will now be tested in a real-world setting to evaluate its impact on nurses' ability to correctly identify patients who need urgent care and to reduce the number of unnecessary appointments.
The study will recruit nurses working with triage and they will collect data from 1198 patients who arrive at the Moorfields A&E. Nurses at the A&E will use the DOTS system to support triage patients, and their performance will be compared to a previous period when the system was not used. The study will also look at factors such as patient waiting time, the user-centred design of the system, and user trust in the recommendations provided by the system.
The findings of the study could lead to the wider adoption of AI technology in emergency eye care services, ultimately benefiting patients and healthcare providers alike. The use of AI-driven technology can help emergency eye care services to manage increasing patient demand effectively, resulting in better patient outcomes and reduced costs for the healthcare system. The study will contribute to the growing body of evidence supporting the use of AI in healthcare and provide insights into how technology can support healthcare providers in delivering high-quality care.
Summary of Results
A substantial proportion of attendances to ophthalmic emergency departments are for non-urgent presentations. We developed and evaluated a machine learning system (DemDx Ophthalmology Triage System: DOTS) to optimise triage, with the aim of reducing inappropriate emergency attendances and streamlining case referral when necessary.Methods
DOTS was built using retrospective tabular data from 11,315 attendances between July 1st, 2021, to June 15th, 2022 at Moorfields Eye Hospital Emergency Department (MEH) in London, UK. Demographic and clinical features were used as inputs and a triage recommendation was given (“see immediately”, “see within a week”, or “see electively”). DOTS was validated temporally and compared with triage nurses’
performance (1269 attendances at MEH) and validated externally (761 attendances at the Federal University of Minas Gerais - UFMG, Brazil). It was also tested for biases and robustness to variations in disease incidences. All attendances from patients aged at least 18 years with at least one confirmed diagnosis were included in the study.Findings
For identifying ophthalmic emergency attendances, on temporal validation, DOTS had a sensitivity of 94.5% [95% CI 92.3–96.1] and a specificity of 42.4% [38.8–46.1]. For comparison within the same dataset, triage nurses had a sensitivity of 96.4% [94.5–97.7] and a specificity of 25.1% [22.0–28.5]. On external validation at UFMG, DOTS had a sensitivity of 95.2% [92.5–97.0] and a specificity of 32.2% [27.4–37.0]. In simulated scenarios with varying disease incidences, the sensitivity was ≥92.2% and the specificity was ≥36.8%. No differences in sensitivity were found in subgroups of index of multiple deprivation, but the specificity was higher for Q2 when compared to Q4 (Q4 is less deprived than Q2).
Interpretation
At MEH, DOTS had similar sensitivity to triage nurses in determining attendance priority; however, with a specificity of 17.3% higher, DOTS resulted in lower rates of patients triaged to be seen immediately at emergency. DOTS showed consistent performance in temporal and external validation, in social-demographic subgroups and was robust to varying relative disease incidences. Further trials are necessary to validate these findings. This system will be prospectively evaluated, considering human-computer interaction, in a clinical trialSummary of Results
Background
A substantial proportion of attendances to ophthalmic emergency departments are for non-urgent presentations. We developed and evaluated a machine learning system (DemDx Ophthalmology Triage System: DOTS) to optimise triage, with the aim of reducing inappropriate emergency attendances and streamlining case referral when necessary.
Methods
DOTS was built using retrospective tabular data from 11,315 attendances between July 1st, 2021, to June 15th, 2022 at Moorfields Eye Hospital Emergency Department (MEH) in London, UK. Demographic and clinical features were used as inputs and a triage recommendation was given (“see immediately”, “see within a week”, or “see electively”). DOTS was validated temporally and compared with triage nurses’
performance (1269 attendances at MEH) and validated externally (761 attendances at the Federal University of Minas Gerais - UFMG, Brazil). It was also tested for biases and robustness to variations in disease incidences. All attendances from patients aged at least 18 years with at least one confirmed diagnosis were included in the study.Findings
For identifying ophthalmic emergency attendances, on temporal validation, DOTS had a sensitivity of 94.5% [95% CI 92.3–96.1] and a specificity of 42.4% [38.8–46.1]. For comparison within the same dataset, triage nurses had a sensitivity of 96.4% [94.5–97.7] and a specificity of 25.1% [22.0–28.5]. On external validation at UFMG, DOTS had a sensitivity of 95.2% [92.5–97.0] and a specificity of 32.2% [27.4–37.0]. In simulated scenarios with varying disease incidences, the sensitivity was ≥92.2% and the specificity was ≥36.8%. No differences in sensitivity were found in subgroups of index of multiple deprivation, but the specificity was higher for Q2 when compared to Q4 (Q4 is less deprived than Q2).
Interpretation
At MEH, DOTS had similar sensitivity to triage nurses in determining attendance priority; however, with a specificity of 17.3% higher, DOTS resulted in lower rates of patients triaged to be seen immediately at emergency. DOTS showed consistent performance in temporal and external validation, in social-demographic subgroups and was robust to varying relative disease incidences. Further trials are necessary to validate these findings. This system will be prospectively evaluated, considering human-computer interaction, in a clinical trial.REC name
Yorkshire & The Humber - Sheffield Research Ethics Committee
REC reference
23/YH/0068
Date of REC Opinion
17 May 2023
REC opinion
Further Information Favourable Opinion