The value of AI in mental health services
Research type
Research Study
Full title
Evaluating the efficacy and acceptability of an AI decision support tool for community mental health services in Greater Manchester
IRAS ID
339898
Contact name
Panos Constantinides
Contact email
Sponsor organisation
University of Manchester, Faculty of the Humanities
Duration of Study in the UK
2 years, 0 months, 0 days
Research summary
Recent research has shown that Artificial Intelligence (AI) technologies can be used to assist decision making in mental health care that can significantly reduce the burden of staff and improve the management of mental health crises. However, within the English NHS there are currently limited AI technologies available to NHS mental health providers that can help identify and support the care of service users. This research will examine how an AI decision support tool can enable augment the decision-making ability of healthcare staff to improve operational effectiveness and patient outcomes?
The University of Manchester has partnered with the Greater Manchester Mental Health NHS Foundation Trust (GMMH) and Holmusk UK, a data science company, to examine the value of an AI-enabled decision support tool called MaST within their adult (ages 18-65) Community Mental Health Teams (CMHTs). MaST is already being used in 15 CMHTs in Greater Manchester boroughs of Bolton, Salford, Trafford and Manchester.
The research will take place at the 15 CMHT in GMMH over the duration of 2 years (2024-2026). We will carry out interviews with CMHT staff, as well as observations of their use of MaST to evaluate the acceptability of the technology in everyday practices. We will also apply relevant statistics, machine learning and AI algorithms on anonymized MaST data to evaluate the efficacy of AI predictions.
We expect our research to significantly reduce the burden on clinicians and staff. More importantly, identifying people at high risk of mental health crisis earlier and responding quickly will reduce the likelihood of the service users (i.e., patients)’ mental health deteriorating. Our research will provide opportunities to share good practice of MaST use and digital adoption in mental health services, which may be shared with Trusts in the English NHS.Results Summary:
Summary Background. Mental illness accounts for 23% of the UK illness burden with estimated societal costs of over £100 billion per year. Algorithmic technologies can be used to assist decision-making in mental health care, helping to reduce the burden of staff while improving the management of mental health crises. We aimed to evaluate the algorithmic technology called Management and Supervision Tool (MaST), developed by Holmusk Europe and used by Community Mental Health Teams (CMHTs) within a large NHS Foundation Trust in the UK (hereafter the Trust) to predict the likelihood of using mental health crisis services.
Methods. We used a mixed-methods approach, combining a qualitative and quantitative evaluation of MaST. First, between January 2025 and May 2026, we conducted 85 interviews with CMHT staff at the Trust. Additionally, we conducted 90 observations of zoning meetings, multidisciplinary team meetings and supervision sessions held within CMHT at the Trust. These interviews and observations enabled an in-depth understanding of how CMHT staff assess the likelihood of using mental health crisis services with and without the algorithmic technology, as well as how they make decisions to manage risks and crises, following established UK NHS protocols. Second, we accessed fully anonymized data from 35,943 service users, who had a referral to a CMHT recorded in their Electronic Health Records (EHRs) between March 2023 and March 2026, for a mental health illness. We used these data to evaluate the underlying mechanism and predictive performance of the algorithm used by MaST to predict the likelihood of using mental health crisis services.
Findings. The quantitative evaluation showed that MaST demonstrates strong predictive performance for the use of mental health crises services. It achieves an area under the receiver operating characteristic curve (AU-ROC) of 0.8507±0.009 and an area under the precision-recall curve (AU-PR) of 0.2320±0.016, capturing 75% and 88% of crises within its top quintile and two quintiles of predicted risk propensity scores, respectively, and outperforming existing benchmarks reported in the literature. The qualitative analysis showed that while MaST provides a number of benefits such as enhanced visualization for caseload management to monitor contact frequency, evaluate key performance indicators, and support supervision of staff, CMHT staff rely more on their experience, tacit knowledge and ongoing interaction with service users to relationally manage the risk of crisis, rather than on MaST’s algorithmic predictions. CMHT staff face multiple challenges such as resource constraints, including demand outweighing capacity, training gaps and data workflow complexities, all of which contribute to lower or less effective use of MaST.
Interpretation. We make four distinct contributions. First, we show that technological implementation is a fundamentally epistemic challenge, demonstrating that algorithmic technologies create an enduring tension between data-driven logic and clinicians' relational knowledge. Second, we illuminate the critical, yet invisible professional labor required of CMHT staff to critically evaluate, translate, and contextualize algorithmic outputs. Third, while validating the robust predictive performance of MaST’s algorithm, we show how modeling factor interactions and temporal dynamics through interpretable machine learning could further enhance accuracy. Fourth, we emphasize that algorithmic innovation cannot resolve underlying structural constraints such as workforce shortages and under-resourced service capacity, warning that positioning digital tools as efficiency solutions to systemic socio-economic challenges risks shifting institutional burdens onto clinicians through data-driven prioritization.REC name
North West - Greater Manchester West Research Ethics Committee
REC reference
24/NW/0236
Date of REC Opinion
16 Aug 2024
REC opinion
Further Information Favourable Opinion