The Language Explorer Clinical Evaluation Study
Research type
Research Study
Full title
Clinical evaluation of an automated language transcription and analysis app to assist speech and language therapists with clinical decision making in the evaluation of developmental language disorders
IRAS ID
276718
Contact name
Yvonne Wren
Contact email
Sponsor organisation
North Bristol NHS Trust
ISRCTN Number
ISRCTN12718815
Duration of Study in the UK
0 years, 11 months, 31 days
Research summary
Research Summary
Ten percent of children have delayed language in the preschool years. Some make progress without help; others need advice from a health visitor or a speech and language therapist (SLT). If a child’s problems with language development continue, they may be diagnosed with Developmental Language Disorder (DLD), which can affect their learning and development.
The Language Explorer is an app which helps SLTs analyse children’s language abilities. The app requires children to listen to a story told with pictures via the app, and then re-tell the story in their own words. This will provide an example of their language which is recorded and analysed by the app to provide a report on the child’s language skills. The app also asks questions about the story and asks the children to repeat sentences.
Children living in Bristol, Hackney or Newcastle, who have a diagnosis of DLD aged between 4 years and 7 years and 11 months will be invited to participate either in a clinical setting (if aged 4;0-5;11) or remotely (if aged 6;0-7;11). All Children will be supported to complete the Language Explorer app, a formal language assessment (CELF-2 Preschool UK for children aged 4;0-5;11 and Q-Interactive version of CELF-5 for children aged 6;0-7;11) and clinicians will complete a Clinical Judgement form.
The children’s performance on the Language Explorer app will be compared with the normative data from the Citizens Science workstream to investigate if Language Explorer can distinguish between children with DLD and children with typical language. The Clinical Judgement forms and all available data from the language assessment will also be used to determine if Language Explorer can be used to distinguish patterns of performance in children with DLD.
The parents and all clinicians involved in the study will also be invited to be interviewed about their experience using the app.
Summary of Results
BACKGROUND Language sampling is used by speech and language therapists (SLTs) to assess children with suspected Developmental Language Disorder, a condition which places children at risk of poor outcomes in education, employment, relationships and mental health. It consists of recording a child as they speak, often in a story retell task, and then analysing the language produced. Language sampling provides SLTs with important information about the patient’s use of spoken language within a naturalistic communication environment and informs decisions around diagnosis and treatment. Consistent with previous research into language sampling practices, the main barrier to more frequent use of this important clinical activity is the time needed for transcription and analysis.
Following a successful application to the NIHR i4i scheme, funding for the development and evaluation of a software tool to assist SLTs with the collection, transcription and analysis of language samples was provided to the applicant team, led by Rebecca Bright of Therapy Box. The application consisted of seven work packages covering project management, citizen science activity (to gather data from 600 typically developing children using an early version of the app as a remote tool with parents), tool development (three work packages), clinical evaluation and commercialisation. This report relates solely to the clinical evaluation.
The tool developed for this work was given the title Language Explorer and was developed with the aim of reducing the time required to carry out the collection, transcription and analysis of language samples through the automatisation of the transcription and analysis functions. The original aims of the application were stated as:
• To produce and evaluate language analysis software to assist SLTs in assessing children referred for suspected Developmental Language Disorder.
• To prepare a clinical decision-making product capable of producing a validated analysis of the child’s language in response to a narrative based language sample produced by using mobile technology, that can be compared with measures obtained through existing measurements carried out by SLTs.
Given the status of the tool as a medical device, a submission was made to MHRA by Therapy Box as the app developers. Approvals were sought and obtained from the REC and HRA for the work to be carried out in the Clinical Evaluation.CLINICAL EVALUATION
The evaluation comprised an investigation of the app’s performance in terms of its ability to assist in SLT management and a qualitative analysis of user feedback on the tool. The evaluation was consistent with the MHRA definition of a feasibility study, meaning that it is a first and preliminary assessment of whether the app met its intended purpose: to assist Speech and Language Therapists with clinical decision making in the evaluation of Developmental Language Disorder. The clinical investigation was carried out using data from children with known or suspected Developmental Language Disorder and compared to data from children with typical language. As such, the sensitivity of the app to detect language difficulties was assessed in a ‘best-case’ scenario (i.e., where group differences should be maximized), with the potential for a pivotal study to be undertaken at a later date, if indicated. In addition, some aspects of the acceptability of the app to clinicians and parents were evaluated.
As defined in the approved Clinical Investigation Plan submitted to MHRA, we planned to address the following Research questions:
Primary
• Can Language Explorer be used to distinguish children with known or suspected Developmental Language Disorder from those who have typically developing language?
Secondary
• Can different patterns of performance on Language Explorer be observed for different groups of children with Developmental Language Disorder?
• What are clinician and parent/carer views of Language Explorer?Data analysis and limitations
We have completed these analyses on the available data (an appendix with these data are available). However, there were a number of limitations that have affected delivery of a full report and the interpretation of the results without serious caveats. In summary these limitations were:
• The app was under continued development throughout the evaluation period, and some parts were never made available for evaluation. For this reason, we cannot give a view of whether the app in its entirety met the intended purpose of distinguishing children with known Developmental Language Disorder from those with typically developing language.
• The full raw data set collected during the evaluation was not made available, i.e., comparable data for the typical and Developmental Language Disorder groups were not shared for all language measures, and in some cases we received only summary data for measures.
• Where data were made available, our preliminary data quality checks identified issues which suggested the data contained errors.Outcome
The results of the analysis completed indicate that the app does not distinguish between children with Developmental Language Disorder and those with typical language for the majority of language measures. Furthermore, where a difference was evident, this was in the opposite direction to predicted in some cases. As such, the app cannot not be used to assist SLTs with clinical decision making in the evaluation of Developmental Language Disorder. We would not expect this outcome to fundamentally change were a complete data set provided.REC name
North of Scotland Research Ethics Committee 1
REC reference
20/NS/0128
Date of REC Opinion
11 Nov 2020
REC opinion
Further Information Favourable Opinion