Study design and participants

The study was designed as a prospective, cross-sectional feasibility study in a group of service users accepted for an assessment for psychosis including people with first episode psychosis or assessed as being at risk of developing psychosis. Control cohort was recruited independently at the University of Exeter (UoE). Demographic and clinical characteristics of participants can be found in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Demographic and clinical characteristics of participants.

https://doi.org/10.1371/journal.pdig.0000339.t001

Service users (SU) were identified and recruited by Devon Partnership NHS Trust (DPT) and Avon and Wiltshire Mental Health Partnership NHS Trust (AWP). In total we recruited 32 participants, all of which were included in the analysis (we do not have data about number of screened participants, none of the participants dropped-out). The inclusion criteria were being accepted for an assessment for psychosis or risk of developing psychosis by a consultant psychiatrist or a trained specialist with experience in at-risk mental states. The exclusion criteria were:

1. Lacking capacity to provide informed consent for inclusion. The clinical team had an opportunity to assess the mental capacity at the CAARMS (comprehensive assessment of at-risk mental state (ARMS) [33] appointment before potential participants were approached to request consent to contact.
2. Insufficient understanding of English to follow the test instructions.
3. Any suspected organic cause of psychosis (i.e., head injury, epilepsy or dementia).
4. Taking antipsychotic medication for longer than 4 months before the start of the study.

Each participant was offered £10 for participating and if necessary, reimbursement for reasonable travel expenses (after producing a receipt). SU were recruited between 19/07/2018–23/05/2019. The study was reviewed by Research Ethics Committee (REC) and received approval from Health Research Authority (HRA) and Health and Care Research Wales (HCRW); IRAS (Integrated Research Application System) ID: 236262, REC reference: 18/SW/0065, protocol number 1718/26.

Control cohort (CC) was identified at UoE. In total we recruited 86 eligible participants, of which 43 played the same version of MG as SU and were used to identify the n = 32 CC matching by age and gender the SU as close as possible. Participants were volunteers recruited by personally approaching potential participants, putting posters around UoE campus and at Exeter’s community centres, social media adverts, and snowball sampling. The exclusion criteria were:

1. Moderate, or more severe, symptoms of depression, assessed by means of Patient Health Questionnaire-9 (PHQ-9) [34]. For ethical reasons we excluded the question concerning thoughts of suicide and self-harm. All participants scoring above 9, indicating at least moderate levels of depression, were signposted to several sources of support.
2. A diagnosis of depression, an anxiety disorder or schizophrenia.
3. Taking any psychopharmacological medication. Participants who indicated that they were having difficulties with mental health were directed to the UoE wellbeing centre.
4. Suffering from seizures.
5. English not being one of their first languages. This criterion was introduced to try to minimise the chances of misinterpretations due to the extensive use of questionnaires in the study.

CC was additionally screened using Community Assessment of Psychic Experiences-42 (CAPE-42) [35]; question 14, which asks about suicidal ideation and loads onto the depressive subscale, was excluded for ethical reasons. Each participant was offered £5 or one course credit for participating. CC was recruited by AW between 25/05/2018–26/11/2018 as part of his Master’s degree project. The recruitment of the CC was approved by University of Exeter, College of Life and Environmental Sciences (CLES), Psychology Ethics Committee, eCLESPsy000568 v2.1.

Participant flow chart is presented in Fig 1. All participants gave written informed consent prior to the study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Participant flow chart.

https://doi.org/10.1371/journal.pdig.0000339.g001

Further information about the study can be found at https://www.hra.nhs.uk/planning-and-improving-research/application-summaries/research-summaries/movement-and-perspective-taking-as-a-diagnostic-aid-for-psychosis/. Full study protocol can be accessed at http://hdl.handle.net/10871/132205.

Mirror game

The mirror game (MG) used in this study was based on the algorithm described by Zhai and colleagues [36] and followed closely our earlier work on establishing socio-motor markers of schizophrenia [18]. MG is a movement task that can be used to assess socio-motor functioning (motor coordination and interpersonal synchronisation) [18,19]. We used two MG tasks. The first task was a Solo game, where participants were asked to move their hand freely in a horizontal direction. Participants were given the following instruction: “Please move your hand left and right, create an interesting motion and enjoy playing.” The second task was a Leader-Follower game. In the second task an animated image of a robot appeared on the screen. The animation showed the robot controlling its own dot. The dot moved horizontally according to a pre-generated movement pattern. Participants were tasked with following the dot’s movement as closely as possible whilst it was on screen. Participants were given the following instruction: “Please try to follow the movement of an animated robot as accurately as you can.” During the Leader-Follower game the robot was also presenting parametric positive social feedback (smiling) as described by Cohen and colleagues [37].

The two tasks were grouped into one session. The session consisted of the Solo game, three repetitions of the Leader-Follower game and another Solo game. Each game lasted for one minute. The session was repeated three times. Participants were free to take breaks between the games and sessions. Each Leader-Follower game used a different pre-generated movement pattern. Patterns were the same for each participant. We excluded the 1^st Solo game and the 1^st Leader-Follower game to allow participants to get familiar with the task. The SU participants were sitting in front of a 17” diagonal laptop computer (1100x680 pixels screen resolution), the CC was using a 23” diagonal computer monitor; the image displayed on the computer monitor was scaled down to use the central 17” diagonal part of the monitor and have the same resolution as the laptop display. Movement of the hand was recorded using a leap motion sensor [38] and displayed as a dot on the screen. Participants used their dominant hand to control the horizontal position of a dot on the screen. The computer set-ups were different in the two groups due to the need of collecting the data simultaneously in multiple locations and additional research goals for the experiments with the CC that are not a part of the presented analysis.

In our analysis we used the recorded position of the participant hand (Solo and Leader-Follower) and the trajectory of the movement generated by the computer (Leader-Follower). Recorded position data is in arbitrary units in the range [-0.5, 0.5]; variable sampling rate, 90–140 Hz (S) and 40-70Hz (Leader-Follower). Pre-processing included:

• resampling to 100Hz with linear interpolation,
• low pass filtering with 5 Hz cut-off done using phase preserving Butterworth filter of degree 2,
• omitting the first and last 5s of the recording,
• estimation of movement velocity, using a fourth-order finite difference scheme.

Trail-making task

The trail-making task (TMT) [16,39] is a valid, public domain test of visual attention, working memory and executive control [40]. It has two parts, which were alternated. In each part, participants must click on 25 dots in a specified order as quickly and accurately as possible. The visual attention part (TMT A) had participants click numbers in ascending order, 1–25. The executive control part (TMT B) had participants alternate between clicking on numbers and letters, both in ascending order (1-A-2-B-3-C etc.). Participants completed each part three times, alternating between TMT A and B, starting with TMT A. Only the last two repetitions were included in the analysis. We excluded the 1^st repetition to allow participants to get familiar with the task. We used a digital version of the task implemented in PEBL: The Psychology Experiment Building Language [41]. In the original study protocol the TMT was used as a non-diagnostic attention measuring task. It was retrospectively included in the analysis after literature review [37,42–44] and data analysis indicated that including participants’ performance in this task could be beneficial for differentiating between the SU and CC.

For analysis we used the times between each individual mouse click made by the participant, we also used times between mouse clicks made on the correct targets.

Testing procedure

The stages of the research session are presented in Table 2. Both tasks and examples of collected data are shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Illustration of the Mirror Game (MG) and the Trail Making Task (TMT) together with examples of collected data.

In the MG participant sat in front of a computer with a connected leap motion sensor. In the Solo game (first row) the participant was instructed: “Please move your hand left and right, create an interesting motion and enjoy playing.” We recorded the horizontal hand movement (blue). In the Leader-Follower game (second row) the participant was instructed: “Please try to follow the movement of an animated robot as accurately as you can.” We recorded movement generated by the computer (leader (L), green) and movement of the participant (follower (F), blue). In the TMT participant sat in front of the same laptop computer but was using a computer mouse to complete the task. In the TMT (third row) the participant was asked to connect a set of 25 dots as quickly and accurately as possible (in order given by numbers (Part A) and alternating numbers and letters (Part B)). We recorded the time between each click a person made on the screen and analysed both parts together (Part A, dots; Part B, crosses). For the sake of clarity, we show simplified illustration with 9 dots instead of 25. Part of the MG illustration is based on the same source file as Fig 1 in [18], Fig 1 in [18] is distributed under the terms of the CC BY 4.0 license (CC BY 4.0).

https://doi.org/10.1371/journal.pdig.0000339.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Stages of the research session.

https://doi.org/10.1371/journal.pdig.0000339.t002

Sample size

For the feasibility study we recruited as many eligible SU as possible for the duration of the study. We approached all eligible participants that had been identified as appropriate for assessment for risk of psychosis by the Specialist Teams for Early Psychosis. Convenience sampling allowed us to proceed with the study as quickly as possible and assess what are feasible sample sizes for future research. Sample size of the CC was driven by the research objectives of AW’s Master’s degree project.

Features extracted from data

The selection of features for classification was informed by our earlier work [18], and modified to better fit the machine learning methodology employed in the current study. Instead of using distributions (histograms) as in the previous work, here we use a set of their descriptive statistics (e.g., mean, standard deviation or median) or point measures (e.g., power at 5Hz frequency). As previously, the data was concatenated or averaged across the repetitions. Before averaging or concatenating the movement data, we remove parts where participants’ moves reached the edges of the sensor range (-0.5 or 0.5 value). The complete list of the features, and their description, is presented in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. List of features and data used to estimate them.

https://doi.org/10.1371/journal.pdig.0000339.t003

Classes

We used the group (CC or SU), as the primary classification outcome (predicted variable), binary classification of the full dataset. Additionally, we used CAARMS score (CAARMS > 0 –at-risk mental state, psychotic or CAARMS = 0 –neither) as classification outcome of an independent binary classification within the SU group. CAARMS is one of the gold standards in assessing service users at risk of or with first episode of psychosis. CAARMS was completed by a consultant psychiatrist or a trained specialist with experience in at-risk mental states of STEPs before completion of the MG and TMT tasks.

Classification methods

We compared following classifiers: k-nearest neighbours (kNN), naïve Bayes (NB), support vector machines (SVM), bagged trees (BT), linear discriminant (LD) [48]. To find the k-nearest neighbours for kNN classifier we used a cosine distance to measure distances between the points in the n-dimensional feature space (each coordinate of the feature space corresponds to a single z-scored feature). Cosine distance is defined as 1–cos(θ), where θ is the angle between vectors defined by coordinates given by the set of features and the origin of the coordinate system. We used a NB classifier as implemented in Matlab 2022b function fitcnb [49], a SVM classifier as implemented in Matlab 2022b function fitcsvm [50], a BT classifier as implemented in Matlab 2022b function TreeBagger [51] and a LD classifier as implemented in Matlab 2022b function fitcdiscr [52]. All classifiers were trained using default Matlab 2022b settings.

To avoid overfitting, we used only three out of the 18 features (Table 3), namely one from the 6 features estimated from the TMT data, one from the 4 features estimated from the MG Solo task and one from the 8 features estimated from MG Leader-Follower task. To select the features, we used the value of Cliff’s delta [53], a non-parametric measure of effect size. The set of classification features was based on the results from the training phase, meaning that it was selected separately for each training-testing split (fold).

Additionally, we analysed how performance of the kNN and LD classifiers depends on the number of features. We compared kNN with 3 (selected as described above) and all 18 features and LD with 2 (selected as described above but without MG Solo task), 3 (selected as described above) and all 18 features.

Training and testing

To evaluate the performance of the classifiers, we used two training-testing splits. A 25% hold-out (HO) training-testing split and a leave-one-out (L1O) training-testing split (corresponding to 2% hold-out in our case). Parameters of the classifier and the three features used for classification were identified using only the training set. Hold-out data is used only for testing and is unseen by the classifier during the training.

In the 25% HO split we selected at random 25% of the data (8 out of 32 participants in each cohort). We train the classifier using the remaining 75% of the data (24 CC and 24 SU datasets). We used the 16 participants (8 CC and 8 SU datasets) unseen by the classifier during training to construct confusion matrix, and compute performance metrics. To estimate 95% confidence intervals of the classifier performance we repeated the 25% HO split 1000 times (1000 folds).

The leave-one-out (L1O) training-testing split used n-1 participants to train the classifier and 1 participant to test the model. L1O training-testing split allowed to test the methodology n times. The L1O split simulated situation where a new participant would be diagnosed using classifier based on all the data available prior to the arrival of the new participant. To construct confusion matrix and compute performance metrics we compared the original classes with the set of individual predictions of each of the L1O splits, i.e., we compared original class of the participant unseen by the classifier with the class predicted by the model trained using the other n-1 participants.

Classification results

All tested methods allowed classification of the CC and SU participants with an excellent (0.8–0.9) [54] or outstanding (0.9–1) [54] accuracy, sensitivity, specificity and precision. The only exception being acceptable (0.7–0.8) [54] or poor (0.6–0.7) [54] sensitivity in few cases (see Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Results of using 8 classification methods: k-nearest neighbours (kNN), k-nearest neighbours with 3 features (kNN3), naïve Bayes, support vector machines (SVM), bagged trees (BT), linear discriminant (LD), linear discriminant with 3 features (LD3) linear discriminant with 2 features (LD2) and two types of training-testing split (HO and L1O) for classifying SU and CC and results of L1O training-testing split classifying SU with and without at-risk-mental-state (CAARMS = 0 vs. CAARMS > 0). n in Testing dataset column shows number of datasets from the indicated cohort used for testing (see also the Training and testing section).

https://doi.org/10.1371/journal.pdig.0000339.t004

Comparing how classification results depend on the number of features we found that the performance of the kNN classifier was comparable when using 3 or 18 features. The kNN classifier using three features had higher AUC but lower specificity and precision than the kNN using all 18 features. The performance of the LD classifier showed stronger dependence on the number of features. It performed best using three features and its performance decreased when using two and 18 features.

Furthermore, all the analysed methods failed to differentiate between SU with and without at-risk-mental-state (CAARMS score of 0 and CAARMS score > 0); binary classification within the SU cohort using CAARMS score as group label, CAARMS = 0 vs CAARMS > 0. Since there were only 16 participants with CAARMS = 0 and 16 participants with CAARMS > 0 we only used the L1O training-testing split. See Table 4 for details.

To better understand the difference in performance of the proposed methodology in the two cases (CC vs. SU and SU CAARMS = 0 vs SU CAARMS > 0), we compared distributions of the features in the 3 groups. We did not find any statistically significant differences between the SU CAARMS = 0 vs SU CAARMS > 0 groups. Interestingly, we observed that the values of most features in the SU CAARMS > 0 cohort differ more from the CC compared to the difference between SU CAARMS = 0 cohort and CC; see Fig 3 and Table 5. Moreover, we observed that the effect size (Cliff’s delta) is overall higher for the features from the MG task while remaining relatively unchanged for TMT. The statistical significance of the difference observed in the pattern of change between the SU CAARMS = 0 vs SU CAARMS > 0 groups was investigated using a bootstrap test with 10000 random splits of the SU cohort of observing simultaneous low change in effect sizes for TMT (smaller than the median change of the 6 TMT summary statistics measures), mixed change in effect sizes for Solo MG task (larger than the median change of the 4 Solo MG summary statistics measures) and large change in effect sizes for Leader-Follower MG task (larger than the median change of the 8 Leader-Follower MG summary statistics measures). We found this to be statistically significant with p < 0.0031.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Distributions of summary statistics measure values in CC (blue) and two groups of SU, CAARMS = 0 (gray) and CAARMS>0 (orange).

Violin plots illustrate distributions of the values, white dot shows median, gray vertical bar shows IQR (middle 50% of values), scatterplots in each violin plot show all individual values. In all plots y-axis shows z-scored values in arbitrary units.

https://doi.org/10.1371/journal.pdig.0000339.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Table of difference between summary statistics (point) measures in CC and SU (combined), CC and SU with CAARMS = 0 and CC and SU with CAARMS>0.

https://doi.org/10.1371/journal.pdig.0000339.t005

Features’ importance and classifiers’ interpretability

Overall excellent performance of the classifiers can be explained by the large effect sizes of the differences between the cohorts in terms of the features used for classification. The effect sizes (presented in Table 5, column ‘Effect size’) are directly related to the differences (distance) between feature values and affect performance of all the considered classifiers. This conclusion is directly confirmed by the almost identical performance of the kNN classifier using 3 and 18 features, meaning that the 3 features with the highest effect size capture most of the information contained in the 18 features.

The best performance of the LD3 classifier demonstrates that the SU and CC cohorts are linearly separable. While decrease of performance of the LD2 classifier shows that all 3 tasks are important to discriminate between the cohorts. Table 5, column ‘Effect size’, shows that features derived from the TMT data are the most important and features from the MG Solo are the least important for classification. Worse performance of LD with 18 features could be a result of theoretical (‘curse of dimensionality’) [65] or numerical (e.g., separating points in higher dimension might require estimation of more complex decision boundaries) limitations.

Study limitations

There are two main limitations of the study. First, we did not control the level of education in the two groups and it is know that the performance in TMT is affected by years of education [17]. However, even using only the features from the MG allow to classify the CC and SU with AUC = 90 (0.69–1) and sensitivity = 0.75 (95%CI 0.5–1) and specificity = 0.88 (95%CI 0.62–1); kNN, 25% hold-out and 1000 folds. The second, potential source of bias is the short exposure to antipsychotic medication; less than 4 months. We allowed 4 months of antipsychotic medication in order to facilitate recruitment of participants while minimising potential for manifestation of motor side-effects associated with anti-psychotic drugs [66]. Therefore we anticipate the effect of medication status to be minimal and additionally confound with CAARMS score. In an earlier study we have shown that the obtained classification results are independent from anti-psychotic medication status [18]. Furthermore, a recent study showed that neurological signs (e.g., tests of coordination and balance) and their change over 10 years is likely unrelated to exposure to anti-psychotic drugs [31].

Feasibility of the real-world implementation

As a part of the study, we collected feedback from the participants and healthcare practitioners regarding acceptability and easiness of use of the tests. 65% of participants found the test acceptable as a part of a routine clinical assessment for risk of psychosis (answer >5 on a 10-point scale, 1 –not at all and 10 –very much, to a question: ‘How acceptable would you find the activity as a part of a routine clinical assessment for risk of psychosis?’) with only 5 out of 32 participants giving a score <5. 87.5% of participants replied that they would be comfortable taking such a test at home (answer >5 on a 10-point scale to a question: ‘Would you be comfortable taking such a test at home?’) with 17 out of 32 participants giving a score 9 or 10 and only 2 participants giving a score <5 (all questions and SU answers are available at https://doi.org/10.17605/OSF.IO/RNZYS). The healthcare practitioners (doctors and research nurses) who collected the data from the service users provided oral feedback about the tasks and the data collection set-up, they praised its portability (ability to run the tests at service users’ homes) and easy set-up (simply plugging in the USB cables into a laptop computer).

We envisage that most immediate potential future implementations of digital batteries of behavioural tests will be taking place within a healthcare setting, in a clinic or practitioners office). We foresee the tests as part of a decision support tool and to be employed as any other medical test. The data collected during the test will be treated as any other patients’ data collected during medical examination in terms of ethics and privacy. We believe that augmenting neuropsychological evaluation by means of digital batteries of behavioural test could be saving clinicians time for meaningful conversation with a help-seeking service user. We hope that the time saved by the tests, together with their demonstrated accuracy will help to humanise and destigmatise mental health diagnosis. We are aware that future deployment of such technology would require careful consideration of privacy and related ethical issues, which is beyond the scope of this paper.

Implications for clinical and research practice

Our findings reinforce the benefits of digital behavioural test and quantitative analysis of their results and their potential for being used as a mobile assessment platform; assessable in home settings as well [67]. Cheap, portable off-the-shelf equipment allows the assessment to take place in a range of indoor locations, while automatic data collection greatly simplifies the necessity for training of clinical personnel.

Digital behavioural tests would benefit research on prognostic instruments for psychosis. Recent review [54] identified heterogeneity in recruitment strategies for high-risk services as one of the factors limiting development of prognostic instruments for psychosis. Digital behavioural test could alleviate this limitation by stratifying and enabling the identification of more homogeneous clinical high-risk populations.

Finally, with further development standardised digital test batteries could supplement and augment neuropsychiatric/ neurological tests making them quicker and easier to apply in routine clinical practice. This would have wide ranging implications for home health, care-coordination and care referral. Therefore, future work should focus on identification of optimal set of tests for establishing standardised digital batteries of behavioural tests and their optimal technological implementations. Such innovative and cost-effective testing methods have the potential to be extended beyond STEPs users’ stratification [68] and would facilitate rapid assessments (in clinic or at home) for all people referred to mental health early intervention services [69,70], improving their quality of life and wellbeing.