Advertisement

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Effectiveness of tailored digital health interventions for mental health at the workplace: A systematic review of randomised controlled trials

Abstract

Mental health problems in the workplace are common and have a considerable impact on employee wellbeing and productivity. Mental ill-health costs employers between £33 billion and £42 billion a year. According to a 2020 HSE report, roughly 2,440 per 100,000 workers in the UK were affected by work-related stress, depression, or anxiety, resulting in an estimated 17.9 million working days lost. We performed a systematic review of randomised controlled trials (RCTs) to assess the effect of tailored digital health interventions provided in the workplace aiming to improve mental health, presenteeism and absenteeism of employees. We searched several databases for RCTs published from 2000 onwards. Data were extracted into a standardised data extraction form. The quality of the included studies was assessed using the Cochrane Risk of Bias tool. Due to the heterogeneity of outcome measures, narrative synthesis was used to summarise the findings. Seven RCTs (eight publications) were included that evaluated tailored digital interventions versus waiting list control or usual care to improve physical and mental health outcomes and work productivity. The results are promising to the advantage of tailored digital interventions regarding presenteeism, sleep, stress levels, and physical symptoms related to somatisation; but less for addressing depression, anxiety, and absenteeism. Even though tailored digital interventions did not reduce anxiety and depression in the general working population, they significantly reduced depression and anxiety in employees with higher levels of psychological distress. Tailored digital interventions seem more effective in employees with higher levels of distress, presenteeism or absenteeism than in the general working population. There was high heterogeneity in outcome measures, especially for work productivity; this should be a focus of attention in future studies.

Author summary

Work-related stress has become a major issue at the work place. It is associated with increased physical and mental health risk. Work stress is also associated with long term economic impact due to absenteeism and presenteeism. There have been several workplace interventions to tackle this issues, with varying results. In recent years, digital health interventions have become increasingly popular to promote employees’ physical and mental wellbeing. Some have shown moderate effects improving employees’ psychological wellbeing. However, most interventions were not tailored to the individual employee. So far, no review has explored the effect of tailored digital interventions addressing stress in the workplace compared to usual care. Also, no reviews have taken blended e-health models into account explicitly. This systematic review seeks to address this gap. We found that tailored digital interventions did not reduce anxiety or depression in the general working population but they significantly reduced anxiety in employees with higher levels of psychological distress. However, as the outcomes measured were so diverse, it is difficult to draw conclusions. There is a need for uniformity in the use of assessment tools and outcomes reporting in future research, especially in terms of work productivity.

Citation: Moe-Byrne T, Shepherd J, Merecz-Kot D, Sinokki M, Naumanen P, Hakkaart-van Roijen L, et al. (2022) Effectiveness of tailored digital health interventions for mental health at the workplace: A systematic review of randomised controlled trials. PLOS Digit Health 1(10): e0000123. https://doi.org/10.1371/journal.pdig.0000123

Editor: Laura M. König, University of Bayreuth: Universitat Bayreuth, GERMANY

Received: December 13, 2021; Accepted: September 9, 2022; Published: October 21, 2022

Copyright: © 2022 Moe-Byrne et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are in the manuscript and Supporting Information files.

Funding: The study is funded by the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No. 848180. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist except for CvdFC who received honoraries for lectures from the Loyds Register Foundation and Janssen UK.

Introduction

Background

Mental ill-health conditions such as depression, anxiety, and stress are common in the workplace. They impact employee wellbeing, productivity, absenteeism (sickness absence), and presenteeism [1,2]. Presenteeism is working with difficulty to do the tasks at hand [3]. Originally coined as “showing up at work while being sick” [4] because of chronic medical conditions [5] or because of work or personal characteristics, the emphasis in interpretation has shifted toward worker slowdowns in general and the economic costs associated with that [6]. The prevalence of presenteeism is high, amounting to an average of 40% in a survey conducted amongst workers in 34 countries [7–9].

Globally, mental health problems as a cause of the burden of illness are on the rise and account for a fifth of all years of living disabled [10]. The Global Burden of Disease study reported that in 2017, an estimated 792 million people are affected by mental ill-health worldwide [11] and a 2018 Lancet Commission report on mental health estimates mental health disorders will cost the global economy 16 trillion US dollars by 2030 [12].

Mental ill-health costs employers between £33 billion and £42 billion a year, with an annual cost to the UK economy of between £74 billion and £99 billion [13]. According to a Health and Safety Executive report in 2020 there were an estimated 828,000 workers affected by work-related stress, depression, or anxiety; these three conditions are responsible for 51% of all cases of work-related ill health and 55% of all days lost due to work-related ill health in the UK [14,15]. Work-related stress is a significant risk factor for a range of physical and mental health conditions such as cardiovascular disease, depression and related mortality [16,17]. In addition, a 2020 survey undertaken on 3,614 UK workers reported that 41% of employees surveyed have experienced mental health symptoms that were caused or worsened by work and 51% said their symptoms were due to pressure they experienced at work [18].

Understanding how employee health impacts productivity has advanced substantially in the past two decades [19–22]. According to a survey conducted in 2019, 69% of UK line managers were aware that supporting employee wellbeing is a core management requirement; however, only 13% have received any form of wellbeing or mental health training [23].

Non-digital workplace interventions have been developed to promote employees’ wellbeing and potentially prevent or treat mental disorders [24] and a few studies showed some positive results with workplace interventions such as cognitive-behavioural therapy, mindfulness and relaxation [24–28]. Also, one review which explored the effect on absenteeism and productivity found a small number of studies that suggested some effect for short interventions [29].

In the past decade, the advancement of digital technology has led to the creation of digital health interventions in physical and mental health. Nowadays, web based and smartphone based digital health interventions have become increasingly popular within the health industry due to their easy implementation [30]. In addition, digital interventions have the potential to be scaled up to provide care to large populations and could offer greater anonymity and stigma reduction compared to traditional face-to-face interventions [16,31]. Several reviews published in 2017 and 2018 found that digital interventions could effectively improve employees’ psychological wellbeing and reduce stress at the workplace. They explored the impact of digital interventions without further support from health professionals on health-related outcomes, and most of the interventions examined were not tailored to the individual employee [30,32,33]. Tailoring in this context differs from mode tailoring as described elsewhere [34]. It indicates an automated process that adapts the intervention based upon the employee’s input, for example, by applying an algorithm to the individual responses from the baseline questionnaire to generate specific feedback and provide modules relevant to the user. Also, blended eHealth interventions delivered by healthcare professionals combined with a digital intervention tailoring to the treatment is now available. So far, their application and effect in the workplace setting has not been explored.

This systematic review seeks to evaluate the impact of tailored digital health interventions aiming to improve mental health, stress-related physical symptoms, presenteeism and absenteeism in the workplace in employees in case of mental health issues or work-stress.

Methods

Protocol and registration

The review protocol has been registered in the PROSPERO database under ID CRD42021213292 [35]. This review complies with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines (See S3 Appendix) [36]. This study was funded by the European Union’s Horizon 2020 research and innovation program under grant agreement number 848180.

Inclusion criteria

RCTs, including pilot and feasibility trials, with individual or cluster randomisation and stepped wedge trial designs (SWTD), were eligible if they assessed the impact of digital health interventions compared with standard guidance or waiting list control. The participants had to be employees aged 16 years and above, and the interventions had to be provided in or via the workplace setting.

Included studies had to report interventions that were delivered using digital technology. Digital technology includes digital decision aids or materials delivered through a computer, tablet, smartphone or email. This material could be delivered as a website, app or downloadable software. This digital intervention could be combined with further support such as group sessions, individual counselling, or direct feedback from a health professional (such as an occupational physician, a psychotherapist, a coach in the workplace, or psychiatric consultation), so-called blended e-Health interventions. This further support was not necessary for inclusion, however. The studies had to report on mental health outcomes or work-stress. Additionally, they could report on physical outcomes, presenteeism or absenteeism. We have also included studies related to insomnia as sleep-wake disorders are a classification in the Diagnostic Statistical Manual of Mental disorders (DSM-5) [37]. To be included, a digital intervention needed to have some tailored feedback and content for the individual employee. Feedback from an outside therapist or group, not included in the digital intervention, did not meet this criterion. Also, online training or monitoring without tailoring and personalisation would not meet this criterion.

Exclusion criteria

Studies were excluded if they reported interventions that were not tailored or personalised in some way to the individual employee; if the feedback provided to the participants was not tailored or individualised in some way; or if the participants were provided with material to practice on their own, such as mindful breathing or meditation, with no tailoring based upon client assessment and symptomatology. Feedback from an outside therapist or group, not included in the digital intervention itself, did not count as relevant here.

The studies were also excluded if the interventions solely addressed addiction or consisted of online training/teaching or online education without tailoring. Interventions aimed at physical outcomes, productivity, performance, presenteeism, absenteeism or lifestyle interventions without mental health or work-stress outcomes were also excluded. Studies reporting on cost-effectiveness were out of scope.

Search strategy

Relevant studies were identified by searching the following electronic databases: MEDLINE, EMBASE, PsycINFO, and The Cochrane Central Register of Controlled Trials (CENTRAL) published from 2000 onwards. We searched Google Scholar for grey literatures and the reference lists of all included articles and relevant systematic reviews were also checked to identify potentially eligible studies. Experts who published in the domain of workplace stress, mental health and digital interventions were contacted to suggest any articles or grey literature that they deemed relevant. We did not apply any language restrictions (See S1 Appendix for a complete list of search terms and combinations). The search was supported by the Centre for reviews and Dissemination (CRD) at the University of York.

Study selection

The studies retrieved from the searches were exported into EndNote. After deduplication, the rest of the studies were exported into Rayyan (Rayyan QCRI software for screening) [38]. The first 10% of the titles and abstracts of the studies were screened in duplicate by two independent researchers (TMB and JS) against the inclusion/exclusion criteria. We coded either 1 for inclusion or 0 for exclusion. We used Cohen’s Kappa statistic to calculate the percentage of positive agreement and negative agreement between the reserachers.

We considered the second round of 10% double screening if the kappa score was less than 0.8. However, we have achieved an excellent inter-rater agreement in our first round (Cohen’s Kappa score 0.96), and no potentially eligible studies were missed. Hence, two reviewers continued screening half of each of the remaining studies. Full texts of potentially eligible studies were then screened independently and in duplicate by two reviewers (TMB and JS). Disagreements were resolved through discussion to achieve consensus, and if necessary, a third reviewer (CFC) was consulted. In all cases, a consensus was reached.

Data extraction

Data extraction was carried out using a standardised data extraction form. This extraction form was piloted (by filling in the first two studies) and refined. Data were extracted as follows: study design (e.g., RCT, Cluster randomisation or Stepped wedge), study participants (age, sex) and interventions (population, sample size, intervention/comparator details), outcome measures (outcome description and measures), results, effect size and findings. We also extracted the types of personalisation and tailoring used for participants and the theoretical background upon which the digital intervention materials were based, and created a table showing triage, personalisation, and intervention tailoring used by the individual trials, in the results section. Tailoring methods were extracted from each article, using the original authors language and terminology. Commonalities and overlaps were examined, this process was done inductively based upon the language used in the articles, and five categories of tailoring emerged. They are: 1) Algorithm used to triage based upon assessment or questionnaire scores, 2) Algorithm used for tailored Summarised Feedback, 3) Participant choice, 4) Use of automated messages to users, 5) Interventions blended with in-person support. Table 1 shows each of the five categories.

thumbnail
Download:
Table 1. Tailoring and Personalisation table.

https://doi.org/10.1371/journal.pdig.0000123.t001

Data were extracted in duplicate by two reviewers independently (TMB and JS). Disagreements were resolved through consensus, and if necessary, a third reviewer (CFC) was consulted. Data from studies with multiple publications were extracted and reported as a single study. The study authors were contacted for further information when appropriate.

Quality assessment

The quality of the RCTs was assessed according to the Cochrane Risk of Bias tool (low, high, uncertain) [39] supported by a standardised data extraction form. The tool assessed several sources of bias, including selection, performance, detection, attrition, and reporting bias. The clustered RCTs were assessed using the same tool but amended by adding the following sources of bias (recruitment bias, baseline imbalance, loss of clusters, incorrect analysis, and comparability with individual randomised trials) which are particular to the Clustered RCTs [40].

We produced the overall risk of bias assessment for each study (low, high, uncertain) based on reviewing all sources of bias identified in the individual trial. Two reviewers (TMB and JS) independently assessed the quality of the studies. Disagreements were resolved through consensus, and if necessary, a third reviewer (CFC) was consulted. The funnel plot to assess the publication bias was not conducted as there were less than ten trials in the review [41].

Data analysis

Due to the lack of homogeneity across outcome measures, we did not consider formal meta-analysis a feasible option. Therefore, a narrative approach was used to summarise the findings as presented in tables in the Results section. We have also presented the percentage range for adherence and attrition in each group. We considered less than 25% as low, less than 50% as moderate and 50% and above as good.

Results

Study selection

The study selection is shown in Fig 1 below. A total of 5992 publications were identified through the initial database search, and an additional 49 studies were identified through google scholar and citation search. After the removal of duplicates, 4774 publication titles and abstracts were screened. Fifty studies were eligible for a full-text review. We could not retrieve a full-text review of 1 study as the authors did not reply to our email [42]. Out of 50 studies, 42 were excluded because the interventions were not tailored to the individuals, reported no mental health outcomes, addressed a mixed or wrong population, had no waiting list control, the full text was not available, and some were ongoing trials or a cost-effectiveness study which was out of scope.

thumbnail
Download:
Fig 1. PRISMA 2009 Flow Diagram.

https://doi.org/10.1371/journal.pdig.0000123.g001

Study characteristics

Eight publications (7 studies) met the inclusion criteria: five RCTs [43–46], one crossover RCT [47] and two cluster RCTs [48–50]. Two studies [48,49] were part of a larger trial [51] which used the same data to report different outcomes. One study was conducted in the USA [43], two studies (three publications) in the Netherlands [48–50], two in Germany [44,46] and two in the UK [45,47]. The total number of participants included in this review was 2008, with study sample sizes ranging from 48 to 532. The studies were conducted between 2004 and 2019.

Participant characteristics

Four studies only included participants with particular characteristics. One study included only participants that screened positive for depression, somatisation, or anxiety based on the following assessment tools: the Patient Health Questionnaire for depression (PHQ-9), for somatisation (PHQ-15), and for Generalised Anxiety Disorder (GAD-7); this study also required that participants were on sick-leave between 4 and 26 weeks [50]. One study required participants to screen positive on the General Health Questionnaire (GHQ 12) [45] and to be sick-listed at least 4 weeks; and one required participants screening positive on the Perceived Stress Scale (PSS-10) [44]. A fourth study only included participants who self-identified as having poor sleep [47]. One study excluded participants if they were on sick leave for more than two weeks [48,49]. The remaining two studies did not report any specific inclusion/exclusion criteria [43,46].

Participants were recruited from a variety of workplaces. Four studies recruited the employees from organisations such as an insurance company, a technology company, or large corporations [43,44,47,50]. Two studies recruited healthcare professionals from local hospitals [45,48,49]. One study recruited employees from the private and public sector companies in three countries [46].

Intervention characteristics

Three studies (four publications as mentioned above) included in this review designed their digital interventions to increase wellbeing and act as preventative interventions for mental health problems such as anxiety and depression, rather than treating these conditions [43,46,48,49]. One focused on work performance and provided guidance to the Occupational Physician to address any mental wellbeing issues together with the GP, as a blended eHealth model [50]. The length of the administration of the interventions ranged from four weeks to three months. All studies included in this review compared a web-based intervention, [43,44,48–50] a smartphone app intervention [46] and a combined web based and smartphone app intervention [47] with waiting list or usual guidance control. All the studies stated the use of Cognitive Behavioural Therapy (CBT) techniques as theoretical background for the materials chosen and content uploaded into the digital intervention. Additionally, two stated the use of Mindfulness-based content and materials [46,48,49]. Other studies used the transactional stress model or the job demands-resources (JDR) model as background [44,46].

Triage used in the application of digital support systems refers to the assessment by questionnaires to gauge the mental and or physical state of the participant, which informs the tailoring algorithm.

Table 1 below shows each of the five categories and which articles used which tailoring tool.

The use of an Algorithm refers to programmed instructions in the digital intervention that guide the assignment of materials or the generation of feedback. Digital interventions that use algorithms, automate, and personalise the feedback and content that a user sees. The algorithm guides content assignment based on the users’ feedback, assessment scores, actions, or behaviour within the digital intervention. Guidance can differ depending on the focus of the digital intervention. For example, with the Kelaa Mental Resilience App [46], the digital intervention used sensors built in users smartphones to track the users’ sleep and then provide detailed feedback. One study provided support from participants by occupational physicians who received instructions and decision support from the digital intervention, based upon the tailoring [50]. This was the only study evaluating a blended digital intervention. Two studies allowed digital intervention to be combined with regular counselling and medication, or “treatment as usual” [44,45]; this is not considered blended care as the real life treatment is not influenced by the digital intervention.

The characteristics for tailoring are shown in Table 1 and S2 Appendix.

Outcome measures

Most studies reported a mix of psychological (such as depression, anxiety, and stress) and work-related outcome measures (such as absenteeism and presenteeism). Several studies reported physical measures: two studies reported outcomes relevant to sleeping problems [44,46] and one measured the amount of sleep and workplace performance [47]. One study reported physical symptoms related to somatisation [50] and the other two reported levels of physical health impairment using self-reported SF12 and SF 36 questionnaires [44,46]. However, none of the studies reported on long-term medical conditions such as diabetes or cardiovascular disease. Regarding the crossover trial, we only used first period data prior to crossover [47].

Risk of bias

Table 2 and Fig 2 show full details of the risk of bias across all studies. Overall, two studies out of seven were judged to be at low risk of bias [44,50]. For the remaining four studies, three were judged to be at unclear risk of bias [45,47–49] and two were at high risk of bias [43,46].

thumbnail
Download:
Fig 2. Risk of Bias Visualization Plot.

https://doi.org/10.1371/journal.pdig.0000123.g002

thumbnail
Download:
Table 2. Risk of bias in included studies.

https://doi.org/10.1371/journal.pdig.0000123.t002

Physical outcomes

As physical outcomes, no study included in this review published actual measurements of physical conditions, such as cardiovascular disease. However, stress-related physical symptoms related to somatisation and sleep impairment as indicated by study participants were reported.

Sleep impairment

Three studies reported sleep impairment using the following assessment: Sleep Condition Indicator (SCI) [47], Insomnia Severity Index (ISI) [44], and the Sleeping Troubles Copenhagen Psychosocial Questionnaire-Revised Version (COPSOQ II) [46]. One study which included self-identified “poor” sleep participants, reported statistically significant differences in the intervention group compared to the control at eight weeks [47]. The other study, which included participants with high stress scores, showed significant effect in favour of the intervention at seven weeks and six months [44]. The third study, which measured sleep impairment, did not significantly differ between the control and intervention groups [46].

Physical health impairment

Two studies reported physical health levels, using self-reported SF 12 and SF 36 v2 [52–54]. Both studies found no statistically significant differences between the intervention and control [44,46].

Somatisation

One study reported physical symptoms related to somatisation using the Patient Health Questionnaire-15 (PHQ-15) screening tool [55]. There were no statistically significant differences between the eHealth intervention group and the control at one- and three-months post-intervention. But there was a significant improvement of somatisation related physical symptoms in the eHealth intervention group compared to control at 6 and 9 months (p = 0.017 and p = 0.039) [50].

Psychological outcomes

All seven studies reported psychological outcomes [43–50]. Six studies reported measures of anxiety and depression symptoms, and four studies reported on stress levels, which also included posttraumatic stress [43,44,46,49].

Anxiety and depression

Six studies measured anxiety and depression outcomes using various assessment tools: the Beck Anxiety Inventory (BAI) [43], the Brief Symptom Inventory (BSI) [48], the Hospital Anxiety and Depression Scale (HADS) [44,45], the Generalised Anxiety Disorder-2(GAD-2) [47], the Patient Health Questionnaire-9 (PHQ-9), the Generalised Anxiety Disorder-7 (GAD-7) [50], and the Centre for Epidemiologic Studies Depression Scale-Revised (CES-D) [43,44]. Out of six, four studies did not find a statistically significant difference between the digital intervention and control condition in anxiety and depression [43,47,48,50].

In contrast, two studies which included employees with higher levels of psychological distress showed significant improvement with digital intervention compared to control [44,45]. The study which worked following the Lazarus and Folkman’s ’Transactional Model of Stress’ [44] included only higher stress level participants in their sample (22 or more on the Perceived Stress Scale (PSS-10)). It showed a significant reduction of anxiety and depression at seven weeks and six months (p<0.001 for both anxiety and depression) [44]. Another study [45] which included only sick listed employees of a UK hospital (4 or more on the General Health Questionnaire 12 (GHQ-12)) showed significantly lower depression scores in the intervention group at the end of the treatment and one-month post-treatment (p = 0.028 and p = 0.04), and lower anxiety scores at one-month post-treatment (p = 0.021), compared to wait-list control. However, at 3- and 6-months post-treatment, these differences were no longer significant.

Stress

Four studies reported stress outcomes using the different assessment tools (the symptoms of distress scale, the Perceived Stress Score (PSS-10) [56], the Copenhagen Psychosocial Questionnaire-Revised Version (COPSOQ II) [57], the posttraumatic Impact of Event Scale (IES) [58] and the Four Dimensional Symptoms Questionnaire (4DSQ) [43,44,46,49]). Three out of four studies in the intervention group showed significantly lower stress scores than the wait-list control at different time points [43,44,46]. One study showed a significant reduction in stress score at three months (p<0.023) [43] and the other study at seven weeks and six months (p<0.001) [44]. One study which used the JDR model of burnout and measured both general and cognitive stress between the two groups (significant group*time interactions) also showed a significant reduction in stress over time in the intervention group (p<0.001 general stress, p<0.01 cognitive stress) [46]. In contrast, a study [48,49] which included a subgroup analysis of participants with high stress levels, measured both posttraumatic stress and distress did not show significant improvement between the digital intervention group and the control group at six months follow up.

Wellbeing

Two studies reported employees’ wellbeing using the Warwick- Edinburgh Mental Wellbeing Scale [46] and WHO-5 wellbeing index [48] with mixed results. One study reported that participants in the intervention group showed significantly more wellbeing over time (significant group*time interaction) than the wait-list control group at six weeks follow up [46]. However, the other did not find a statistically significant positive effect with digital intervention on wellbeing at six months [48].

Other mental-health related outcomes

Three studies reported other mental health-related outcomes such as, resilience, worry and positive mental health, which measured emotional, psychological and social wellbeing [44,46,48,49]. One study found no statistically significant differences between intervention and control in resilience [46]. Two other studies found significant effects in favour of digital intervention in relevant to positive mental health (F = 3.46, p = 0.03, Cohen’s d = 0.37 at three months follow-up, 0.28 at six months follow-up) [48] and employee’s worry and quality of life regarding mental health (p <0.001 at six months) [44]. The results are shown in Table 3.

thumbnail
Download:
Table 3. Overview of outcomes of the included studies.

https://doi.org/10.1371/journal.pdig.0000123.t003

Work-related outcomes

Seven publications (six studies) reported work-related outcomes [43,44,46–50] regarding presenteeism and absenteeism in several ways and using several instruments, which is summarised in Table 4 below. Only significant findings are shown.

thumbnail
Download:
Table 4. Work-related outcomes.

https://doi.org/10.1371/journal.pdig.0000123.t004

Presenteeism

Five studies reported some form of work performance that in this review were taken as an expression of presenteeism, which illustrates that the definition and operationalisation of presenteeism is not comparable between the instruments and consequently studies. Some studies used two instruments [44,48,49]. Two studies using the Utrecht Work Engagement Scale (UWES) [44,48] reported a small positive effect on work engagement in the intervention group (F = 3.44, p = 0.03, Cohen’s d = 0.25 three months follow-up, 0.15 six months follow-up) [48] and a statistically significant difference between groups in favour of the intervention at the seven week follow up (F = 5.4, p<0.05, Cohen’s d = 0.17) but not at six months [44]. One study [49] used an item of the Work Ability Index in which an individual assesses their own workability by comparing their actual one with the highest ever; and another instrument, the Nurses Work Functioning Questionnaire (NWFQ). This consists of 7 subscales: (1) cognitive aspects of task execution and general incidents, (2) impaired decision making, (3) causing incidents at work, (4) avoidance behaviour, (5) conflicts and annoyances with colleagues, (6) impaired contact with patients and their family and (7) lack of energy and motivation. This study did not find any significant differences between the intervention and control group [49]. Three other studies collected the data for presenteeism using the WPAI and the Work Limitations Questionnaire [43,46]; one did not analyse the data [46], one found a statistically significant difference in presenteeism (p = 0.001) [47] and the other found no significant difference in work productivity between the web-based intervention and wait-list control at three months [43]. One study that assessed presenteeism with a dedicated instrument at 6 months (TiC-P) [44] found a statistically significant improvement in presenteeism (p<0.01).

Absenteeism

One study used the Treatment Inventory Cost in Psychiatric Patients (TIC-P) combined with social insurance data to assess Return to Work (RTW) on sick-listed employees with common mental disorders. They reported absenteeism as duration until first return to work and complete return to work after a long-term sickness. The duration until the first RTW in the intervention group was significantly shorter than in the control group (P = .03). The median duration was 77.0 (IQR 29.0–152.3) days in the control group and 50.0 (IQR 20.8–99.0) days in the intervention group, a difference of 27 days (hazard ratio [HR] 1.390, 95% CI 1.034–1.870, P = .03) at 12 months [50]. Two studies assessing absenteeism with a dedicated instrument at 8 weeks (WPAI) [47] and 6 months (TiC-P) [44] found no statistically significant difference.

Adherence and uptake

In terms of adherence and uptake to the digital intervention, all studies mentioned some kind of adherence level. Five studies reported that participants in the digital intervention groups logged in to the intervention materials at least one time (range 16% to 82%) [43,44,46,48–50]. Two studies reported participants in the intervention group either attended or completed four or more sessions of the study materials [45,47]. Four studies [43,44,47,50] reported good uptake, one of them with partly moderate uptake of the future planning/booster parts [44]. One study [48,49] reported low uptake, and one study moderate uptake [46]. Follow up adherence was unknown in two studies [43,44], very low in one study [48,49], good in one study [47], very good in one [45], and moderate in two [46,50]. In general, uptake was better than adherence. Of all studies, only two had both good uptake and moderate to good adherence [47,50]. The findings regarding adherence and uptake are shown in Table 5.

thumbnail
Download:
Table 5. Adherence and Uptake.

https://doi.org/10.1371/journal.pdig.0000123.t005

Attrition rates

Six out of seven studies reported dropout rates for follow up which were calculated based on numbers at baseline compared with those included in the final analysis for each study arm [44–50]. The employees in the intervention arm were more likely to drop out compared to the control arm (drop out ranged from 17% to 61% in the intervention arm and 0% to 36% in the control arm).

Discussion

Summary of the main results

This review of seven randomised controlled trials evaluated tailored digital interventions versus waiting list control or usual guidance to improve stress in the workplace yielded mixed results.

The study outcomes were positive in terms of presenteeism, sleep, stress levels and physical symptoms related to somatisation. However, they are less so for addressing depression and anxiety and absenteeism.

At first glance, this seems understandable, as the interventions addressed work stress, which is associated with presenteeism. However, this does not automatically lead to improvement in more serious conditions such as depression, anxiety, or sickness absence.

At second glance, although tailored digital interventions did not reduce anxiety and depression in the general working population, they significantly reduced depression and anxiety in employees with higher levels of psychological distress. Moreover, most studies did not show improvement in absenteeism, they did show faster return to work in employees on long-term sick leave who received a tailored intervention. This suggests that tailored digital interventions can be of use for preventive approaches for stress, sleep, presenteeism and somatisation in the workplace, and for more serious depressive and anxiety symptoms and absenteeism. General preventive measures work well to address work stress. But relieving work stress in itself does not reduce depressive or anxiety disorders per se. For that, a tailored approach is needed, that focuses on treatment of depression and anxiety and this is provided in the group with serious depressive and anxiety disorders. So this is a benefit of the tailoring of the intervention.

The literature about interventions to address mental disorders in the workplace shows that treatment for mental disorders does not improve work functioning automatically. For that, separate interventions are needed that address work functioning. This can be achieved by triage and tailoring. Given the results, tailoring seems to be an approach that can bear fruit [59–62]. Technology and clinical practice can complement each other in blended eHealth. For example, the blended study [50] found improved physical symptoms suggesting somatisation and improved first return to work. This study provided the employees with self-help modules and monitored progress based upon their self-reported outcomes. It gave the occupational physician guiding the employee to return to work tailored suggestions to improve guidance and refer for treatment of medical conditions or psychological symptoms if needed. This is an example of a blended digital intervention that may have particular benefits in employees with more severe symptoms and absenteeism.

An advantage of digital interventions for employees in psychological distress may be that participants remain anonymous by delivering the interventions through the internet [63]. This has the potential to reduce stigma associated with addressing mental health problems. The intervention can also be accessible at any time and place and is easily scalable [16,31]. These characteristics enhance the potential of reaching a greater proportion of the eligible population while being cost effective as only a small increase of resources is required [30].

Our systematic literature review indicates presenteeism seems to be a significant outcome for digital tailored interventions in the workplace. However, the definition and operationalisation using different instrument for measurement of presenteeism varies. Hence, transparency and harmonization of the definition and operationalisation of presenteeism in (future) research is needed [64]. Moreover, digital interventions at the workplace are often subject of cost-effectiveness analyses.

Digital intervention tailoring and personalisation

One thing that sets this systematic review apart from earlier ones is the focus on digital interventions which offer tailoring for participants. Personalisation is essential to developing a tailored digital intervention and may influence its effectiveness. The type of personalisation and tailoring offered by a digital intervention and how it targets mental health symptoms impacts the user experience. Traditional in-person treatment is highly personalised, based on clinician and patient interaction. Interventions or techniques can be changed by the clinician at any time, allowing flexibility in care. While digital interventions can never be equivalent, there is the possibility for them to provide effective targeted support, either alongside in-person treatment (blended treatment) or as an alternative [65].

Adherence

Apart from one study [48,49], most studies had moderate or good uptake whereas follow up adherence varied. It was not reported in two studies [43,44], and in the remainder of studies ranged from very low to very good. The general adherence was better than in general in non-tailored digital programs, and it is possible that this may have to do with the tailoring provided as tailoring makes it worthwhile for the employee to engage with the digital intervention repeatedly [66].

Strengths and limitations

The primary strength of this study is that we included data without language restriction from studies identified by a comprehensive search of the published literature. We included all studies exploring the effect of both physical and mental components outcomes. As there have been no reviews on tailored digital interventions for employees with work-related stress before, this systematic review is innovative.

Even though this review found improvements in various symptoms and work productivity as a result of the digital intervention, there are several limitations. Firstly, we only searched google scholar for grey literatures and there was a possibility that some studies could have been missed due to double screening for only 10% of title and abstract [67,68]. Secondly, caution is advised when interpreting the results, as only two of the seven studies were of high quality and there was a variety of outcome measures. Thirdly, the results of most of the studies relied exclusively on self-report measures over a relatively short term, except for one study [50]. There was one study that we could not report on as shown in the flowchart, as despite our efforts to contact the author, we could not include that study due to a lack of response. Fourthly, there was considerable heterogeneity in assessment tools, outcomes, components of digital interventions, and time points, making meta-analysis impossible to perform. Especially, in the seven studies evaluating work productivity, six different instruments were used, most of them not validated. Finally, many of the included studies were conducted in high-income countries; therefore, our findings may not be generalisable to low-income countries.

Recommendations for research

Given the promising results, further research into tailored digital interventions to address work stress is needed. Also, tailor interventions should follow the reporting standards recommended by Harrington and Noar (2012) [69]. Given the scarcity of digital interventions evaluated by research so far, this should be a research priority and including blended aspects in such interventions may be beneficial. It would be recommended to use validated questionnaires to assess work-productivity i.e., TiC-P [70]. Furthermore, future studies should investigate the durability of these effects over longer periods of time. Studies also should report on adherence to the intervention, both in terms of how often employees log in, and whether they finish the intervention. Moreover, more broad reflection on factors affecting the effectiveness of such intervention is needed. It should include such phenomena as individual motivation to solve their own problems, organisational culture and type of work performed, educational level, attitudes towards mental health issues, stigma and many others.

Conclusions

This review shows that tailored digital interventions have promising results in terms of presenteeism, stress levels, sleep and physical symptoms related to somatisation but less so for absenteeism. They did not reduce anxiety and depression in the general working population; however, they can significantly reduce depression and anxiety in employees with higher levels of psychological distress. There is a need for uniformity in the use of assessment tools, outcomes, and proper reporting of components of digital interventions in research in this domain, especially regarding work productivity.

Acknowledgments

We would like to thank Kath Wright (retired) from the Centre for Reviews and Dissemination at the University of York for developing and running the search strategy and deduplication the records. We also like to thank Dr Bostock and Prof. Dr. Van der Feltz-Cornelis for providing requested data.

Also, The EMPOWER Consortium which includes (in alphabetic order): Andysz A14; Ayuso-Mateos JL2,7; Bernard RM10; Borrega O9; Brach M10; Cabello M7; Cabré J9; Chen T8; Cristóbal P1; de Miquel C1; de Mul M4; Félez M1; Gevaert J11; González L9; Gutiérrez D1; Hakkaart-van Roijen L4; Haro JM1,2; Leonardi M12; Lima RA1; López-Carrión M9; Lukersmith S8; Mauro A13; Merezc-Kot D5; Miret M1; Naumanen P6; Olaya B1,2; Ortiz-Tello A7; Porcheddu D13; Raggi A12; Rodríguez-McGreevy K7; Sabariego C10; Salvador-Carulla L8; Seghezzi F13; Shepherd J3; Sinokki M6; Smith N3; Staszewska K14 Tiraboschi M13; Toppo C12; van Aerden K11; Van der Feltz-Cornelis C3; van Krugten F4; Vanroelen C11; Vorstenbosch E1,2;

1Research, Innovation and Teaching Unit, Parc Sanitari Sant Joan de Déu, Sant Boi de Llobregat, Spain

2 Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM), Madrid, Spain

3MHARG, Department of Health Sciences, Hull York Medical School, University of York, York, United Kingdom

4Erasmus School of Health Policy and Management (ESHPM), Erasmus University Rotterdam, Rotterdam, the Netherlands

5 Institute of Psychology, University of Lodz, Lodz, Poland

6 Turku Centre for Occupational Health, University of Turku, Turku, Finland

7 Department of Psychiatry, Faculty of Medicine, Universidad Autónoma de Madrid, Madrid, Spain

8 Centre for Mental Health Research, Research School of Population Health, Australian National University, Canberra, Australia

9 Òmada Interactiva, SLL, Barcelona, Spain

10 Swiss Paraplegic Research (SPF), Nottwil, Switzerland

11 Interface Demography, Vrije Universiteit Brussel, Brussels, Belgium

12 Fondazione IRRCS Istituto Neurologico Carlo Besta, UO Neurologia Salute Pubblica e Disabilità. Milano, Italy

13 Fondazione ADAPT, Milano, Italy

14 Nofer Institute of Occupational Medicine, Lodz, Poland

Reference

  1. 1. Glozier N. Mental ill health and fitness for work. Occupational and Environmental Medicine. 2002;59(10):714–20. pmid:12356935
  2. 2. Harvey SB, Henderson M, Lelliott P, Hotopf M. Mental health and employment: much work still to be done. The British Journal of Psychiatry. 2009;194(3):201–3. pmid:19252144
  3. 3. Aronsson G, Gustafsson K, Dallner M. Sick but yet at work. An empirical study of sickness presenteeism. Journal of epidemiology and community health. 2000;54(7):502–9. pmid:10846192.
  4. 4. Dew K, Keefe V, Small K. ’Choosing’ to work when sick: workplace presenteeism. Soc Sci Med. 2005;60(10):2273–82. Epub 2005/03/08. pmid:15748675.
  5. 5. Burton WN, Pransky G, Conti DJ, Chen CY, Edington DW. The association of medical conditions and presenteeism. J Occup Environ Med. 2004;46(6 Suppl):S38–45. Epub 2004/06/15. pmid:15194894.
  6. 6. Lang S. Economists coin new word, ’presenteeism,’ to describe worker slowdowns that account for up to 60 percent of employer health costs 2004 [cited 2021 July 8]. Available from: https://news.cornell.edu/stories/2004/04/new-word-job-health-problem-presenteeism.
    • 7. Hemp P. Presenteeism: At Work—But Out of It 2004. Available from: https://hbr.org/2004/10/presenteeism-at-work-but-out-of-it.
      • 8. Aronsson G, Gustafsson K. Sickness presenteeism: prevalence, attendance-pressure factors, and an outline of a model for research. J Occup Environ Med. 2005;47(9):958–66. Epub 2005/09/13. pmid:16155481.
      • 9. Robertson-Smith GM, Carl. Employee engagement: a review of current thinking2009.
        • 10. Vos T, Allen C, Arora M, Barber RM, Bhutta ZA, Brown A, et al. Global, regional, and national incidence, prevalence, and years lived with disability for 310 diseases and injuries, 1990–2015: a systematic analysis for the Global Burden of Disease Study 2015. The Lancet. 2016;388(10053):1545–602. https://doi.org/10.1016/S0140-6736(16)31678-6.
        • 11. Dattani S, Ritchie H, Roser M. Mental Health 2021 [cited 2021 July 1]. Available from: https://ourworldindata.org/mental-health.
          • 12. Canady V. Report: Mental illness will cost the world $16 trillion (USD) by 2030. Mental Health Weekly. 2018;28(39):7–8.
          • 13. Stevenson DF, P. Thriving at work:The Stevenson / Farmer review of mental health and employers 2017 [cited 2021 June 24]. Available from: https://www.gov.uk/government/publications/thriving-at-work-a-review-of-mental-health-and-employers.
            • 14. Executive; HaS. Work-related stress, anxiety or depression statistics in Great Britain, 2020 2020 [cited 2021 July 13]. Available from: www.hse.gov.uk/statistics/.
              • 15. Knapp M, McDaid D, Parsonage M. Mental health promotion and mental illness prevention: The economic case. 2011.
                • 16. Ebert DD, Lehr D, Smit F, Zarski AC, Riper H, Heber E, et al. Efficacy and cost-effectiveness of minimal guided and unguided internet-based mobile supported stress-management in employees with occupational stress: a three-armed randomised controlled trial. BMC public health. 2014;14:807. pmid:25099533
                • 17. NHS Staff Survey Results. Results for the 2020 NHS Staff Survey 2020 [cited 2021 June 30]. Available from: https://www.nhsstaffsurveys.com/Page/1105/Latest-Results/NHS-Staff-Survey-Results/.
                  • 18. Business in the Community. Mental Health at Work 2020: key findings 2020 [cited 2021 June 29]. Available from: https://www.bitc.org.uk/report/mhaw2020/.
                    • 19. Rizzo JA, Zyczynski TM, Chen J, Mallow PJ, Trudel GC, Penrod JR. Lost labor productivity costs of prostate cancer to patients and their spouses: evidence from US national survey data. Journal of occupational and environmental medicine. 2016;58(4):351–8. pmid:27058474
                    • 20. Ozminkowski RJ, Wang S, Walsh JK. The direct and indirect costs of untreated insomnia in adults in the United States. Sleep. 2007;30(3):263–73. pmid:17425222
                    • 21. Henke RM, Carls GS, Short ME, Pei X, Wang S, Moley S, et al. The relationship between health risks and health and productivity costs among employees at Pepsi Bottling Group. Journal of occupational and environmental medicine. 2010;52(5):519–27. pmid:20431407
                    • 22. Durden ED, Huse D, Ben-Joseph R, Chu B-C. Economic costs of obesity to self-insured employers. Journal of occupational and environmental medicine. 2008;50(9):991–7. pmid:18784546
                    • 23. Business in the Community. Mental Health at Work 2019: Time To Take Ownership 2019 [cited 2021 June 29]. Available from: https://www.bitc.org.uk/report/mental-health-at-work-2019-time-to-take-ownership/.
                      • 24. European Commission. EU-Compass for Action on Mental Health and Well-being 2015 [cited 2021 July 8]. Available from: https://ec.europa.eu/health/non_communicable_diseases/mental_health/eu_compass_en.
                        • 25. Merrill RM, Aldana SG, Garrett J, Ross C. Effectiveness of a workplace wellness program for maintaining health and promoting healthy behaviors. J Occup Environ Med. 2011;53(7):782–7. Epub 2011/06/15. pmid:21670705.
                        • 26. Patel DN, Lambert EV, da Silva R, Greyling M, Nossel C, Noach A, et al. The association between medical costs and participation in the vitality health promotion program among 948,974 members of a South African health insurance company. Am J Health Promot. 2010;24(3):199–204. Epub 2010/01/16. pmid:20073387.
                        • 27. Baicker K, Cutler D, Song Z. Workplace wellness programs can generate savings. Health Aff (Millwood). 2010;29(2):304–11. Epub 2010/01/16. pmid:20075081.
                        • 28. Widmer RJ, Allison TG, Keane B, Dallas A, Bailey KR, Lerman LO, et al. Workplace Digital Health Is Associated with Improved Cardiovascular Risk Factors in a Frequency-Dependent Fashion: A Large Prospective Observational Cohort Study. PLoS One. 2016;11(4):e0152657. Epub 2016/04/20. pmid:27092940.
                        • 29. Tarro L, Llauradó E, Ulldemolins G, Hermoso P, Solà R. Effectiveness of Workplace Interventions for Improving Absenteeism, Productivity, and Work Ability of Employees: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. Int J Environ Res Public Health. 2020;17(6). Epub 2020/03/19. pmid:32183392; PubMed Central PMCID: PMC7142489.
                        • 30. Stratton E, Lampit A, Choi I, Calvo RA, Harvey SB, Glozier N. Effectiveness of eHealth interventions for reducing mental health conditions in employees: A systematic review and meta-analysis. PLoS ONE. 2017;12(12):e0189904. pmid:29267334
                        • 31. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res. 2006;8(2):e10. Epub 2006/07/27. pmid:16867965; PubMed Central PMCID: PMC1550698.
                        • 32. Howarth A, Quesada J, Silva J, Judycki S, Mills PR. The impact of digital health interventions on health-related outcomes in the workplace: A systematic review. Digit Health. 2018;4:2055207618770861. Epub 2018/06/27. pmid:29942631; PubMed Central PMCID: PMC6016571.
                        • 33. Carolan S, Harris PR, Cavanagh K. Improving Employee Well-Being and Effectiveness: Systematic Review and Meta-Analysis of Web-Based Psychological Interventions Delivered in the Workplace. J Med Internet Res. 2017;19(7):e271. pmid:28747293
                        • 34. Nguyen MH, Smets EMA, Bol N, Loos EF, Van Weert JCM. How Tailoring the Mode of Information Presentation Influences Younger and Older Adults’ Satisfaction with Health Websites. J Health Commun. 2018;23(2):170–80. Epub 2018/01/19. pmid:29345531.
                        • 35. The effectiveness of digital health interventions on employees with work-stress, physical or mental health problems at the workplace: a systematic review [Internet]. 2021. Available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021213292.
                          • 36. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. Epub 2009/07/22. pmid:19621072; PubMed Central PMCID: PMC2707599.
                          • 37. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5): American Psychiatric Pub; 2013.
                            • 38. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Systematic Reviews. 2016;5(1):210. pmid:27919275
                            • 39. Higgins JPT, Altman DG, Sterne JACe. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Churchill R, Chandler J, Cumpston MS (editors): Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017), Cochrane; 2017 [cited 2021 July 21]. Available from: https://training.cochrane.org/handbook.
                              • 40. Higgins JP, Deeks J, Altman DGe. Chapter 16: Specialtopics in statistics. In: Higgins JPT, Green S (editors): CochraneHandbook for Systematic Reviews of Interventions Version5.1.0 (updated March 2011), Cochrane; 2011 [cited 2021 July 21]. Available from: www.handbook.cochrane.org.
                                • 41. Sterne JAC, Egger M, Moher D, Boutron Ie. Chapter 10: Addressing reporting biases. In: Higgins JPT, Churchill R, Chandler J, Cumpston MS(editors): Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017), Cochrane; 2017 [cited 2021 July 13]. Available from: https://training.cochrane.org/handbook.
                                  • 42. Baharav A, Niejadlik K. 0410 Impact of Digital Monitoring, Assessment, and Cognitive Behavioral Therapy on Subjective Sleep Quality, Workplace Productivity and Health Related Quality of Life. Sleep. 2018;41(suppl_1):A156–A.
                                  • 43. Billings DW, Cook RF, Hendrickson A, Dove DC. A web-based approach to managing stress and mood disorders in the workforce. Journal of occupational and environmental medicine. 2008;50(8):960–8. pmid:18695455
                                  • 44. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, et al. Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occupational and environmental medicine. 2016;73(5):315–23. pmid:26884049
                                  • 45. Grime PR. Computerized cognitive behavioural therapy at work: a randomized controlled trial in employees with recent stress-related absenteeism. Occupational medicine (Oxford, England). 2004;54(5):353–9. pmid:15289593
                                  • 46. Weber S, Lorenz C, Hemmings N. Improving stress and positive mental health at work via an app-based intervention: A large-scale multi-center randomized control trial. Frontiers in Psychology. 2019;10. https://doi.org/10.3389/fpsyg.2019.02745.
                                  • 47. Bostock S, Luik AI, Espie CA. Sleep and Productivity Benefits of Digital Cognitive Behavioral Therapy for Insomnia: A Randomized Controlled Trial Conducted in the Workplace Environment. Journal of occupational and environmental medicine. 2016;58(7):683–9. pmid:27257747
                                  • 48. Bolier L, Ketelaar SM, Nieuwenhuijsen K, Smeets O, Gartner FR, Sluiter JK. Workplace mental health promotion online to enhance well-being of nurses and allied health professionals: A cluster-randomized controlled trial. Internet Interventions. 2014;1(4):196–204. https://doi.org/10.1016/j.invent.2014.10.002.
                                  • 49. Ketelaar SM, Nieuwenhuijsen K, Gartner FR, Bolier L, Smeets O, Sluiter JK. Effect of an E-mental health approach to workers’ health surveillance versus control group on work functioning of hospital employees: a cluster-RCT. PloS one. 2013;8(9):e72546. pmid:24069148
                                  • 50. Volker D, Zijlstra-Vlasveld MC, Anema JR, Beekman ATF, Brouwers EPM, Emons WHM, et al. Effectiveness of a blended Web-based intervention on return to work for sick-listed employees with common mental disorders: Results of a cluster randomized controlled trial. Journal of Medical Internet Research. 2015;17(5):No-Specified. pmid:25972279
                                  • 51. Gartner FR, Ketelaar SM, Smeets O, Bolier L, Fischer E, van Dijk FJH, et al. The Mental Vitality @ Work study: design of a randomized controlled trial on the effect of a workers’ health surveillance mental module for nurses and allied health professionals. BMC public health. 2011;11:290. pmid:21569282
                                  • 52. Turner-Bowker D, Hogue SJ. Short Form 12 Health Survey (SF-12). In: Michalos AC, editor. Encyclopedia of Quality of Life and Well-Being Research. Dordrecht: Springer Netherlands; 2014. p. 5954–7.
                                    • 53. SF-36v2. In: Michalos AC, editor. Encyclopedia of Quality of Life and Well-Being Research. Dordrecht: Springer Netherlands; 2014. p. 5940-.
                                      • 54. Jenkinson C, Stewart-Brown S, Petersen S, Paice C. Assessment of the SF-36 version 2 in the United Kingdom. Journal of Epidemiology & Community Health. 1999;53(1):46–50. pmid:10326053
                                      • 55. Kroenke K, Spitzer RL, Williams JB. The PHQ-15: validity of a new measure for evaluating the severity of somatic symptoms. Psychosom Med. 2002;64(2):258–66. Epub 2002/03/27. pmid:11914441.
                                      • 56. Cohen S, Kamarck T, Mermelstein R. A Global Measure of Perceived Stress. Journal of Health and Social Behavior. 1983;24(4):385–96. pmid:6668417
                                      • 57. Pejtersen JH, Kristensen TS, Borg V, Bjorner JB. The second version of the Copenhagen Psychosocial Questionnaire. Scandinavian journal of public health. 2010;38(3_suppl):8–24. pmid:21172767
                                      • 58. Horowitz M, Wilner N, Alvarez W. Impact of Event Scale: A measure of subjective stress. Psychosomatic medicine. 1979;41(3):209–18. pmid:472086
                                      • 59. Schene AH, Koeter MW, Kikkert MJ, Swinkels JA, McCrone P. Adjuvant occupational therapy for work-related major depression works: randomized trial including economic evaluation. Psychol Med. 2007;37(3):351–62. Epub 2006/11/23. pmid:17112401.
                                      • 60. Blonk RWB, Brenninkmeijer V, Lagerveld SE, Houtman ILD. Return to work: A comparison of two cognitive behavioural interventions in cases of work-related psychological complaints among the self-employed. Work & Stress. 2006;20(2):129–44.
                                      • 61. van der Klink JJL, Blonk RWB, Schene AH, van Dijk FJH. Reducing long term sickness absence by an activating intervention in adjustment disorders: a cluster randomised controlled design. Occupational and environmental medicine. 2003;60(6):429–37. pmid:12771395.
                                      • 62. Nieuwenhuijsen K, Bültmann U, Neumeyer-Gromen A, Verhoeven AC, Verbeek JH, van der Feltz-Cornelis CM. Interventions to improve occupational health in depressed people. Cochrane Database Syst Rev. 2008;(2):Cd006237. Epub 2008/04/22. pmid:18425942.
                                      • 63. Carolan S, de Visser RO. Employees’ Perspectives on the Facilitators and Barriers to Engaging With Digital Mental Health Interventions in the Workplace: Qualitative Study. JMIR mental health. 2018;5(1):e8–e. pmid:29351900.
                                      • 64. Hubens K, Krol M, Coast J, Drummond MF, Brouwer WBF, Uyl-de Groot CA, et al. Measurement Instruments of Productivity Loss of Paid and Unpaid Work: A Systematic Review and Assessment of Suitability for Health Economic Evaluations From a Societal Perspective. Value in Health. 2021;24(11):1686–99. pmid:34711370
                                      • 65. Lustria ML, Cortese J, Noar SM, Glueckauf RL. Computer-tailored health interventions delivered over the Web: review and analysis of key components. Patient Educ Couns. 2009;74(2):156–73. Epub 2008/10/25. pmid:18947966.
                                      • 66. Riper H, Spek V, Boon B, Conijn B, Kramer J, Martin-Abello K, et al. Effectiveness of E-Self-help Interventions for Curbing Adult Problem Drinking: A Meta-analysis. J Med Internet Res. 2011;13(2):e42. pmid:21719411
                                      • 67. Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19(1):132. Epub 2019/06/30. pmid:31253092; PubMed Central PMCID: PMC6599339.
                                      • 68. Stoll CRT, Izadi S, Fowler S, Green P, Suls J, Colditz GA. The value of a second reviewer for study selection in systematic reviews. Res Synth Methods. 2019;10(4):539–45. Epub 2019/07/05. pmid:31272125; PubMed Central PMCID: PMC6989049.
                                      • 69. Harrington NG, Noar SM. Reporting standards for studies of tailored interventions. Health Education Research. 2011;27(2):331–42. pmid:22156230
                                      • 70. Bouwmans C, De Jong K, Timman R, Zijlstra-Vlasveld M, Van der Feltz-Cornelis C, Tan Swan S, et al. Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss in patients with a psychiatric disorder (TiC-P). BMC Health Serv Res. 2013;13:217. Epub 2013/06/19. pmid:23768141; PubMed Central PMCID: PMC3694473.