Bias in clinical models can emanate from multiple sources

The data underlying clinical models may contain biases that can be unknowingly propagated to downstream inference and prediction tasks [1,2]. Such biases can emanate from multiple sources–ranging from differences in how physicians report information in EHR and clinical notes to artifacts in images to the miscalibration of medical devices–but they can also reflect existing social determinants of health. In other words, biases encountered in clinical data are often intersectional in nature, i.e., both social and technological [3]. For example, skin tone affects the accuracy of pulse oximetry but is rarely considered in trials measuring medical device performance [4,5,6,7]. Interrogating existing datasets and fully understanding the underlying biases requires a multidisciplinary examination involving more than a single data analyst or research group. Instead, the cooperation of clinicians, engineers, data scientists, social scientists, and industry partners is greatly needed. Indeed, studies have shown that research groups with more diverse expertise are more effective in identifying or addressing issues of bias [8].

Study hypothesis and contribution

In this study, we seek to understand the role of open data and diversity in research expertise towards mitigating biases affecting clinical models. We hypothesize that the groups of researchers who leverage existing open-access datasets are more diverse than those using privately-owned datasets. In what follows, we explain our rationale and motivation for analyzing the profile of researchers who use open vs. private datasets.

Many existing approaches to mitigate bias occur downstream of model development

The timing of efforts to address bias may be critical. When, in the lifecycle of a clinical model, are interventions to mitigate bias most effective? Healthcare systems can deploy interventions either downstream or upstream of the model development phase to mitigate the repercussions of data biases. Researchers in the field of ethics in artificial intelligence (AI) for health have participated in this effort by exploring several downstream approaches. Notably, biases can be mitigated after model fitting and/or deployment [9] through the use of explainable AI (XAI) tools [10,11] that identify biased features contributing to discriminatory outcomes (e.g., by decomposing individual predicted risk scores).

The public release of datasets offers an alternative, upstream approach

In contrast to XAI tools and other technology-based solutions, the public release of datasets in science, medicine, and engineering offers an upstream solution. This human-centric approach, which focuses on better understanding biases in health data, maximizes the number of investigators involved and leverages their cognitive and social diversity to examine clinical data. There exist several examples throughout history whereby research necessitated multiple teams to examine the same dataset to ultimately reach an agreement. The case of right heart catheterization was notable: repeated analyses of the original dataset, collected by Connors et al. in 1996 [12], yielded conflicting results. This confusion left clinicians needing clarification about the effect of the procedure for years. Untangling the confounding factors to reach our current understanding took several teams of biostatisticians. In sum, the coordinated efforts of many investigators, in an iterative learning process, are required to achieve consensus in data analysis and interpretation. In recent years [13,14], additional upstream approaches such as the embedded ethics methodology have been proposed to ensure that interdisciplinary ethical inquiry and deliberation are integrated into AI and healthcare technology development processes starting at project ideation [15]. Others have advanced approaches such as algorithmic impact analysis, which seeks to develop robust public interest methodologies to better understand the impact of AI and automated decision-making systems on people’s lives and society at large [13].

The current landscape of clinical data science research

In the past five years, several organizations–including the National Institutes of Health (NIH), European Commission, and World Economic Forum [16]–have recommended a shift to Open Data, a movement whose goal is to increase the release of FAIR (Findable, Accessible, Interoperable, and Reusable) [17] datasets in scientific research. In particular, large investments have been made in the biomedical sciences [18,19]. However, the clinical data science landscape remains highly siloed and opaque [20]. Expertise at the intersection of computer science and healthcare is currently concentrated among a few academic and industry research teams responsible for the preprocessing of data and the training of models [21,22]. Only researchers who are fortunate enough to be aware of a dataset’s existence, to be granted access to it, and to have sufficient funding to afford the associated licensing fees and computing infrastructure, can effectively leverage it in practice. Therefore, in far too many instances, datasets are often inaccessible to investigators outside the very research team that curated them. For example, researchers who are not clinicians (e.g., computer scientists at MIT working on diabetic retinopathy in Uganda) often have limited access to primary data. Thus, they must rely on second-hand knowledge from clinical investigators at their institution or in their network (e.g., physicians at Harvard Medical School) but whose domain expertise may have been gained from datasets and practices originated in a different context. For example, datasets developed in North America may have significant limitations when used to train models to be implemented in East Africa and vice versa. Such secondary data analysis requires a deep understanding of the data curation process, including biases in data collection and artifacts in clinical measurements, which may vary locally by medical site or by region. Thus, knowledge transfer alone is insufficient to safeguard against the spread of biases. There are numerous reasons why bias occurs in clinical models. For instance, a decision-support model to prioritize screening for diabetic retinopathy that does not appropriately account for differences in the frequency of specialty visits among patients may result in selection bias. This example points to the need for familiarity with relevant socio-demographics and patient care-seeking behavior. Bias also can manifest when analysts inherit datasets without any background about the underlying environment. In such situations, they risk not only using a training dataset that is ill-suited for the target population, but also failing to understand the limitations of their model because local features have not been considered. Ideally, researchers will seek to interface with the team responsible for primary data collection. Without such a dialogue, external teams run the risk of unsafely deploying algorithms that do not generalize well out-of-distribution, for the cohorts of patients they care for.

The promise of new NIH data-sharing policies

The recent imperative by the NIH to share scientific data publicly underscores a significant shift in academic research [23]. Effective as of January 2023, it emphasizes that transparency in data collection and dedicated efforts towards data sharing–with other investigators and the broader public, including citizen scientists–are prerequisites for translational research, from the lab to the bedside. Certain fields of healthcare have paved the way: workshops on data ethics, privacy, consent, and anonymization have been organized in radiology [24]; a common ontology has been developed for data standardization in radiation oncology [25]; multi-center data-sharing platforms have been designed for collaboration in sleep medicine [26]; and distributed learning networks have recently been proposed as a solution to preserve the privacy of patients’ EHR [27]. In the long run, requirements such as submitting a Data Management and Sharing Plan along with funding applications [28] will allow a more diverse population of researchers to interrogate both raw, unprocessed, and curated, pre-processed datasets. In light of the significance of data access to the mitigation of bias, we hypothesize that researchers who leverage existing open-access datasets rather than privately-owned ones are more diverse. We reason that the diversity of the backgrounds of open dataset users may in turn result into greater attention to equity in patient data collection and in a more compelling use of research data to address the pressing needs of the global population. This increased attention to issues of equity could also facilitate a more nuanced interpretation of study results and a stronger willingness to translate these results into practice.

Structure of this paper

In this brief report, we propose to test this hypothesis in the transdisciplinary and expanding field of AI for Critical Care. Specifically, we compare the diversity among authors of publications leveraging open datasets, such as the commonly used MIMIC [29] and eICU databases [30], with that among authors of publications relying exclusively on private datasets, unavailable to other research investigators. To measure the extent of author diversity, we characterize gender balance, geographic diversity (i.e., the number of countries with which authors are affiliated and the income categories these countries map to), and the presence of researchers from minority serving institutions (MSI). Furthermore, we comment on the challenges of increasing the participation of researchers from underrepresented groups and suggest changes that can be made to the current Open Data strategy to enhance representation in authorship in the next decade. By evaluating the association between data accessibility and author diversity, our study pinpoints actionable steps that the broader field of clinical AI can take to foster inclusion in the scientific community and mitigate blind spots in data preparation and/or model development.

Data

We leveraged PubMed [31] to select research studies at the intersection of AI and Critical Care published between 2010 and 2022. We created two separate queries to derive (1) a list of publications related to AI and (2) a list of publications addressing topics in critical care medicine. We used the same process as Celi et al. [19] for the AI-specific query since the authors’ model identified AI-related publications with suitable performance for our task (AUROC = 0.96). Query search terms specific to critical care were selected based on Van de Sande et al. [32] and vetted by two physician authors on our team, LAC and JG. Subsequently, we merged the two publication lists, thereby capturing only the studies related to both fields. Further, we split the resulting set of studies into two groups: “treatment” and “control.” The treatment group comprised publications leveraging either of the two major critical care databases currently in open access, i.e., MIMIC and eICU. We used dataset-specific queries from Google Dataset Search [33] to derive a list of works leveraging MIMIC or eICU critical care databases. Conversely, the control group consisted of publications based on privately-owned datasets that are unavailable to researchers other than the primary investigators. To avoid leakage, we confirmed that the two groups were mutually exclusive, i.e., no publication belonged to both. For studies in the control group, we first downloaded their unique PMID identifiers from PubMed, owing to the large sample size. Then, we used the Dimensions AI platform, an interlinked research information system provided by Digital Science, to collect metadata pertaining to each research article [34]. For studies in the treatment group, we performed the query via Dimensions AI directly as the sample size was much smaller. The platform was accessed in June 2023. Finally, we manually filtered the initial set of papers in the treatment group to exclude outliers and include only relevant MIMIC and eICU manuscripts. Three team members (AP, CL, and GL) completed this manual validation task independently before reconvening and reaching a consensus. Details about the creation of the study dataset are available in Fig 1A. Of note, we did not curate the MIMIC and eICU databases ourselves but provide a detailed description of the relevant studies to guide the readers who want to learn more about data collection and preprocessing in S1 Text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

Flow diagram illustrating (A) the methodology for the analysis of authorship diversity in scientific publications at the intersection of AI and Critical care and (B) the number of publications considered.

https://doi.org/10.1371/journal.pdig.0000486.g001

Labeling of author gender

Each author’s first name was processed by the Genderize.io Application Programming Interface (API) (Fig 1A). The API is based on a global collection of first names that have been manually annotated and linked to their most likely gender. Building on this international database of first names, the probability that an author is a woman or a man can be derived from the API. The API returns an “unknown” label when the uncertainty is too high. We assigned the most probable gender label associated with each author’s first name (“female,” “male,” or “unknown”). Importantly, Genderize.io is considered a reliable software to infer gender based on names. We chose to rely on the Genderize package rather than on the Gender API or the gender R package because a recent evaluation study by VanHelene et al. [35], based on a sizable multi-national dataset of over 32,000 gender-labeled clinical trial authors, reported it was generally more accurate than its counterparts (overall accuracy of about 96.6%; Gender API: 96.1%; gender R: 85.7%), while also being less expensive to use.

Labeling of minority-serving institutions

To measure the extent of the representation of minority-serving institutions (MSI) within the control and treatment groups, we developed our own fuzzy-matching pipeline between a pre-specified list of institutions and each author’s affiliation(s). In particular, we built upon the fuzzy-match Python package [36], specifically the Levenshtein Partial Ratio Function with a matching threshold of 97 percent. The list of MSIs used in this study was obtained by combining two data sources: the 2020 list from [37] comprising 774 distinct MSIs and the 2022 list from [38] containing 865 distinct MSIs (Fig 1A). A total of 566 institutions were shared by the two sources (according to our exact and fuzzy matching process based on the institution’s name), while the remaining MSIs were unique to each list. The integration of these two data sources allowed for a more comprehensive set of MSIs.

To confirm the accuracy of the mapping between institutional affiliations and their potential MSI status, we performed manual verification of outputs from the fuzzy matching process for both the 2020 and 2022 MSI datasets. In cases when an author’s institutional affiliation was incorrectly mapped to an MSI, we rectified the mistake manually. Verification was limited to reducing false positives, i.e., we only determined institutional affiliations that were erroneously linked with MSIs. However, our matching process was deemed comprehensive, since we selected a high threshold value of 97 percent to limit the number of false negatives.

Labeling of institutions based in low- and middle-income countries

The 2022 World Bank country classification was used to map countries associated with researcher affiliations to the low- and middle-income category (LMIC) or the high-income category (HIC) (Fig 1A). Countries are ranked according to their gross national income. For authors with multiple affiliations, each was considered separately and mapped to the corresponding income category.

Diversity metrics

A total of three diversity metrics were considered. First, for each paper, we quantified the overall number of authors, the number of probable women among the authors, and whether the first/last author was likely a woman. For both the control and treatment groups, we derived the proportion of probable women by article and the overall percentage of articles featuring an author who was likely a woman as the first and/or last author. Second, for each paper, we measured the number of authors affiliated with an institution based in an LMIC and whether the first/last author was based in an LMIC. For both groups, we derived the proportion of LMIC authors by article and the overall percentage of articles featuring an LMIC author in a leading role. Third, for each article, we quantified the number of authors affiliated with an MSI and the MSI status of the first and/or last author’s affiliation. For both groups, we similarly derived the proportion of MSI authors by article and the overall percentage of articles featuring MSI authors. Note that papers whose first or last author had an unknown gender or unidentifiable LMIC or MSI status based on their affiliation were excluded from the corresponding analyses.

Statistical analysis

For each of the three diversity metrics of interest (i.e., gender representation, geographic diversity, and MSI status), we performed a one-sided proportional Chi-squared test of independence to determine if there was a significant difference between the control and treatment groups. We provide further details about the assumptions underlying the statistical tests used in our study in S2 Text. We set the threshold for statistical significance to 0.05, following common practice. A p-value less than 0.05 would thus indicate a statistically significant difference between the control and treatment groups (e.g., in terms of the representation of women, LMIC, or MSI authors), in favor of the latter. We applied a Bonferroni correction to handle multiple hypothesis testing. We conducted three sensitivity analyses to assess the impact of missing data and to test the robustness of our results–in terms of both gender representation and geographic diversity. In the first (respectively, second) counterfactual scenario for gender representation, we assumed that all authors whose gender could not be inferred were women (respectively, men). In the third counterfactual scenario, we used mean imputation to derive gender labels, i.e., using the distribution of either the control or treatment group, whichever the article belonged to. Similarly, in the first counterfactual scenario for geographic diversity, we assumed that none of the papers with missing author affiliation had any authors from LMICs. Conversely, in the second counterfactual scenario, we assumed that all of these papers had at least one author affiliated with an institution based in an LMIC. Lastly, in the third counterfactual scenario, we assumed that missing income category labels could be imputed via dataset-specific distributions derived from labeled data, i.e., using either that of the control or treatment group, depending on the group the article belonged to (Fig 1B).

Gender

The proportion of papers with at least one author inferred to be a woman was qualitatively comparable between the two groups, albeit slightly higher in the control group (73.0% vs. 71.8%, z = -0.726, p = 0.468). The representation of women among the first and the last authors was similar between the two groups (28.1% vs. 30.9%, z = -1.74, p = 0.0811; 23.6% vs. 21.7%, z = 1.28, p = 0.199, respectively). Importantly, in both groups, the proportion of women serving as a last author (overall average of 22.5%), often reflecting a senior research leadership role, was lower than that of women serving as a first author (overall average of 29.7%), generally awarded to the person leading study design and analysis. This difference was more pronounced (9.2 vs. 4.5 percentage points) in the control than in the treatment group. The results of our three sensitivity analyses–assuming in turn that authors whose gender could not be inferred by the Genderize package were women (S1 Table) or men (S2 Table) or imputing missing gender labels by sampling from labeled examples (S3 Table)–were similar to those obtained in the main analysis, restricted to authors whose gender could be inferred. Thus, we believe that our conclusions with respect to the gender dimension are robust to potential misclassification error.

Low- and middle-income countries (LMIC)

Overall, out of the 4,181 articles with non-missing affiliations included in our study, 318 (i.e., 7.6%) had at least one LMIC author. The proportion of authors from LMICs was substantially higher in the treatment than in the control group (10.1% vs. 6.2%, z = 4.5, p<0.001). Moreover, we found that the diversity of first and last authors in terms of country of affiliation was greater in the treatment group, i.e., among studies leveraging MIMIC and eICU open critical care datasets. Indeed, 7.9% (vs. 4.8%, z = 4.30, p<0.001) of papers in the treatment group had their first author affiliated with an LMIC country. Furthermore, 7.8% (vs. 4.5% z = 3.14, p<0.001) had their last author affiliated with an LMIC country. The first sensitivity analysis (S4 Table), imputing missing data using the distribution based on labeled samples, also led to the conclusion of a greater representation of LMIC authors among researchers leveraging open datasets (overall: 10.1% vs. 8.3%, z = 4.5, p<0.001). The second sensitivity analysis (S5 Table), which made the optimistic assumption that all papers with missing affiliation information had authors from LMICs, confirmed the robustness of our results (overall: 42.0% vs. 16.7%, z = 23.5, p<0.001). The third sensitivity analysis (S6 Table), which made the pessimistic assumption that none of the papers with missing country information had authors from LMICs, yielded results qualitatively similar to the main analysis, albeit not statistically significant (overall: 6.5% vs. 7.3%, z = 1.1, p = 0.14; first: 4.6% vs. 4.9%, z = 1.3, p = 0.10; last: 4.6% vs. 5.2%, z = 0.77, p = 0.22).

Minority serving institutions (MSI)

Our analysis of authorship among MSIs was restricted to the United States (US). The control group comprised 4,914 distinct authors with institutional affiliations within the US for a total of 16,743 distinct authors with an affiliation worldwide (i.e., 29.3%). Among them, 277 different authors were affiliated with MSIs, accounting for approximately 5.6% of the total. In contrast, the treatment group comprised 2,207 authors with institutional affiliations within the US, for a total of 8,210 distinct authors with non-missing affiliations worldwide (i.e., 26.9%). Among them, 190 different authors were affiliated with MSIs, accounting for approximately 8.6% of the total. Thus, the proportion of US-based authors affiliated with an MSI was 1.5 times higher among articles in the treatment than in the control group; this difference was statistically significant, suggesting that open data resources attract a larger pool of participants from minority groups (z = 4.69, p<0.001).

In addition to overall statistics, we characterized the involvement of MSI researchers at the team level, i.e., per paper. The control (resp. treatment) group consisted of 991 (resp. 456) distinct papers with at least one author having an institutional affiliation in the US, out of 2,877 (resp. 1,672) papers worldwide (i.e., 34.4% and 27.3%, respectively). Among these papers, 175 (resp. 126) had at least one author affiliated with an MSI, representing approximately 17.7% (resp. 27.6%) of the total US research output in critical care AI involving the use of private (resp. open) databases. Thus, the proportion of papers featuring US-based authors affiliated with an MSI was 1.6 times higher in the treatment than in the control group; this difference was statistically significant, suggesting that MSI researchers effectively benefit from open data resources, which further translates into publications and preprints (z = 4.34, p<0.001).

Out of the 991 distinct papers in the control group, the percentage of papers with an MSI-affiliated first author reached only 4.7% (47 / 991). The proportion of MSI researchers serving as first authors was significantly greater (z = 2.40, p = 0.008) in the treatment group, reaching 7.9% (36 / 456). This result suggests that barriers remain for MSI-affiliated authors to lead research studies when the underlying datasets are inaccessible to the broader public. In contrast, opening critical care datasets can bolster the participation of MSI researchers as first authors. Of note, the percentage of papers with an MSI-affiliated last author was larger (z = 0.45, p = 0.327) in the treatment group (28 / 456, i.e., 6.1%) than in the control group (55 / 991, i.e., 5.5%), but this difference was statistically insignificant owing in part to a reduced sample size in the analysis focused on MSI representation.

Intersectionality