Introduction

Accurate assessment of glomerular filtration rate (GFR) is critical for patient care, especially for those with chronic diseases. Chronic kidney disease (CKD) affects approximately 9% of the world’s population (697 million people) and GFR estimation from serum creatinine is recommended and used globally [1]. CKD is common in patients with other chronic diseases, increases the risk for adverse outcomes for such diseases, and impacts on the decisions regarding their management and treatment. For example, approximately 20–50% of individuals diagnosed with heart or liver disease also have CKD [2–5]. The presence of CKD increases the risk of mortality by almost two-fold in heart failure patients [6] and a seven-fold increase in cirrhosis patients [3]. Many of the medications used to treat these diseases are cleared by the kidney, or are toxic to it, requiring assessment of the level of GFR prior to dosing. Standard practice for assessing kidney function in adults [1, 7, 8] is to estimate GFR (eGFR) using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equations [9], which combine the endogenous filtration markers creatinine and/or cystatin-c as measured in blood with sex and age. Creatinine is measured as part of a basic metabolic panel and eGFR using creatinine (eGFRcr) is reported by clinical laboratories, allowing eGFR to be used for all routine decisions [10]. eGFR using both creatinine and cystatin-C (eGFRcr-cys) is generally more accurate than either GFRcr or eGFRcys and is used as a confirmatory test when eGFRcr is not thought to be accurate and the clinical decision is based on the level of GFR. [1]

In principle, when equations developed in one population are applied to a new population, observed differences in eGFR compared to measured GFR (mGFR) are predominantly due to the presence of non-GFR factors that affect the serum concentration of endogenous filtration markers (non-GFR determinants) [11–13]. Non-GFR determinants include generation, renal tubular reabsorption and secretion, and extra-renal elimination of the filtration markers. The CKD-EPI equations were developed in diverse populations with and without CKD and with respect to many key factors such as the level of GFR, diabetes, and age, but did not include populations with other chronic diseases. Thus, while these equations have been demonstrated to be reasonably accurate in general or CKD populations, they are less accurate in populations with other chronic diseases. For example, in patients with heart failure or cirrhosis, eGFR errors of over 30% can occur in 35–60% of cases [14–16], compared to roughly 15% in the development populations [9]. Non-GFR determinants of filtration markers may affect GFR estimation in two ways: First, increased variability in the distributions of non-GFR determinants in the application population relative to the development population may produce greater imprecision of the eGFR relative to mGFR in application populations. Second, systematic differences in the distributions of non-GFR determinants between application populations and the development populations may produce systematic differences in the relationship between mGFR and filtration markers, leading to biased estimation of GFR in the application populations.

We have hypothesized that the greater accuracy of eGFRcr-cys is due to non-overlapping non-GFR determinants of these two markers [17]. For example, eGFRcr is overestimated in patients with muscle wasting, and eGFRcys is underestimated in patients with higher levels of adiposity, smoking and chronic inflammation [18]. Thus, using both markers in eGFRcr-cys in theory balances the opposite effects of these non-GFR factors which are common in people with chronic disease. However, this balance cannot be expected to be present in all settings. For example, some medications may affect kidney tubular secretion or extra-renal elimination of creatinine, but these same medications are not known to affect the level of cystatin-C [19–21]. In these cases, eGFRcys may be more accurate than eGFRcr-cys. It is not possible for a physician to be able to appreciate the presence and magnitude of the non-GFR determinants of each filtration marker in an individual patient, leading to uncertainty as to which eGFR to use. To overcome this central challenge of GFR estimation, we hypothesized that a larger number of endogenous filtration markers (here on referred to as panel eGFR) will be required [11, 13, 22]. Ideally, by incorporating additional markers with minimally overlapping non-GFR determinants, a panel eGFR could diminish the importance of any single marker. However, methods for robust prediction within application populations will be required.

We note that previous statistical literature on robust prediction has focused on reducing the effect of outliers within the development data set. [23] These methods, such as least trimmed squares [24], MM regression [25, 26], robust model selection [27], and robust partial least squares [28] typically estimate robust model coefficients by down-weighting large residuals within the development data such that outliers do not heavily influence the model coefficients. Multivariate methods, such as robust principal components [17, 21], can also be used to identify outliers within the development data. However, to our knowledge, the question of how predictive models constructed from a development population can be modified to reduce the impact of an increased rate of outliers in an application population has yet to be addressed. Separately, the literature also has many examples [29–32] of redeveloping equations in new applications, especially geographic regions, to overcome systematic differences in distributions of non-GFR determinants between the application population and the population contributing to the development data sets. However, the cost of obtaining an adequate number of participants with mGFR to estimate separate equations in each target population with sufficient accuracy and robustness for clinical use is likely to be prohibitive. Here, the application of transfer learning may be useful to tailor estimates for specific patient applications.

In this analysis, we evaluated a panel of eight markers (5 metabolites including creatinine and 3 low molecular weight proteins including cystatin-C) in 3,554 participants from 9 studies representing diverse patient populations. In separate investigations, we are in the process of identifying additional metabolites to be included in a metabolite panel eGFR [22]. This new panel is intended as an independent complementary test for clinical settings in which eGFRcr-cys is not sufficiently accurate for the medical decision-making. Critical to this use are methods that can robustly estimate GFR in new applications, namely, methods that de-emphasize outlying markers within new individuals and tailor estimates to specific groups. The goal of this paper is to propose and evaluate methods for estimating GFR based on multiple filtration markers in application populations with potentially higher rates of outlying predictors than in the development data. We further apply transfer learning techniques to account for systematic differences between the application population and development populations. We demonstrate these methods using our existing data as a proof-of-concept, but we intend this work to be applied to the panel under development [22]. We expect that certain applications may have higher rates of outlying predictor values of the markers compared to our current datasets; therefore, we simulated these applications by altering the relative frequencies of outliers in application datasets.

Methods

Statistical methods

Our overall goal was to develop methods to estimate GFR from multiple markers while considering alterations in non-GFR determinants of the markers in new applications (Fig 1). First, we combined three approaches for detecting outlying predictors in application data with robust estimation methods, enabling us to evaluate nine distinct approaches for mGFR estimation (Table 1). Next, because our current development datasets do not include many patients with acute illnesses, these datasets would likely exhibit fewer outlying predictors than would be seen in real-world experiences. Therefore, we evaluated these approaches using statistical simulation to emulate application scenarios with increased outlying predictors. Finally, we evaluated the possible merits of transfer learning in estimating GFR. We then integrated both methods (outlier detection with robust prediction and transfer learning) to customize equations for distinct patient groups.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic overview of our approach to estimate GFR robustly in new applications.

Application of GFR estimating equations that was developed in one population (blue figures on the top left) will be less accurate in different populations (green figures on the top right) due to variation in the distribution of factors that determine the level of the filtration markers other than GFR (referred to as non GFR determinants). Non-GFR determinants of markers can affect GFR estimation accuracy by distorting markers in individual patients (plots on the left hand side) and causing systematic differences between development and application populations (plots on the right hand side). We propose methods to address both errors, using techniques to detect outlying predictor variables, robust estimation, and transfer learning. Finally, we combine these approaches for robust GFR estimates in new applications.

https://doi.org/10.1371/journal.pone.0313154.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of each combination of outlier detection methods and robust estimation approaches.

We combined each outlier detection method with each estimation approach such that there were nine different appoaches for robust GFR estimation in new application data.

https://doi.org/10.1371/journal.pone.0313154.t001

In descriptive analysis, we present the average within-study correlations of the markers with mGFR across studies. We fit single-marker and multi-marker (including all eight markers) regression models for mGFR separately for each study, and assessed predictive accuracy using root mean square error (RMSE), with lower values indicating better accuracy. To assess how predictive accuracy changes by including more markers in this data, we fit ordinary least squares linear regression models for all possible collections of the eight markers (i.e., all models containing a single marker, all models containing two markers, etc.). Application data error rates were approximated through ten iterations of ten-fold cross-validation.

Results

The study populations (N = 3,554) include participants with and without cancer, cardiovascular disease, and CKD across a wide range of ages, geographic locations, and race groups (Table 2). Mean ages ranged from 35.4 (standard deviation (SD) 7.6) years in the Consortium for Radiologic Imaging Studies of Polycystic Kidney Disease (CRISP) to 80.2 (3.9) years in Age, Gene/Environment Susceptibility)-Reykjavik Study (AGES). The proportion of black individuals varied across the cohorts, spanning from 0% in AGES and Pakistan to 100% in African American Study of Kidney Disease and Hypertension (AASK). Average mGFR across cohorts varied, with mean values of 33.0 (11.6) mL/min/1.73m² in the Modification of Diet in Renal Disease (MDRD) study to 96.1 (14.6) in Assessing Long Term Outcomes in Living Kidney Donors (ALTOLD).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of analytic dataset (N = 3,554).

https://doi.org/10.1371/journal.pone.0313154.t002

The individual filtration markers varied both in their association with mGFR and their consistency across studies (Fig 2). Five of the markers had stronger average Pearson correlations with mGFR across study than that of creatinine (r = -0.58): pseudouridine (r = -0.74), cystatin-C (r = -0.73), B2M (r = -0.72), acetylthreonine (r = -0.71), and BTP (r = -0.61), whereas phenylacetylglutamine and tryptophan had weaker correlations with mGFR than creatinine at -0.41 and 0.30, respectively. Estimated regression lines were generally similar by study for pseudouridine and cystatin-C, but considerably different for phenylacetylglutamine and tryptophan. S2 and S3 Tables present both single-marker and multi-marker linear regression models for mGFR by study. Notably, there is observable variability in the multivariable coefficients across different studies. For example, the multivariable coefficient for cystatin-C ranged from 0.00 in Pakistan to -0.37 in MDRD. Similarly, the multivariable coefficient for pseudouridine was -0.05 in Onco-GFR and -0.71 in Pakistan. R² ranged from 0.86 in AGES to 0.37 in ALTOLD.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Regression models for mGFR using each marker by study.

The average, minimum, and maximum correlations [Average r (minimum, maximum)] of each marker with mGFR across studies is provided within each plot.

https://doi.org/10.1371/journal.pone.0313154.g002

S3 Fig shows cross-validated RMSEs for all possible regression models of the eight markers. The 28 models containing six markers are able to predict mGFR almost as accurately as the single model with eight markers: cross-validated RMSEs ranged from 0.186 to 0.196 in models containing six markers, with 89.8–91.1% of individuals within 30% of their mGFR in cross-validation, compared to a cross-validated RMSE of 0.185 and 90.8% of individuals within 30% of their mGFR when using all eight markers. Therefore, if up to two markers are outlying, we can still make almost as accurate GFR estimates from the remaining six markers.

With no added contamination and application of the outlier detection and robust estimation approaches (S4 Fig), RMSEs were similar. The RMSE was smallest using the multivariate outlier approach with KNN marginalized estimation (0.182) and the univariate outlier approach with linear regression with screen predictors estimation (0.189). In general, RMSEs were largest using weighted trimmed mean estimation (RMSEs 0.208–0.221).

Fig 3 displays RMSE for each outlier detection and robust estimation strategy after contaminating a single marker using mean and variance contamination. Given that the markers differ in their association with mGFR, we expected the impact of contamination on prediction to vary across the individual markers. Overall, contaminating markers that have stronger associations with mGFR, such as pseudouridine and cystatin-C, had a much larger impact on estimation error and bias than contaminating markers with weaker associations with mGFR, such as tryptophan. For example, when adding contamination to cystatin-C values in the test data but paying no special attention to outlying predictors, the cross-validated RMSE was 0.258. After identifying univariate outliers and applying the three robust estimation approaches, RMSEs improved to 0.215–0.232, but were still notably higher than linear regression with 8 markers and no added contamination (0.185). Similarly, using the multivariate outliers strategy, RMSEs ranged from 0.226–0.248. However, when removing inconsistent outliers, the RMSEs were very similar to the case when no contamination is present. For example, using linear regression with screened predictors, the RMSE was 0.200. The results using other contamination models (mean or variance contamination) are similar (S5 Fig). In terms of bias, models were generally least biased when considering inconsistent outliers in the test data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Comparison of RMSE using all modeling and prediction approaches after mean and variance contamination of each marker individually.

The color of the points represents the underlying outlier detection strategy, and the shape represents the robust estimation approach. Results are averaged across ten cross-validation iterations. RMSE: Root Mean Square Error.

https://doi.org/10.1371/journal.pone.0313154.g003

In general, in the presence of substantial contamination, RMSEs were lowest when estimating GFR based on the six most consistent metabolites, whereas RMSEs were substantially influenced by contamination under the univariate and multivariate outlier methods. These findings remained consistent across each of the contamination models and markers. Similarly, models were least biased when focusing on the most consistent predictor markers. When comparing the robust prediction methods within each outlier strategy, RMSEs was generally best using linear regression and KNN marginalized estimation, though linear regression with screened predictors was a close second, though linear regression with screened consistent markers was the least biased approach.

The RMSEs associated with the contamination of two markers using mean and variance contamination are given in Fig 4. Unsurprisingly, contaminating two predictors with high correlation with mGFR (pseudouridine and cystatin-C) of mGFR had a larger impact than contaminating one or no good predictors (RMSE 0.421 using traditional linear regression with no outlier detection). Across all outlier detection methods, again focusing estimation on the consistent markers yielded the smallest RMSE and bias. In terms of estimation, linear regression and KNN marginalized estimation and linear regression with screened predictors were generally similar, though the average bias was slightly smaller using linear regression with screened predictors. For example, after contaminating pseudouridine and cystatin-c, RMSEs ranged between 0.202 and 0.212 when considering the consistent predictors. The RMSE for KNN marginalized estimation and linear regression with screened predictors were similar (e.g., RMSE 0.202 vs 0.205), though the average bias was slightly smaller using linear regression with screened predictors estimation (0.016 vs. 0.011). Similar results are found using mean or variance contamination models (S6 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of RMSE using all outlier detection and robust estimation after mean and variance contamination of two markers.

The color of the points represents the underlying outlier detection strategy, and the shape represents the robust estimation approach. Results are averaged across ten cross-validation iterations. We show four of the 28 possible pairs of contaminated markers. The selected pairs represent the results after contaminated two excellent predictors (cystatin-c and pseudouridine, average correlation r with mGFR across studies -0.73 and -0.74, respectively), one excellent predictor and one good predictor (cystatin-c and creatinine, average r for creatinine and mGFR = -0.58), one excellent predictor and one poor predictor (cystatin-c and tryptophan, average r for tryptophan and mGFR = -0.30), and two poor predictors (tryptophan and phenylacetylglutamine, average r for phenylacetylglutamine and mGFR = -0.41). Results are averaged across ten cross-validation iterations. RMSE: Root Mean Square Error.

https://doi.org/10.1371/journal.pone.0313154.g004

In Fig 5, the potential advantages of transfer learning are illustrated. The graph illustrates RMSEs on the y-axis against the training sample size depicted on the x-axis, with all models incorporating 8 predictors. In all panels, the red lines represent the RMSE from a linear regression trained on all studies except the given target application data and then applied to the given target application data (external model). The blue line shows the study-specific linear models trained within the target application data across various randomly selected sample sizes and subsequently tested on the remaining observations. Finally, the green line depicts transfer learning models, developed across a grid of increasing sample sizes and tested on the remaining observations. In some studies, there is an observed improvement in accuracy using transfer learning with relatively small target datasets compared to fitting a new model in the target population. For instance, in both the MESA and Pakistan studies, the transfer learning model exhibited considerably higher accuracy for sample sizes less than 50 in MESA and less than 100 in Pakistan, after which the RMSEs became comparable to linear models. In other studies, such as in Onco-GFR and AASK, a larger target dataset was required to improve estimation beyond the external model. Analogous plots for bias are provided in S7 Fig. Both transfer learning and study-specific models exhibited considerably less bias compared to external models models fit to the pooled data set without accounting for study variation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Comparison of RMSEs from naïve (red), study specific linear (blue) and transfer (green) learning models for various training sizes.

RMSES are shown on the y-axis and the training sample size is shown in the x-axis. All models include all 8 predictors. RMSEs from linear models that were fit using all studies except for a single held out study used as the test dataset are shown in the horizontal red line (external Model). RMSEs from linear models fit within single studies are shown in blue. In this case, models were developed using a random sample of observations from the given study, and tested on the remaining observations in the study. RMSEs from transfer learning models, shown in green, were developed using a random sample of training observations from the target data and tested on the remaining observations. Given its relatively small total sample size (n = 55), we did not include Crisp in this analysis. Results are averaged across ten cross-validation iterations. RMSE: Root Mean Square Error.

https://doi.org/10.1371/journal.pone.0313154.g005

Lastly, we integrated outlier identification and robust estimation with transfer learning. In general, when using only 25 participants from the target study, combining outlier identification and robust estimation with transfer learning resulted in improved accuracy even further beyond the scenario when no contamination was added (Table 3). For example, in MESA the average RMSE from a linear model, developed using a random sample of 25 study participants and applied to the remaining observations in MESA, was 0.180 in the absence of added contamination. However, introducing contamination to one or two predictors in the application data increased the RMSEs to 0.364 and 0.433, respectively. Applying outlier detection and robust estimation using the most consistent markers and linear regression with screened predictors resulted in RMSEs slightly better than when no contamination was added (RMSE under a single contaminated marker and under two contaminated markers 0.173 and 0.173, respectively). The combination of outlier identification and robust prediction with transfer learning yielded further improved RMSEs 0.160 and 0.159, respectively. Bias was generally small after outlier identification (S4 Table); the benefit of transfer learning generally was limited to smaller sample sizes. Similarly, when using 50 and 100 observations from the target population, the further application of transfer learning resulted in minimal changes beyond outlier identification in terms of RMSE and bias (S5–S8 Tables).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of RMSEs after combining outlier detection and robust prediction with transfer learning for each study.

https://doi.org/10.1371/journal.pone.0313154.t003

Discussion

Patients with chronic disease or in diverse settings may have inaccurate eGFR because of elevated rates of outlying predictors or systematic differences in non-GFR determinants of one or more filtration markers from the development populations. This paper presents multiple methodologies for outlying predictor identification and robust estimation aimed at improving the accuracy of GFR estimates in patients with chronic diseases. Our findings demonstrate that even in cases of contamination, where a subset of markers are outlying for a particular patient, accurate and unbiased estimates remain feasible. Our results showed that if up to two markers are outlying for a given individual, we can still achieve accurate GFR estimates using the remaining six markers. In the presence of substantial contamination, applying linear regression using the most consistent markers resulted in the most robust estimation in terms of best accuracy and least bias. Further, transfer learning can be a useful tool to tailor equations to specific groups of patients, especially if data is limited in the target population. We found that if only a small dataset is available from a unique population with high rates of outliers, combining outlier identification and robust prediction with transfer learning can address both sources of error. However, once the target data available grows large, the benefit of transfer learning over a model developed directly in the target population becomes minimal.

Clinical practice guidelines recommend the initial evaluation of GFR using creatinine, and subsequently, in cases where eGFRcr proves inaccurate or where highly accurate GFR estimates are required, eGFRcys and eGFRcr-cys are recommended as confirmatory tests [7]. However, when creatinine and cystatin-C based estimates disagree for a given patient, it can be difficult to adjudicate which is more accurate with only two markers. In general, eGFRcr-cys has been shown to be the most accurate in studies where this has been evaluated, but this is not expected to be the case for all individuals [9, 39–43], as individuals with chronic diseases are not likely to be included in these studies. More generally, there are two significant drawbacks to eGFRcr-cys as a confirmatory test [17]. First, eGFRcr-cys is not independent of eGFRcr, thus it does not meet the requirement for a true confirmatory test. Second, with only two markers, any discrepancies between them cannot be easily adjudicated as to which is correct because the non-GFR determinants of cystatin-C are not well understood. For example, in patients with cirrhosis, liver transplant, heart failure, neuromuscular disease, and critical illness, there is variation in the relative performance of eGFRcr vs. eGFRcys, and here eGFRcr-cys was not consistently more accurate [16].

We found the outlier strategey mattered more than the estimation approach. Across all outlier strategies, robust estimation improved on naïve estimates. However, relying solely on the univariate distribution for outlier detection proved ineffective, particularly when employing a mean contamination model (S8 Fig). To identify most of the problematic points, most of the distribution of the contaminated markers would need to shift well beyond the 99^th percentile (our definition of an upper-tailed outlier), which is especially apparent under a mean contamination model. Consequently, this method is only effective when marker values deviate to an extreme degree, as smaller shifts would remain undetected. Winsorization, despite its merits, poses drawbacks in its treatment of patients with identified problematic values, characterized by a large multivariate Mahalanobis distance from the development data. In this approach, all eight markers undergo uniform shrinkage toward the development data, impacting not only the contaminated markers but also the uncontaminated ones. Consequently, this method may only be useful when all markers exhibit problematic behavior, a scenario in which accurate estimation is unlikely.

Additionally, there are situations in which the transfer learner fit with small sample sizes is slightly worse than the naïve model (for example for smaller sample sizes in Onco-GFR and ALTOLD). In these settings, a small sample may lead to overfitting by not capturing the full variability in the target applications. If a transfer learner is used in these situations, the development data would need to be chosen very carefully to fully represent the target population and minimize negative transfer. Nevertheless, we showed that it is generally possible to accurately estimate GFR given outlying predictors. In practice, the optimal application of GFR estimating equations could apply these methods if multiple markers are measured.

The robust prediction approaches presented here can be implemented using pre-constructed component models so that there is minimal additional computation time compared to a simpler approach such as a single linear model. There are 28 different models comprising six of the eight filtration markers that we evaluated; these models can be pre-fitted in the development data at the outset. In the era of advanced EHR and integrated laboratory information systems within these EHRs, the multiple algorithms required could be embedded into laboratory reporting procedures.

To our knowledge, this work is the first to address the handling of outliers or anomalous predictors in application data. The strengths of our study include our characterization of the two ways in which GFR estimates are altered by the non-GFR determinants of the predictor variables. While the present paper focuses on how to robustly predict GFR in new applications, the methods developed here may be applicable to other clinical algorithms susceptible to outliers.

There are also several areas for improvement in future work. We did not consider the presence of outlying predictors and outliers within the development data. To do so was beyond the scope of this paper, as our focus was solely on anomalies in application data. Future work should investigate the presence and influence of outliers within the development datasets, as there is substantial statistical literature in this area with some reassurance that positive and negative outliers have effects that can cancel each other on model estimates. Further, our method for identifying inconsistent markers could be improved. It is possible that multiple outlying markers could interact with one another; this situation may be more difficult to identify using the methods presented here. Additionally, we were limited to only 8 filtration markers in our analysis. With a larger set of markers, we could employ more sophisticated anomaly detection algorithms like Isolation Forests [44] and Local Outlier Factors (LOF) [45]. These strategies would allow for the possibility that all markers are sufficiently consistent, so we could retain more markers for estimation while still effectively identifying outliers. Additionally, future work could incorporate additional covariates, such as body size, to help identify outlying predictor variables. Lastly, we could explore more advanced models that can be integrated with transfer learning, such as neural networks.

While this analysis focused on the set of markers available to us at this time, our ultimate goal is to apply these methods to a metabolite panel eGFR that is currently under development. For this panel, we have identified 17 candidate markers using untargeted metabolomics data from seven diverse studies. We are presently in the process of developing and validating the targeted mass spectrometry (MS) assay for the panel of markers to assess their analytical precision and to evaluate the performance of a panel eGFR in comparison to mGFR across diverse clinical settings, and the methods developed here may be a useful tool to increase the robustness of panel eGFR in these settings.

Overall, the methods presented provide a specific application for GFR estimation with broader implications. When an underlying process, such as organ function, requires precise estimation beyond what is possible with any single marker due to contamination by biologic processes other than the one being measured, a solution is still possible, using multiple markers.

Supporting information

Acknowledgments

We acknowledge collaborators from studies included in our analysis. A full list of study collaborators is outlined in the S1 File.