Introduction

Multilevel models provide a generalisation of linear models to settings in which the model parameters (e.g. regression coefficients) are in some way stratified by groups within the population [1]. For example, individuals in the population may belong to a much smaller set of groups or clusters, and data may be available on the level of the individual and the level of the group. Such hierarchical data structures occur naturally in a wide array of scientific applications, examples of which include phylogenetics, education, healthcare and medicine [2, 3]. This concept can be arbitrarily extended to any number of groupings that exist within the population, either hierarchically or without nesting. A simple linear model that does not include the multilevel structure is generally regarded as an inferior model choice in such situations as it neglects information inherently within the group structure. Instead, multilevel models explicitly model at each level of granularity. A wide variety of structures are possible, which raises an important question: how may we identify an optimal model structure from a number of competing hypothesised models? For example, should we include hierarchical structure, and should we prefer a multilevel model with varying intercepts or both varying slopes and intercepts? The answer to this question is generally context-specific, relating to the overarching goals of an analysis, e.g. inference or prediction, and to any prior knowledge the researcher has about the problem. In conjunction with this, there exists an array of criteria that can be used to compare the suitability of two separate models. For example, in the frequentist setting, the mostly widely used is the Akaike information criterion (AIC) [4], though other approaches include false-discovery rate [5] and likelihood ratio tests [6, 7].

In this work, we instead focus on Bayesian approaches to model selection, where the usual strategy is to calculate the Bayes factor of two competing models. This is defined as the ratio of the model evidence for each model, where the model evidence is the likelihood integrated over all model parameters with respect to the prior. A key advantage of using the model evidence for model comparison is that it implicitly discourages overfitting by penalising model complexity, since including additional parameters will increase the dimension of the parameter space to be integrated over. By way of contrast, the penalty on model complexity has to be artificially introduced in the AIC framework.

Direct calculation of the model evidence and Bayes factor is well-established for linear models under a normal-inverse-gamma prior (e.g. [8]), but cannot be obtained analytically for multilevel models, as the integral is intractable. As a result, the Bayes factor must be estimated, either by directly approximating the integral as a sum, for instance using importance sampling [9] or sequential Monte Carlo [10], or by jointly estimating posterior probabilities of proposed models through approximate Bayesian computation (ABC) methods, or by numerical methods [11, 12]. In ABC methods, a hierarchical Markov Chain Monte Carlo (MCMC) sampling scheme alternates between two sampling steps, first across the indices denoting each model and then for model parameters of the current chosen model. This requires specification of prior probabilities for the individual models, in addition to priors for the parameters of each model. The relative acceptance frequencies in the chain for the model index then provides an approximation to the posterior probabilities for the models. This, alongside the given priors, allow an estimation of the Bayes factor that bypasses the need to estimate model evidence for each model. A key challenge in such an approach though, is to ensure sufficient mixing in the MCMC chain for the model index, since if the MCMC spends too long exploring only one model, the resulting extreme autocorrelation biases the posterior probability estimates. There are several approaches to this ABC framework, including reversible-jump MCMC [13] and product-space MCMC [14]. In contrast to this imposed hierarchical structure of models, sequential Monte Carlo (SMC) can be run separately on each model, as a by-product of the algorithm is a direct estimate of the log model evidence. This is achieved by a combination of Metropolis-Hastings and importance sampling, in which the likelihood is optimised using a simulated annealing process. Whilst these MCMC approaches are widely used, the estimates tend to suffer dramatically in high-dimensional settings, due to challenges in adequately sampling associated complex high-dimensional parameter spaces.

This motivates the approach to the estimation of Bayes factors that we take here, using partially-integrated likelihoods instead of full likelihoods. We treat (potentially high-dimensional) non-variance parameters, such as the regression coefficients in the model, as nuisance parameters, and we analytically integrate these out with respect to conjugate Gaussian priors, since this reduces the dimension of the problem. This reduces the full likelihood on all parameters to an integrated likelihood on only variance parameters. We can then estimate the model evidence by returning to sequential Monte Carlo (or any of the aforementioned estimation methods), which yields improved results, reduces the bias and variance in estimates, and typically improves computational efficiency.

We illustrate our technique using both simulated data and Minnesota radon contamination dataset introduced by [1]. For the former, we simulate four datasets with multilevel structure that correspond to four models described in Methods and then estimate the model evidence for each model and each dataset. In the latter, we estimate the model evidence for various multilevel models proposed by the authors in [1]. This dataset contains measurements of the radon level in houses in the US state Minnesota, as well as predictors at the individual house level and at the county level. The grouping of houses within counties provides an inherent hierarchical structure. As radon is a carcinogen, identifying areas with higher concentrations of radon may be an important consideration in decision-making for homeowners and county authorities. The Minnesota radon contamination dataset has been used by several software packages to illustrate multilevel modelling approaches, such as in the Python module PyMC [15, 16].

Methods

As multilevel linear models are a generalisation of linear models, we can also view a simple linear model as the single-level case within the multilevel framework. For both clarity and computational reasons, we consider linear models and multilevel linear models separately, first summarising notation and then providing the integrated likelihoods in each case, given suitable priors. Multilevel linear models are often interchangeably described as mixed models, where fixed and random effects are equivalent to the population-wide and group-specific variables. We use the vocabulary of multilevel linear models, to mirror the work of [1] as this allows for higher-level generaliation. We provide open-access code for our work at [17]. In this code, we use PyMC for SMC sampling, given the full likelihoods and the integrated likelihoods that we have derived.

Example: Simulation study

We illustrate this approach first on simulated datasets based on the Prophet model of [25], which seeks to model a variable y as a non-linear function of time t. By specifying a suitable flexible multi-dimensional transform of t, which includes piece-wise linear and Fourier transform terms, we convert this problem to a linear model (or by extension a multilevel model). The piece-wise linear component requires pre-specified points s_n, n = 1, …, d₁, at which the function is continuous but not smooth, i.e. the gradient changes. The Fourier transform component requires a specified periodicity P and is truncated at 2d₂ terms. The dimension of x is then d = 1 + d₁ + 2d₂. The basic model structure is then as follows, where E[⋅] denotes expectation: Though this describes a non-linear relationship between y and t, the model itself is linear because it is linear in the coefficient β. For each model type we have described in previous sections (linear, simple multilevel, general multilevel and linear model with fully conjugate normal-inverse-gamma prior), we generate a simulated dataset corresponding to that model, i.e. as if this is the ‘true’ model. We then evaluate each model on all four datasets, estimating the model evidence via the integrated likelihood and the full likehood.

To generate the data, we first simulate multilevel group structure and t_ij, which in turn generates covariates x_ij that have the form above. The covariates z_ij, which are those that vary by group within a general multilevel model, are defined as a (centred) subset of the x_ij. Both group structure and covariates, including z_ij, are shared across all datasets. The underlying structure is not explicitly used in the linear model or the associated linear dataset, as there is no relationship between the group membership and outcome variables. For each dataset, we sample ‘true’ model coefficients, which are then regarded as fixed, and we then compute the outcome variable, y_ij, as defined by the form of the corresponding model. Together with the multilevel structure and the covariates, this outcome variable forms the dataset, . We describe the datasets in more detail below.

In all datasets, we set J = 15 and n = 1000, which are the number of groups and of observations respectively. To assign multilevel group membership within the data, we sample the integers j = 1, …, J with replacement with probability p_j. In order to generate unequal group sizes, we sample p_j from a Dirichlet distribution with parameter α = (2, …, J + 1), such that ∑_jp_j = 1 and that E[p_j] = (j + 1)/J². We also have:

• For all datasets: The constants added to z_ij are such that E[z_ij] = (1, 0, 0). We also specify a covariance hyperparameter S for simulating ‘true’ coefficients b from a multivariate Gaussian distribution, where S is a d × d positive-definite: The elements of S₁ were chosen to allow flexibility in the gradients of t in each interval [s_n, s_n+1], with larger values as δ_n represents only a change in the gradient from the interval [s_n−1, s_n] to the interval [s_n, s_n+1]. Similarly, λ was set to a small value so that the Fourier element did not dominate the piece-wise linear component.
• Linear dataset:
• Simple multilevel dataset:
• General multilevel dataset:
• Linear dataset using normal-inverse-gamma distribution: The joint distribution for b and s² is , which is a conjugate prior for the Gaussian linear model, with E[β] = 0, Cov[β] = S.

As the expected value of is b/(a − 1), and η_j and ϵ_ij are independent, in each case the outcome variable y_ij should have similar expected value and variance, if the entire data generation was repeated multiple times, i.e. E_θ[E[y_ij|θ]] = 0 and E_θ[var(y_ij − b^Tx_ij|θ)] = 0.2, where θ here indicates both covariate values and coefficients, e.g. θ = (t_ij, b, s²). This is important as it means the choice of model priors should contribute less to the model evidence than the model structure, when we evaluate each model against each dataset. All four simulated datasets are shown in Fig 1 and available in the accompanying repository.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Simulated datasets, .

Each dot represents a datapoint, with the model covariates x = g(t) a deterministic and multi-valued non-linear function of t. In addition, the line b^Tx is shown for all values of t ∈ [0, 1] for each dataset. For , the lines b^Tx + h_j are also included, for j = 1, …, J = 15. Similarly, for , the lines are also included, where z is also a deterministic multi-valued function of t.

https://doi.org/10.1371/journal.pone.0280046.g001

We now regard the ‘true’ coefficients, e.g. b and s, as fixed but unknown, and we specify models corresponding to each dataset. We specify priors for each model, which are similar to the distributions from which we generated the ‘true’ coefficients. In place of the covariance matrix S used to generate data, the prior for β has covariance Σ, which shares the diagonal terms of S but is zero elsewhere. The models are as follows:

• For all models:
• Linear model:
• Simple multilevel model:
• General multilevel model:
• Linear model using normal-inverse-gamma distribution prior:

For each dataset, we expect the model with ‘true’ structure to have the largest model evidence. Table 1 shows the performance of the models on the respective datasets, illustrating how our approach not only leads to a decrease in the uncertainty of the estimated model evidence, but can also prevent model misspecification where sampling using the full likelihood is unable to do so, correctly identifying the ‘true’ model for each dataset. For models to , we cannot evaluate the bias of the log model evidence estimates, with no direct solution to compare to. However, for the linear model with normal-inverse-gamma prior, direct computation shows that estimates using the integrated likelihood are unbiased and closer to the analytic solution than the estimates using the full likelihood. Furthermore, in every instance within this example the computational cost of running the SMC sampling algorithm on the variance parameters with the integrated likelihood was reduced compared to sampling all parameters with the full likelihood.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of model evidence for each simulated dataset and model.

This was computed with sequential Monte Carlo (SMC), using the integrated likelihood and separately using the full likelihood. This was repeated for 8 random initialisations with 2000 draws at each step in SMC, and we present the mean and standard deviation of the model evidence from each run. For each approach, the model with the strongest evidence is marked with * (this is not clear for with rounding here, but see the repository [17] for full results). For each model, we also report the time taken to complete the sampling for all initialisations, averaged across the different datasets.

https://doi.org/10.1371/journal.pone.0280046.t001

We can also compare the posterior distributions for model coefficients, compared to the ‘true’ coefficients. Table 2 shows the Mahalabonis distance between b and the posterior distribution for β, for each combination of dataset and model. In every case, the ‘true’ coefficient is closer to the posterior distribution from the MCMC using the integrated likelihood than it is to the posterior from the MCMC using the full likelihood. The Mahalabonis distance is defined as the following, where the posterior for β has mean μ and covariance Σ:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of Mahalabonis distance between ‘true’ coefficient b and model posterior for β, for each simulated datasets and model.

This was computed with sequential Monte Carlo (SMC), separately using the integrated likelihood and the full likelihood.

https://doi.org/10.1371/journal.pone.0280046.t002

For the integrated likelihoods, the posterior mean and covariance can be recovered by averaging and (or and ) over all values of σ² (or and ) in the MCMC posterior trace. For the full likelihoods, the posterior mean and covariance are computed directly as the sample mean and sample covariance from the MCMC trace for β. For example, for , the Mahalabonis distances for integrated and full likelihoods are as follows, where and are the expressions in Eq 5:

Example: Minnesota radon contamination

We next investigate real-world data, the Minnesota radon contamination dataset [19]. We describe various models that fit within this framework outlined above, as proposed for this dataset in [19]. We deviate from their notation (e.g. renaming coefficients) for consistency with our notation above. The hierarchical structure in this dataset is given in decreasing geographic granularity, with 919 individual measurements grouped within 85 counties, and data are available at the individual measurement-level and the county-level. The maximum number of measurements per county is 116 and the minimum is 1. The explanatory variable is the measurement of the radon level on a logarithmic scale, which has mean (standard deviation) 1.265Â (0.819). Comparing across counties, the minimum and maximum values for the average log radon level was 0.410 and 2.606. The covariates considered here are an individual-level indicator variable identifying the floor the measurement was taken on (0 for basement, 1 for first floor), and county-wide uranium levels on a logarithmic scale. 83% of the measurements were taken in the basement, and the mean (standard deviation) of the log uranium levels was 0.014 (0.384), with a minimum and maximum across all counties of −0.882 and 0.528.

We denote the standardised (mean 0 and standard deviation 1) log radon measurements by y_ij, the indicator floor variable by t_ij, and the county-wide log uranium levels (also standardised) by v_j. Unless otherwise specified, we include an intercept term in each model, but adjust the model matrix x_ij so that it becomes . We could equivalently rewrite this as , where the indicator function, 1_A, is equal to 1 if condition A is True and 0 otherwise. This means that we index by t_ij rather than including it as a binary variable. The primary reason for this is that we then express the same prior uncertainty for measurements that come from the basement floor and from the first floor, instead of increased uncertainty when t_ij = 1, as discussed in [16]. In the context of linear model notation, we discard the group-level j index, so that the index i runs over all individuals, i = 1, …, n. To denote the group j ownership for a particular individual i, we instead using the index notation j[i]. For example, if the individual 10 belongs to group 4, then j[10] = 4. The models suggested by Gelman and Hill include the following single-level linear models:

• Complete pooling: all counties are pooled to a single group, with a single intercept and gradient used for all counties, whilst the county-wide uranium levels are not included in the model. By ‘averaging’ the intercept term, this completely ignores any variation in the radon levels across counties. The model is:
• Complete pooling, with county-level variables: as above, but with county-wide log uranium measurements included in the model. This at least contains some county-wide information, but does not directly model at the level of counties, as in the multilevel models.
• Unpooled intercept: each county has a separate intercept term. Although the county-level data is included via indicator variables that identify group membership, there is again no explicit model at the county-level. This is referred to as no pooling in the PyMC multilevel modelling notebook [16], though the coefficient for the floor/basement indicator variable is pooled across counties. We could also include the county-wide log uranium measurements here, but this will result in a non-identifiable model with collinear predictors.
• No pooling: each county is modelled completely independently of others, with separate intercepts and gradients. This will usually overfit the data, and perform relatively poorly for counties with limited data. In practice, 25 out of 85 counties have no measurements from the first floor, and we exclude those components in the vectors β and x_i. The dimension of β is then 85 + 60 = 145.

The multilevel models are the following:

• Partial pooling: county-wide variability is modelled directly as η_j, a deviation from the ‘average’ intercept. This uses first multilevel model formulation as described in (3).
• Varying slopes and intercepts: in this model, we allow variability in both intercept and slope (i.e. the floor the measurement was taken on) across counties. This uses the more general multilevel model (described in Eq 4). We evaluate a version of this that includes an off-diagonal (correlation) term in the Σ_η prior.

Where complete pooling and no pooling represent two extremes in model dimension within the linear model framework, Gelman and Hill [19] describe the multilevel model as akin to partial pooling, in which there is natural shrinkage of the non-pooled parameters (e.g. those featuring the index j[i]) to the mean (the ‘average’ in the complete pooling case). This can be seen as a compromise between the two linear model extremes.

In each of these models, we set a multivariate normal prior on β and inverse-gamma prior on each univariate variance component (σ², and ). In model , Σ_η was parameterised by , where the first two components were diagonal terms, which had priors, ρ_ν had a truncated normal prior on the interval [−1, 1] with mean 0 and variance 1, and the non-zero off-diagonal term was ρ_νσ_ν,1σ_ν,2. The number of unconstrained model parameters, k, in linear models is equal to the number of independent variables, which is the same as the dimension of the model evidence integral. In multilevel models, integration also happens over latent variables, while k is just number of independent variables plus the number of variance components. Table 3 compares these models, in terms of the model evidence and the AIC, where we use SMC to estimate the model evidence using the full likelihoods and the derived integrated likelihoods. S1–S6 Figs. shows measurements and model fits for a subset of counties.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of models for the Minnesota radon contamination dataset, using AIC and model evidence.

The model evidence was computed with sequential Monte Carlo (SMC), using separately the integrated likelihood presented and the full likelihood. This was repeated for 8 random initialisations with 2000 draws at each step in SMC, and we present the mean and standard deviation of the model evidence from each run. The table also shows the number of model parameters, k, and the ranking (where smaller is better) of models for each approach.

https://doi.org/10.1371/journal.pone.0280046.t003

Discussion

Multilevel structure within data unlocks an increasing number of modelling choices for statisticians, though this additional modelling flexibility presents a challenge in deciding what and how to model the data. We present an approach to Bayesian model selection for multilevel models that estimates the model evidence using integrated likelihoods instead of full likelihoods. We treat a subset of variables (regression coefficients with Gaussian priors) as nuisance variables that we analytically integrate out, which reduces the dimensionality of the model, as is standard in conjugate analysis. By converting the problem in this manner, we limit the impact of issues surrounding high-dimensional sampling, a key difficulty in sampling schemes for estimation of the desired quantities. As both examples show, estimates of the model evidence using the integrated likelihood are more consistent and robust. For the simulated data, this approach correctly identifies the ‘true’ model for each dataset and the ‘true’ coefficients are more closely described by the posterior distribution when using the integrated likelihood than when using the full likelihood. For a linear model with normal-inverse-gamma prior, we can also compute the log model evidence directly, and the estimates using the integrated likelihood have less bias and variance than those using the full likelihood. We believe the bias in the model evidence estimates using the full likelihood is likely shared by other models, particularly the multilevel models, because of higher dimensionality, though there is no gold standard to confirm this. These observations extend to the Minnesota radon contamination dataset, where the discrepancy between estimates and their variance is significant. The integrated likelihood is more consistent with the frequentist AIC, following a similar ranking, though this does not measure exactly the same thing. Although static SMC is asymptotically unbiased in the data size n [20], it is sometimes unclear, when dealing with high-dimensional models, what constitutes an unbiased estimate in practice when there is no analytic solution available. In high-dimensional settings, methods that directly estimate the model evidence integral may easily accumulate errors, leading to poor estimates. In Table 1, we notice some improvement in computational cost for the simulated datasets, but we believe the computational cost depends on a number of factors, such as dimensions n, d and m and the number of groups J, and so are reluctant to make a general statement about this. Sampling using a highly-nonlinear low-dimensional integrated likelihood may in some instances be more computationally challenging than using the high-dimensional product of simpler likelihoods. In the second example, both full and integrated likelihood methods were broadly similar in terms of computational cost for the linear model and simple multilevel linear model, but the integrated likelihood was more expensive for the general multilevel model, as this involved repeated computation and inversion of a large number of covariance matrices.

Bayesian model selection can be extended to a wide range of related problems fairly straightforwardly, such as variable selection and nested models (i.e. a comparison of two models where one is entirely contained with the other, as opposed to a nested structure in the data). It is worth emphasising a distinction between the model that best describes the data and the model that best achieves the research objective, which may not always coincide. For example, if the goal is to make inference on parameters associated with specific variables, then we should not exclude these variables on the basis of an evaluation of some model selection criteria. As George Box stated in one of the most well-known aphorisms in statistics [26]: ‘All models are wrong, but some are useful’. In a model selection problem, Bayesian approaches are particularly advantageous, because they factor prior uncertainty about model parameters in a way that naturally imposes a penalty on model complexity to prevent overfitting to the data. An important consideration is the choice of suitable priors for a given problem, to adequately balance previous scientific knowledge and information that the new data provides. Informative or weakly informative priors are typically preferred to non-informative priors, which will often not be suitable if there is insufficient data available. For linear models, the Bayes factor is a monotonic function of the classical F statistic in the limit as the prior variances tend to infinity [8]. Similarly, the posterior distribution of β given σ² also aligns with classical frequentist inference in this limit. It is worth emphasising that posteriori maximisation of the model evidence over prior hyperparameters is generally not appropriate in the context of an inference question about particular dataset. Some caution should be taken to avoid ‘retro-fitting’ priors based on the data, as this can be viewed as converting a priori fixed hyperparameters (part of the model definition) into tunable parameters, under which the inference question may not remain as initially intended. However, empirical Bayes [27], which performs such an optimisation on a wider dataset (for example, using radon contamination from other states to estimate sensible priors), can be viewed as a shrinkage approach and a bridge from frequentist estimation to the fully Bayesian approach. With a priori justification, it is certainly possible to compare a discrete set of models that are identical except from different prior hyperparameters. For example, two statisticians may have wildly different prior beliefs based on previous research, and therefore propose separate prior distributions, which in turn can influence inference they make on model parameters, and we could ask whose model best describes the data. As always, prior predictive checking should be used to ensure priors give a reasonable coverage in predicted values.

We have limited this work to the simplest generalised linear models, with normal distribution and identity link (the canonical link function), choosing normal priors to mirror conjugate priors for a Gaussian likelihood. As any likelihood from the exponential family has a conjugate prior distribution, this analytic marginalisation can be similarly extended to generalised linear models under similarly chosen priors; for example, logistic regression (generalised linear model with Bernoulli distribution and logit link) with a beta conjugate prior on the Bernoulli parameter p. This allows the approach we have presented to be generalised to a much larger class of data and models.

Data and source code availability

The source code and data is available in the following repository:

• Project name: Bayesian model selection for multilevel models using integrated likelihoods
• Project home page: https://github.com/tedinburgh/model-evidence-with-integrated-likelihood
• Operating system(s): Platform independent
• Programming language: Python 3.9.12
• Other requirements: Python modules—numpy 1.21.5 or higher, pandas 1.4.2 or higher, pmyc 4.1.5 or higher, arviz 0.12.1 or higher, scipy 1.7.3 or higher, argparse 1.1 or higher, statsmodels 0.13.2, palettable 3.3.0.
• License: MIT License

The current version of the repository has a permanent DOI at Zenodo [17]. The Minnesota radon dataset is contained within the module PyMC and can be opened directly from there and the simulated datasets can be reproduced exactly when running the relevant Python script from the repository above. Additionally, all datasets are available as.csv files on the repository above.

Supporting information

Acknowledgments

We would like to thank Dr Torben Sell (University of Edinburgh) for his insight and advice about challenges in MCMC sampling. A CC BY or equivalent licence is applied to the AAM arising from this submission.