Unscaled Evidence Strength.

According to unscaled evidence strength models, confidence reports depend only on the unsigned distance between the decision variable and the decision criterion [26–28]. In the case of the clockwise grating shown in Fig 1C, the evidence only weakly supports a clockwise decision. That is, the decision variable is close to the decision criterion, so a hypothetical observer may respond with low confidence. Unscaled evidence strength models do not independently quantify sensory uncertainty, such that a given distance will always be associated with the same level of confidence, regardless of other properties of the stimulus. These are perhaps the simplest models to reject because any empirical difference in observers’ confidence judgments across levels of sensory uncertainty cannot be accounted for by the model.

Scaled Evidence Strength.

According to scaled evidence strength models, confidence is assumed to be influenced not only by evidence strength but also by sensory uncertainty, the amount of noise in the signal. In scaled evidence strength models, an estimate of sensory uncertainty is used to scale evidence strength, for example, sensory uncertainty can be used to scale the confidence criteria used to delineate distance [29–32] or to normalise the distance itself [33, 34]. In effect this means that under conditions of high sensory uncertainty (light green line in Fig 1C and 1D), greater evidence strength is required to produce the same level of confidence as under conditions of low sensory uncertainty (purple line in Fig 1C and 1D). Several models fall into this class where different scaling rules can be applied. Adler and Ma (2018), for example, tested models where category and confidence response criteria were estimated as linear (depicted in main plot of Fig 1D) or quadratic (depicted in inset of Fig 1D) functions of sensory uncertainty. Here we consider category and confidence criteria that are linear and quadratic (and other) monotonic functions of sensory uncertainty, but we do not draw any theoretical distinctions between the different scaling rules.

Confirming that the stimulus manipulations affected category responses.

To determine the effect of the stimulus manipulations on category responses, we transformed stimulus values to ‘evidence for category 2’. For each task and modality, we computed the evidence for category 2 by evaluating the probability density of the presented stimulus value for the category 2 distribution, normalised by the total probability density for both category distributions for that stimulus value, (1) where s is the true value of the stimulus, C is the category, μ₂ and σ₂ are the mean and standard deviation of the category 2 distribution and N denotes the normal density function.

Eq 1 computes evidence for category 2 for each stimulus value, which could range from 0 (stimulus could be from category 1 only) to 1 (stimulus could be from category 2 only). Note, however, that for the different SDs task evidence for category 2 never reaches 0 because the category 1 distribution falls entirely within the range of the category 2 distribution. The minimum evidence value for this task is 0.2 (see Fig 2B).

We fit linear mixed models using a logistic link function to predict category response (1 or 2) from stimulus intensity and evidence for category 2, for each task and modality. The models included an intercept term, intensity, evidence and the interaction term between intensity and evidence as random effects. For each regression analysis, we report the odds ratio for the fixed effect and 95% confidence intervals in Table 1. As this analysis was intended to provide a descriptive summary of the effect of the stimulus manipulations on responses across modalities, we report other statistical metrics and follow-ups in S2 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. GLMM with Stimulus Manipulations as Predictors.

https://doi.org/10.1371/journal.pcbi.1011245.t001

As expected, stimulus values that indicated increasing evidence for category 2 were positively predictive of category 2 responses in all tasks and modalities (see ‘Evidence for Category 2’ in top panel of Table 1 for odds ratios). Increasing stimulus intensity was also positively predictive of category 2 responses for all tasks except the auditory different SDs task (see ‘Stimulus Intensity’ in top panel of Table 1 for odds ratios). These main effects were qualified by significant two-way interactions between category 2 evidence and intensity, for all tasks and modalities (see ‘Evidence × Intensity Interaction’ in top panel of Table 1 for odds ratios). As illustrated in Fig 3B, the effect of increasing category 2 evidence on category 2 responses was strongest for the highest intensity stimuli and decreased with stimulus intensity (see S2 Table for follow up tests). Of note, in the auditory modality the empirical data cluster across intensity levels, suggesting that participants did not appear to be able to distinguish between the two lowest intensity levels and the two highest intensity levels in the auditory modality in the same way as they did in the visual modality. The model predictions, however, do not capture this effect directly. We provide additional discussion of this feature of the data in S1 Text (see Figs A and B and Tables A and B).

These findings demonstrate that participants understood the category distributions across tasks and modalities. Specifically, when there was an increase in the relative probability that a stimulus was sampled from category 2, participants were more likely to report that that stimulus had been sampled from category 2. This was true regardless of the specific category structure (i.e., for both the different means and different SDs tasks) and stimulus modality. The effect of category 2 evidence was modulated by stimulus intensity, contrast for the visual tasks and loudness for the auditory tasks, such that as the intensity of the stimulus was reduced, category responses were more random and less influenced by the category distributions. This interaction between evidence and intensity was observed across modalities and confirmed that the chosen stimulus manipulations had the intended effect on participants’ category responses in both the visual and auditory domains. Because we were interested in quantifying the computations underlying metacognitive judgements across modalities, this analysis also provided evidence for similarities in the perceptual task requirements, allowing us to isolate any differences in metacognition itself [46].

Confirming that the stimulus manipulations affected confidence.

As above, the expected pattern of confidence responses differed across tasks (see Fig 2B). To compare the effect of the stimulus manipulations on confidence across the different versions of the tasks, we transformed stimulus values to ‘category diagnosticity’ prior to fitting the GLMMs. For each task and modality, we computed category diagnosticity by evaluating the probability density of the presented stimulus value for the most probable category distribution, normalised by the total probability density for both category distributions: (2) where s is the true value of the stimulus, C is the category, μ₁ and σ₁ are the mean and standard deviation of the category 1 distribution, μ₂ and σ₂ are the mean and standard deviation of the category 2 distribution and N denotes the normal density function.

Eq 2 computes category diagnosticity for each stimulus value, which could range from 0.5 (both categories are equally likely) to 1 (stimulus could be from the most probable category only). We fit linear mixed models to predict confidence from stimulus intensity and category diagnosticity, for each task and modality. The models included an intercept term, intensity, diagnosticity and the interaction term between intensity and diagnosticity as random effects. For each regression analysis, we report the fixed effect and 95% confidence intervals in Table 1.

Increasing category diagnosticity (see ‘Category Diagnosticity’ in bottom panel of Table 1 for coefficients) and stimulus intensity (see ‘Stimulus Intensity’ in bottom panel of Table 1 for coefficients) were predictive of increasing confidence in all tasks. These main effects were qualified by significant two-way interactions between category diagnosticity and intensity for all tasks (see ‘Diagnosticity × Intensity Interaction’ in bottom panel of Table 1 for coefficients). Increasing category diagnosticity was associated with increasing confidence but the strength of this effect was modulated by stimulus intensity. As illustrated in Fig 3C, the effect of increasing category diagnosticity was strongest for the highest intensity stimuli and decreased with lower stimulus intensities (see S2 Table for follow up tests). As observed in the category response data, the empirical confidence data also cluster across intensity levels in the auditory modality. See S1 Text for additional discussion.

Overall, these results indicate that as the relative evidence for a given category increased, participants’ confidence in their chosen category increased. Taken with the results of the category GLMMs, showing that participants’ category responses corresponded with the most probable category given the value of the stimulus, these findings suggest that participants understood the category distributions and used them to accurately guide their confidence responses. The effect of the category distributions on confidence was modulated by stimulus intensity, such that as the intensity of the stimulus decreased, confidence responses were uniformly low. This effect was observed across both modalities.

To further investigate similarities in the category and confidence response profiles across modalities, we also fit GLMMs to the evidence and intensity data from all tasks and modalities concurrently (see S2 Text). This allowed us to investigate if all the data could be described by the same set of effects or alternatively, if there were interactions between the effect of certain stimulus manipulations and the different task-modality configurations. These analyses (see Table A and Fig A in S2 Text) provided converging evidence that participants’ category and confidence judgements were associated with the stimulus features of interest in a similar way across modalities and tasks.

Log Posterior Probability Ratio.

In the log posterior probability ratio model, we assumed that the observer compared the log posterior probability ratio of the stimulus measurement to a set of category and confidence boundaries in log posterior probability ratio space. In the different SDs task, the main failure of this model was that it overpredicted confidence at lower intensity levels across modalities (see Fig E in S3 Text). Even though the boundary positions were sensitive to the estimates of Σ and the model was able to generate different predictions at each intensity level, the boundary parameters themselves were shared across all intensity levels such that the fit to low-intensity trials was constrained by high-intensity trials. This constraint did not appear to allow sufficient flexibility in the model to account for the distinct patterns of responses across intensity levels. We tested several variations of the log posterior probability ratio model which relaxed the assumptions of the Bayesian framework.

Log Posterior Probability Ratio with Orientation Dependent Noise.

Adding an orientation-dependent noise parameter did not improve the fit of the log posterior probability ratio model to visual modality data in either the different means task or the different SDs task. The predictions of this model still showed the same systematic deviations from the data as the log posterior probability ratio model. Specifically, both models overpredicted confidence at lower intensity levels across tasks (see Fig F in S3 Text).

Log Posterior Probability Ratio with Decision Noise.

Adding a Gaussian noise term on the calculation of the log posterior probability ratio improved the fit of the log posterior probability ratio model in all tasks and modalities, except the auditory different means configuration. Including the decision noise parameter shifted the best-fitting boundary parameters away from the means of the category distributions. As a result, the model predicted more low confidence, category 1 responses at the lower intensity levels for all stimulus values (see Fig G in S3 Text). This aligned with the data better in terms of confidence but overpredicted category 1 responses as the experimental data showed roughly equal proportions of category 1 and category 2 responses for a wide range of stimulus values at low intensities (i.e., participants are likely guessing a category more often). As a result of the shift in boundary positions away from the centre of the category distributions across all intensity levels, the model underpredicted confidence for stimulus values close to the mean of the category distributions and also underpredicted category 2 responses for extreme stimulus values at the higher intensity levels. In the different SDs task, the log posterior probability ratio model with decision noise was the best performing of the Bayesian models in both modalities. In the different means task, it was also the best performing Bayesian model in the visual modality. In the auditory modality, the log posterior probability ratio model with decision noise was outperformed by the log posterior probability ratio model with free category distribution parameters see Table 2.

Log Posterior Probability Ratio with Free Category Distribution Parameters.

The log posterior probability ratio model assumed that the observer knew the true values of the means and standard deviations of the category distributions. We tested a model in which the observer had imperfect knowledge of the parameters of the generative model, and we estimated these values as free parameters instead. Like the standard log posterior probability ratio model, this model overpredicted confidence at lower intensities and underpredicted confidence for stimulus values at the centre of the distributions at the highest intensity (see Fig H in S3 Text). This misprediction was seen across modalities. The model did, however, do a better job of matching confidence and proportion of category 2 responses at the more extreme values at higher intensities (relative to the log posterior probability ratio model with and without decision noise).

In the different means task, the best-fitting category distribution parameters were similar to the true category distribution parameters (see Best Fitting Model Parameters). In the different SDs task, the best-fitting category distribution parameters had a smaller standard deviation for the category 1 distribution and a larger standard deviation for the category 2 distribution in both modalities (see Best Fitting Model Parameters). The performance of this model demonstrated that even with a data-informed subjective prior, the Bayesian model class still produced qualitative mispredictions of category and confidence data in both the visual and auditory tasks. Overall, it appeared that regardless of the flexibility of the model, observed responses in the visual and auditory tasks were not fully consistent with a Bayesian framework in either modality.

Consistency in Best Performing Model Class.

The analyses described so far characterised how well each model captured the general patterns of responding across participants. As models were fit to data from each individual, however, we were also able to examine the best-fitting model for each participant. If a single class of models reliably characterised the computational processes underlying participants’ category and confidence decisions collectively, we would expect that data from individual participants should also be best characterised by the same class of models across modalities and tasks. If, however, a single class of models did not reliably characterise the underlying computational processes, we would see variation in the performance of the model classes across individuals. Our goal was not to investigate individual differences per se, which would require a larger sample size, but to examine the consistency in model fits across participants.

As shown in Table 2, in both the visual and auditory modality the scaled evidence strength models performed best for all participants. This analysis revealed that even at the individual level, there was a preference for the scaled evidence strength class across modalities and tasks, consistent with the summed AIC and BIC scores which were the lowest for the scaled evidence strength models, as described above.

Consistency in Best Performing Model.

While the main motivation of the model-based comparison was to focus the analysis at the level of model classes, to further understand the specific implementation of the scaled evidence strength models, we also examined differences at the individual model level. We compared the best performing model for each participant and found that within the scaled evidence strength class, there were notable differences in model performance.

In the auditory different means task, either the linear model (62.5% of participants), quadratic model (25% of participants) or free-exponent model (12.5%) performed best for individuals, while the free-exponent model was the best performing model on average. For the visual different means task, by contrast, either the free-exponent model (with orientation dependent noise; 75% of participants) or the quadratic model (25% of participants) was the best performing model for participants best characterised by a scaled evidence strength class model. In the different SDs task in the visual modality, either the linear (25%), quadratic (50%) or free-exponent (25%) performed best and in the auditory modality, the free-exponent model was preferred (62.5%; remaining 37.5% for linear). Therefore, although the scaled evidence strength class consistently outperformed the other model classes, a specific scaling rule within that class (linear, quadratic or other exponent) did not seem to apply consistently across different versions of the task, across modalities or across participants.

The importance of accounting for sensory uncertainty.

Our model comparison results indicated that observers modulate their confidence with sensory uncertainty, not just evidence strength, in both the visual and auditory domain. Specifically, the unscaled evidence strength models, in which confidence depends entirely on the strength of the signal, performed consistently worse than the other model classes across all task-modality configurations. Unlike the other model classes, the unscaled evidence strength class did not independently quantify sensory uncertainty, such that a given distance between the decision variable and the decision criterion was always associated with the same level of confidence [26–28]. This finding is particularly relevant because the importance of sensory uncertainty for confidence has not yet been demonstrated in the auditory modality. Thus, we were able to provide evidence that observers take sensory uncertainty into account across modalities, contributing to our limited understanding of the computational basis of confidence across domains. Furthermore, our study provides insight into the extent to which these sensory-uncertainty-dependent processes are consistent with Bayesian or non-Bayesian accounts of confidence.

Implications for Bayesian models of confidence.

Although we found consistent evidence that observers take sensory uncertainty into account, we observed significant differences between our empirical data and the predictions of the Bayesian models. In our tasks, the Bayesian class of models assumed that observers use knowledge of the level of sensory uncertainty associated with the measurement of the stimulus and (sometimes veridical) knowledge of the generating category distributions to compute a log posterior probability ratio for category membership. The log posterior probability ratio was then compared with a set of thresholds in log posterior probability ratio space to determine confidence. We found that this class of models was outperformed by the scaled evidence strength class, both for the group as a whole and for all individual participants. The relatively poor performance of the Bayesian models did not seem to be due to lack of flexibility. The Bayesian models continued to perform poorly even after allowing for a noise term in the calculation of the log posterior probability ratio or inaccurate subjective beliefs about the generative model (i.e., freely estimated category distribution parameters).

While the poor performance of the Bayesian models is in disagreement with some studies [3, 15, 37, 40, 42, 43], it is consistent with others [3, 29, 33, 44, 45, 48]. Importantly, however, our study extends existing research on Bayesian models of confidence in two ways. First, most previous studies have focused exclusively on the visual domain. Here we compared Bayesian and non-Bayesian accounts of confidence in both the visual and auditory domains, thus showing converging evidence for non-Bayesian metacognitive computations in a broader range of sensory contexts. Second, most previous studies have used more simple task structures. Here we found that the Bayesian class did not perform as poorly when there was a single clearly defined category boundary, as in the different means task. In the different SDs task, however, where the categories overlapped to a greater extent and optimal categorisation required observers to consider the variability of both categories, the Bayesian models performed much worse. This suggests that the support for Bayesian computations in previous research may be partly attributed to low task complexity. Because the computations involved in confidence generation become increasingly complex as the task structure becomes more complex (i.e., tasks where there is a high overlap of exemplars from different categories), deviations between human behaviour and predictions from Bayesian principles may become more pronounced. Our findings suggest that future research evaluating Bayesian models should rely more on complex category structures, that likely reflect the dynamic conditions encountered in the real world, to assess human behaviour relative to Bayesian optimality.

Although our study used a highly representative set of Bayesian models from the broader confidence literature, we could not feasibly test the full space of possible models that might be considered “Bayesian”. We, therefore, cannot rule out the entire Bayesian class of confidence models completely. In particular, our Bayesian models may have been limited by only incorporating a point estimate of sensory uncertainty, rather than a distribution over uncertainty [34]. Furthermore, we did not consider models with non-Gaussian noise, lapse rate parameters or non-Gaussian priors. Our findings, nonetheless, strongly suggest confidence judgements in the visual and auditory modalities are unlikely to derive from posterior probability computations as defined by the set of Bayesian models considered here. Thus, although Bayesian models are particularly powerful for their generalisability, that is they involve the computation of a probability metric which can be compared directly across domains, their poor fit to empirical data highlights the need to explore non-Bayesian frameworks that can better capture human behaviour in different contexts. Our study, in particular, demonstrates that non-Bayesian algorithms such as the scaled evidence strength models can operate on standardised units and provide a powerful account of human confidence judgements across modalities.

Non-Bayesian sensory uncertainty dependent computations.

We found that the scaled evidence strength class of models, in which positioning of the confidence and choice criteria depended on a point estimate of sensory uncertainty but did not involve the computation of posterior category probabilities, provided the best account of category and confidence responses across modalities [29, 30, 33]. We found consistent evidence for this class of models both at the group level, where the models were best able to capture the main patterns of behaviour across participants in both modalities and tasks, and at the individual participant level, where the same class of models was preferred for all participants across modalities and tasks. Using response criteria that depend on an estimate of sensory uncertainty allows observers to avoid assigning high confidence to highly variable stimuli [33, 49], such that choice/confidence boundaries can be compressed (or expanded, where appropriate) under conditions of high sensory noise and vice versa under conditions of low sensory noise. This sort of algorithm produces behaviour that is similar to the optimal Bayesian observer but is less computationally expensive, as it does not involve the computation of the posterior probability distribution [29].

A Scaling Factor for Sensory Uncertainty.

Although we found that observers use sensory uncertainty-dependent boundaries for category and confidence judgements, the specific way sensory uncertainty scales the boundaries was not universal across modalities or tasks. Within the scaled evidence strength class, we tested different assumptions about the scaling rule that was used to adjust the confidence criteria. We compared models which used either a linear (k + mσ), quadratic (k + mσ²), or free-exponent (k + mσ^a) scaling rule. We estimated a value of k and m for each boundary, which allowed for the strength of the effect of sensory uncertainty on the positioning of the boundaries to differ across response options (i.e., in the different means tasks, the confidence 4 boundary could shift more than the confidence 1 boundary). In the case of the linear scaling rule, m, was a constant that was applied to the estimate of sensory uncertainty which determined the magnitude of the sensory-uncertainty-based shift in choice/confidence boundaries. In the cases of the non-linear scaling rules, an exponent was also used on the estimate of sensory uncertainty assuming that the impact of sensory uncertainty on confidence criteria was not proportional to the amount of sensory uncertainty. Rather, these models assumed that the impact of sensory uncertainty became more pronounced at higher levels. These different models allowed us to compare the relative importance of different scaling rules which quantified different assumptions about the effect of sensory uncertainty on choice/confidence criteria. We found, however, that both the linear and non-linear scaling rules accounted for the empirical data well and we did not find consistent evidence in favour of either alternative.

The lack of consistency within the scaled evidence strength class suggests that the scaling of sensory uncertainty itself is important, but that the scaling factor depends on contextual factors such as modalities, task structure and individuals. Thus, it is likely that the scaling rule used to update confidence criteria reflects properties such as task difficulty or stimulus familiarity, rather than reflecting qualitative differences in the computational processes used by individuals within or across modalities. From a measurement perspective, this supports the case for freely estimating the scale factor, as it potentially provides a better assessment of how people’s confidence judgments are affected by sensory uncertainty. Future research might benefit from using this approach to investigate what contextual factors moderate different strategies (and scale factors) for adjusting response criteria by an estimate of sensory uncertainty.

Different means task.

As shown in Fig 2, category distributions in the different means task had different means and the same standard deviation. In the visual modality, where 0 degrees represented a horizontal Gabor patch, orientations that were rotated counter clockwise relative to horizontal (i.e., negative orientations) were more likely to be sampled from category 1 (μ_cat1 = −4°, σ_cat1 = 5°). Orientations that were rotated clockwise relative to horizontal (i.e., positive orientations) were more likely to be sampled from category 2 (μ_cat2 = 4°, σ_cat2 = 5°). In the auditory modality, the category distributions had the same properties, but parameter values were shifted into the frequency domain. Lower frequency tones were more likely to be sampled from category 1 (μ_cat1 = 2300 Hz, σ_cat1 = 475 Hz) and higher frequency tones were more likely to be sampled from category 2 (μ_cat2 = 3100 Hz, σ_cat2 = 475 Hz).

Stimuli with orientation/frequency values equal to where the two category distributions intersected, at 0 degrees for orientations and 2700 Hz for frequencies, were equally likely to be sampled from either category 1 or category 2. Because the relative likelihood of category membership changed monotonically around this point, to maximise correct responses, an ideal observer would report that any stimulus value below this point was sampled from category 1 and any stimulus value above this point was sampled from category 2.

Different SDs task.

In the different SDs task, category distributions had the same mean and different standard deviations. In the visual modality, larger rotations clockwise or counter clockwise relative to horizontal were more likely to be sampled from category 2 (μ_cat2 = 0°, σ_cat2 = 12°) whereas smaller rotations relative to horizontal were more likely to be sampled from category 1 (μ_cat1 = 0°, σ_cat1 = 3°). In the auditory modality, lower and higher frequency tones were more likely to be sampled from category 2 (μ_cat2 = 2700 Hz, σ_cat2 = 500 Hz), whereas intermediate frequency tones were more likely to be sampled from category 1 (μ_cat1 = 2700 Hz, σ_cat1 = 125 Hz).

To maximise correct responses, an ideal observer would use the two points where the category distributions intersect, at -5 and 5 degrees for orientations and 2485 Hz and 2915 Hz for frequencies, reporting that stimulus values contained within these intervals were sampled from category 1 and stimulus values outside these intervals were sampled from category 2.

Visual stimuli.

The visual stimuli were drifting Gabors which had a spatial frequency of 0.5 cycles per degrees of visual angle (dva), a speed of 6 cycles per second, a Gaussian envelope with a standard deviation of 1.2 dva, and a randomized starting phase. Visual stimuli appeared at fixation for 50 ms.

Auditory stimuli.

The auditory stimuli were tones of varying frequency (including 5 ms linear onset and offset amplitude ramps to eliminate onset and offset clicks). Stimuli were synthesized with an ASIO4ALL sound driver with a sampling rate of 28 kHz. Tones were matched for loudness according to the International Standard ISO 226:2003: Acoustics- Normal Equal-Loudness-Level-Contours (International Standardization Organisation, 2003) and extensive piloting was undertaken to select intensity levels. Auditory stimuli were presented in stereo via headphones (Sennheiser HD 202, see S7 Text for calibration details) for 50 ms.

Category training.

At the start of each training trial, participants fixated on a central cross for 1 s. The fixation cross was then extinguished, and a stimulus was presented. A stimulus value (i.e., orientation for visual stimuli and frequency for auditory stimuli) was sampled randomly from the relevant category distribution and the stimulus was presented (50 ms duration for auditory stimuli and 300 ms duration for visual stimuli). Immediately after the offset of the stimulus, participants were able to respond category 1 or category 2 by pressing the F or J key on a standard keyboard with their left or right index finger, respectively. No confidence ratings were collected during training. After participants made their response, corrective feedback (i.e., the word “correct” in green or “incorrect” in red) was displayed for 1.1 s. The inter-trial interval was 1 s, after which the fixation cross reappeared. Within a training block, stimuli were sampled equally from each category and the order of categories was randomised across trials.

Participants completed 3 training blocks (120 trials per block, in total 360 trials) per session. During training, only the highest intensity level was used for stimulus presentation.

Test.

The trial procedure in testing blocks was the same as in training blocks, except that trial-to-trial feedback was withheld, stimuli were presented at four different intensity levels (different contrast values for the visual stimuli and different loudness values for the auditory stimuli) and stimuli were presented for 50 ms, regardless of modality. Participants then made a category response, followed by a confidence report, using the 1–4 number keys to indicate their confidence. We chose to use a 4-point confidence rating scale as it is common in the study of confidence and allowed for meaningful comparison of models to previous research [29, 48] (see [62] for investigation of alternative methods for collecting confidence ratings).

At the end of each block, participants were required to take at least a 30 s break. During the break, they were shown the percentage of trials they had correctly categorized in the most recent block. Participants were also shown a list of the top 10 block scores (across all participants, indicated by initials) for the task they had just completed. This was intended to motivate participants to maintain a high level of performance. Participants completed 6 testing blocks per session (120 trials per block, a total of 720 trials per session). Within a testing block, an equal number of test stimuli were sampled from each category and each intensity level (i.e., 15 trials per cell of the design). This meant that within a testing block, participants saw stimuli from both categories presented at every intensity level. The order of both category and intensity level were randomised.

Measurement Noise.

In all models, we assume that the observer’s internal representation of the stimulus, x, is a noisy measurement. We approximate this measurement noise using a Gaussian distribution (referred to as the ‘measurement distribution’) in both modalities. Although orientation is circular and traditionally modelled using a von Mises distribution, we used a Gaussian distribution because we only used a small range of orientations in both the visual tasks [29]. The Gaussian distribution was centred on the true stimulus value presented on trial i, s_i, with standard deviation, σ. We assume that σ depended on stimulus intensity, where greater stimulus intensity was associated with less measurement noise. In contrast to Adler and Ma (2018), we did not enforce a power law relationship between intensity and measurement noise because we were agnostic about the functional form of this relationship. Therefore, the standard deviation of the measurement distribution was estimated separately at each intensity level with a monotonicity constraint, (4) where N denotes the normal density function. The standard deviation of the measurement distribution, therefore, approximated the level of sensory uncertainty in the observer’s perception of the stimulus.

Orientation Dependent Noise.

Following Adler and Ma (2018), for modelling performance in the visual modality, we also tested a variation of some models which assume additive orientation-dependent noise in the form of a rectified 2-cycle sinusoid [63]. In these models, the standard deviation of the measurement distribution was: (5) where I represents the intensity level, γ_I is a free parameter representing the baseline amount of measurement noise at a given intensity and ψ scales the amount of additive measurement noise across stimulus orientations. We did not fit a frequency dependent noise parameter in the auditory tasks as each frequency was corrected for equal loudness prior to adjusting overall intensity.

Different means task boundaries.

In the different means task, we assumed that one of the boundaries, b₁, divides the stimulus range into the two different category response regions. That is, where the observed stimulus value is below this boundary, a category 1 response is given and where the observed stimulus value is above this boundary, a category 2 response is given. Confidence associated with the category response is determined by a further three boundaries, b₂, b₃, b₄, which created four different confidence response regions within each category. Within a category response region, we assumed that the confidence boundaries increased monotonically across successive boundaries such that higher confidence response regions were aligned with more extreme category evidence (see Fig 4B). We also assumed that the confidence boundaries were symmetrically positioned across categories. Due to symmetry, confidence regions on equidistant but opposite sides of the category boundary, b₁, were treated as equivalent by the model. The boundaries in the different means task were ordered such that: −b₄ < −b₃ < −b₂ < b₁ < b₂ < b₃ < b₄, where the position of each boundary was freely estimated, but constrained by monotonicity.

Different SDs task boundaries.

In the different SDs task, two boundaries, −b₄ and b₄, divided the stimulus range into two different category response regions. Where a stimulus value falls within the −b₄ and b₄ region, a category 1 response is given. Where a stimulus value falls beyond −b₄ or b₄, a category 2 response is given. To make a confidence response, the observer uses 4 symmetrical boundaries for each category: b₁−b₄ for category 1 responses and b₄−b₇ for category 2 responses, as shown in Fig 4B. In all models, we assumed that the boundaries increased monotonically across successive boundaries such that higher confidence response regions were aligned with more extreme category evidence (see Fig 4B). We also assumed that the boundaries are symmetrically positioned around the means of the category distributions. For ease of exposition, we refer to this centre point as b₀. We assumed the boundaries are symmetrical in this task because two measurements that are equidistant from the middle of the category structure but on opposite sides of b₀ should logically have the same confidence associated with them. Due to symmetry, confidence regions on equidistant but opposite sides of b₀ were treated as equivalent by the model. The boundaries in the different SDs task were ordered such that: .

Unscaled evidence strength models: Distance Model.

In the distance model, the boundary positions were not dependent on (estimated) values of sensory uncertainty (σ) and therefore occupied fixed positions across intensity levels. The position of each boundary in perceptual space was described by a free parameter, k, where perceptual boundaries b_r = k_r.

Scaled evidence strength models: Linear, Quadratic and Free-Exponent Models.

In the linear and quadratic models, the positions of the boundaries were linear or quadratic functions of σ. Each boundary position depended on 3 free parameters: k, m and σ. k, represented the base position of each boundary which was offset by a scaled value of sigma, ±mσ (linear model), or by a scaled value of sigma squared, ±mσ² (quadratic model), where m was fixed for each boundary. The resulting boundaries in perceptual space were given by b_r(σ) = k_r+m_rσ (Linear) and b_r(σ) = k_r+m_rσ² (Quadratic). In the free-exponent model, we estimated the exponent on σ, a, as a free parameter, b_r(σ) = k_r+m_rσ^a. The additional free parameter allowed for the changes in boundary position across intensities to occupy values between and beyond linear or quadratic values. We constrained a to be greater than 1.

Parameter settings of the free-exponent models across modalities.

While the primary focus of this study was to determine whether a single class of metacognitive models could account for the way participants make confidence judgements across modalities, we also wanted to determine the extent to which this process was sensitive to different parameter settings. Accordingly, rather than fit the model to each modality separately, we fit the free-exponent model to data from both modalities simultaneously. We formulated several levels of flexibility in the parameter settings, with more flexible versions allowing the parameter values to be different across modalities and less flexible versions constraining parameter values to be the same across modalities. We chose the free-exponent model to test the flexibility in parameter settings, as it was the most flexible of the scaled evidence strength models and included the linear and quadratic models as special cases.

Common settings model.

For the common settings model, both modalities had the same parameter settings. We assumed that sensory uncertainty (σ) did not differ across modalities. Thus, we fit σ as a free parameter for each intensity level and used the same value of σ across modalities. We also assumed that the same boundary positions were used across modalities so that all perceptual boundaries were given by b_r[I](σ) = k_r+m_rσ_[I]^a where I is the intensity level.

Different noise settings model.

For the different noise settings model, sigma was a free parameter for each intensity level in each modality. k and m were fixed across modalities, assuming that only measurement noise changed across modalities. The perceptual boundaries were given by , where I is the intensity level and M is the sensory modality.

Flexible settings model.

In the flexible settings model, all parameters in the standard free-exponent model were freely estimated in each modality.

Log posterior probability ratio.

When the perceived value of a stimulus on a given trial is x, we assume that the observer uses the log posterior probability ratio to make a decision. The log posterior probability ratio represents the (log) ratio of posterior beliefs about category membership, given the properties of the stimulus, such that . When the log posterior probability ratio is positive, there is greater evidence for category 1. We assumed that category and confidence reports depended on the sensory signal, x, only via d. On a given trial, the observer chooses a response by comparing d to a set of boundaries k = (k₁,k₂…k₇) that partitions d space into eight response regions, each representing a unique category and confidence combination. The positions of these boundaries in d units were free parameters. Following Adler and Ma (2018; see S8 Text), we used the following task-specific derivations of d for the different means task: (6) and the different SDs task: (7) where μ₁ and σ₁ are the mean and standard deviation of the category 1 distribution, σ₂ is the standard deviation of the category 2 distribution and σ is the fitted value of measurement noise, approximating the amount of sensory uncertainty in the observers’ internal representation of the stimulus (see Eq 4). In terms of the generative model, in the different means task the observer needs knowledge of μ₁ and σ₁ to make an optimal decision and in the different SDs task the observer needs knowledge of σ₁ and σ₂ to make an optimal decision. Because we sampled equally from each category, we assumed that p(C = 1) = p(C = 2) = 0.5 such that is 0 in both Eq 6 and Eq 7.

Log posterior probability ratio with decision noise.

For the log posterior probability ratio model with decision noise, we assumed that there was an added Gaussian noise term on d, σ_d. This allowed for the observer’s calculation of d to be noisy, in addition to sensory noise in the internal representation of the stimulus, σ_i [29].

Log posterior probability ratio with free category distribution parameters.

The log posterior probability ratio model assumes that the observer knows the true values of the means and standard deviations of the category distributions, μ₁ and σ₁ in Eq 6 and σ₁ and σ₂ in Eq 7. We also tested a version of the model in which the observer had imperfect knowledge of the parameters of the generative model, and estimated these values as free parameters instead.

Fitting the models to data.

Model parameters were estimated for each participant using maximum likelihood estimation (see S9 Text). Below we describe the process for calculating the likelihood of the dataset, given a model with parameters θ, for the non-Bayesian and Bayesian models, as they require different transformations.

Non-Bayesian models.

For all non-Bayesian models, the boundaries were positioned in perceptual space and no transformation of boundary positions was required. For each trial, we determined the predicted response probability for each combination of category and confidence level by computing the probability mass of the measurement distribution between adjacent boundaries, given the parameters of the model and the participant’s response on that trial, r_i (cf. [64]; see Fig 4D). For the different means task, this quantity is: (8) where b₅ = ∞ and −b₅ = −∞. For the different SDs task, this quantity is: (9) where b₀ = 0, b₈ = ∞ and −b₈ = −∞.

Bayesian models.

In the Bayesian models, boundaries are positioned in d-space (log posterior probability ratio space). To evaluate the probability mass of the measurement distribution, we transformed the boundary positions into perceptual space, b(σ), using parameters k as the left-hand side of Eq 6 or Eq 7 and solved for x at the fitted levels of σ. Solving Eq 7 for x requires calculating the square root of d, which is undefined for negative values. We therefore used the absolute value of d to solve Eq 7 for x and applied the appropriate sign to the result to arrive at a final standardised orientation/frequency value for x. Where the category distribution means and standard deviations were free parameters, we used the estimated values in place of the true mean and/or standard deviation/s (μ₁ and σ₁ for the different means task and σ₁ and σ₂ for the different SDs task). We then calculated the probability mass of the measurement distribution between adjacent perceptual boundaries using Eq 8 or Eq 9, given the task and participant’s response for that trial.

Log Posterior Probability Ratio with Decision Noise.

For the log posterior probability ratio model with decision noise, we took 101 evenly spaced draws from a normal distribution (spanning the 1^st to 99^th distribution percentiles) for each boundary, with k as the mean and the standard deviation as a free parameter, σ_d. Each of the 101 draws was then converted into perceptual space using Eq 6 or Eq 7 to solve for x at the fitted levels of σ. We then calculated the probability mass of the measurement distribution between corresponding draws of the relevant boundaries, given the participant’s response for that trial. This meant that for a single trial we had 101 response probabilities. We calculated the weighted sum of these probabilities using the normalised densities for each draw of k.

Log likelihood of the dataset.

To obtain the log likelihood of the dataset, we computed the sum of the log probability for every trial i, where t is the total number of trials.

(10)

Non-Bayesian models.

To visualise the fit of the non-Bayesian models, we generated a synthetic dataset of model responses using the best fitting parameters for each participant. Using the experimental data for that participant, for each trial we took 100 samples from a normal distribution with mean s_i (the true stimulus value) and standard deviation σ_I (estimated value of σ for a given intensity I). We then compared each sample to the boundary parameters (b₁−b₃ for the different means task and b₁−b₇ for the different SDs task) to generate 100 category/confidence responses for that trial. We took the mean of the generated category/confidence responses to produce a single average predicted response for that trial.

Bayesian models.

As with non-Bayesian models, for each trial, we took 100 samples from a normal distribution with mean s_i and standard deviation σ_I. We transformed boundary parameters from probability space into perceptual space (using the relevant equation from the model specification section) and compared each stimulus sample to the resulting boundaries to generate a predicted response. This approach generated 100 category/confidence responses for each trial, and we took the mean of the generated category/confidence responses.

Bayesian models with decision noise.

For Bayesian models with decision noise, we compared each stimulus sample to 100 random draws of the category/confidence boundaries. We sampled from a normal distribution with standard deviation σ_d and mean k. Each draw was converted into perceptual space using the relevant equation and σ_I. We generated 100 × 100 category/confidence responses for each trial and took the mean of the generated category/confidence responses to produce a single average response for that trial.

Generating figures.

For plots with stimulus value (or a transformation of stimulus value) on the horizontal axis, stimulus values were binned so that each bin consisted of the same number of trials. For visualisation, we calculated the mean category/confidence response across all participants for both the empirical data and model predictions for each bin.