Peripheral representation of HCTs at high frequencies

Although many studies have recorded auditory-nerve responses to the types of HCT stimuli used in pitch experiments [13,20,48,49], these have not included HCTs at the very high frequencies used in recent human psychophysical work [36–38]. Moreover, recent work has revealed significant differences in peripheral coding between humans and the smaller mammals commonly used in auditory physiology experiments [29,50,51], raising questions as to how useful auditory-nerve recordings of pitch-evoking stimuli in animals such as guinea pigs or chinchillas are in understanding how pitch stimuli are represented in the human auditory periphery. To develop a better understanding of the availability and quality of different F0-related cues at high frequencies as compared to low frequencies in the human auditory periphery, we simulated human auditory-nerve responses to HCTs at low and high frequencies using a modern phenomenological model of the auditory nerve [52], with parameters adjusted to match what is known about human cochlear tuning. This was done after first validating that the cat version of the model could qualitatively replicate key data from relevant studies in cat (S1 Text).

Low-numbered harmonics were at least partially resolved by the model filterbank (emulating the mechanical filtering of the basilar membrane) at both low and high F0s, as reflected by prominent peaks in the pattern of average firing rates over characteristic frequency (CF) in the auditory-nerve population (Fig 1A and 1B, right panel). At low F0s, model fibers tuned to resolved components also demonstrated robust phase locking to the underlying TFS, whereas fibers tuned to unresolved components instead responded with a prominent modulation at F0 (Fig 1A, bottom panel). At high F0s, component frequencies were too high to produce phase-locked responses in the model auditory nerve. As a result, model fibers tuned to resolved components showed responses with little in the way of temporal structure, whereas fibers tuned to unresolved harmonics showed strong modulations at F0 (Fig 1B, bottom panel). At both low and high frequencies, model fibers tuned between resolved components showed responses modulated at F0 (Fig 1A and 1B, bottom panel).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Representation of complex tones in a simulated auditory nerve.

(A). Simulated responses of a population of high-spontaneous-rate auditory-nerve fibers [52] for five periods of a sine-phase HCT composed of harmonics 4–13 at a level of 50 dB SPL per component and an F0 of 280 Hz. The middle panel shows a “neurogram”, or a plot of instantaneous firing rate as a function of time and characteristic frequency. In the neurogram, color (from purple [low] to yellow [high]) indicates the instantaneous firing rate analogous to color indicating intensity in a spectrogram. The top and left panels show the temporal waveform and spectrum, respectively, of the acoustic stimulus. In the left panel, the yellow box highlights the frequency ranges used in the behavioral experiments. The bottom two panels and right panels show the responses of individual nerve fibers over time and the response profile averaged over time, respectively, of the simulated neural response. The bottom two panels show responses for auditory-nerve fibers tuned to component frequencies (4F0, purple; 12F0, blue) or tuned between component frequencies (5.5F0, pink; 9.5F0, maroon). (B). Same as A, except for an F0 of 1400 Hz.

https://doi.org/10.1371/journal.pcbi.1009889.g001

The model simulations suggest that if temporal information is utilized at high frequencies, then it is likely to be based on the envelope modulations at F0, rather than on the phase-locked responses to individual harmonics (an observation further reinforced by our ideal-observer modeling below). However, several lines of evidence suggest that humans cannot utilize that envelope information effectively. First, studies using HCTs composed of unresolved harmonics (i.e., all harmonics above the 10^th) or modulated noises show that the pitch of such stimuli, which is conveyed exclusively by temporal-envelope cues, is weak, yields poor F0 discrimination, and is non-existent for F0s above about 700 or 800 Hz [32–34,40,42]. Second, melody discrimination is possible at very high frequencies (> 7.5 kHz) for HCTs with F0s between 1 and 2 kHz, but when the harmonics are shifted to produce inharmonic tones, performance drops to near chance, even though the temporal-envelope cues are maintained [39]. Third, both melody perception and F0 discrimination remain accurate at high frequencies when odd and even harmonics are presented to opposite ears, even though the temporal-envelope repetition rate in each ear is thereby doubled [38,39]. These lines of evidence, which we evaluate and reconsider further in light of our own results below, suggest that at least some information in the temporal response pattern of the auditory nerve at high frequencies may not be utilized perceptually.

F0 discrimination is affected by HCT maskers at low and high frequencies

If listeners are not using temporal information to perform F0 discrimination at high frequencies, then they must be relying on rate-place information, which in turn relies on the presence of some spectrally resolved harmonics. We tested this idea by attempting to restrict the availability of resolved harmonics via the addition of concurrent HCT maskers. We hypothesized that this stimulus manipulation would yield particularly poor performance at high frequencies (where resolved harmonics are necessary due to the use of a rate-place code) as compared to low frequencies (where resolved harmonics may not be necessary, due to the additional availability of TFS cues, even in the presence of the masker).

Experiment 1.

We began by measuring F0 discrimination at both low and high frequencies and by determining how a single HCT masker impaired F0 discrimination. Experiment 1 measured F0 discrimination thresholds for bandpass-filtered HCTs at low frequencies (F0 = 280 Hz, frequency range = ~1.5–3.0 kHz) and high frequencies (F0 = 1400 Hz, frequency range = ~8–14 kHz). Two conditions were tested: ISO, in which test tones (target and reference) were presented in isolation (except for masking noise in the background), and GEOM, in which target tones with masking noise were presented concurrently with a spectrally overlapping HCT masker with an F0 that was geometrically centered between the F0s of the reference (lower-F0) and the target (higher-F0) tone. The masking noise was broadband threshold-equalizing noise (TEN) [53] with a level within the estimated equivalent rectangular bandwidth (ERB) of the human auditory filter around 1 kHz [54] that was 10 dB below the level per component of the HCTs. Two stimulus variants were tested. In the first variant (Experiment 1a), the test tones contained only harmonics 6–10 of the F0. To help rule out the possibility that listeners were using the spectral edge of the stimulus, rather than the F0, to complete the task, a second variant (Experiment 1b) was tested, in which the test tones contained all harmonics of the F0 up to the Nyquist frequency (i.e., half the sampling rate), but were bandpass filtered with a zero-phase 12^th order Butterworth bandpass filter passing harmonics 6–10 of the nominal F0.

The results of Experiment 1 are plotted in Fig 2. An analysis of variance (ANOVA) revealed significant main effects of F0 (low or high) [F(1, 22.98) = 54.57, p<0.001] and masker (present or absent) [F(1, 22.46) = 149.37, p<0.001] as well as significant two-way interactions between F0 and masker [F(1, 22.29) = 44.71, p<0.001] and between masker and experiment (Experiment 1a or Experiment 1b) [F(1, 130.42) = 9.02, p = 0.013]. No other model terms reached significance. The significant main effects reflected the trends observed in Fig 2 that listeners achieved better (lower) F0 discrimination thresholds at low frequencies than at high frequencies both for the ISO condition [estimated ratio = 5.14, F(1, 22.82) = 63.41, p<0.001] and the GEOM condition [estimated ratio = 2.14, F(1,22.82) = 29.80, p<0.001] and that they achieved better F0 discrimination thresholds in the absence (ISO) than in the presence (GEOM) of the masker for both low frequencies [estimated ratio = 3.70, F(1, 22.69) = 210.15, p<0.001] and for high frequencies [estimated ratio = 1.54, F(1, 22.49) = 17.00, p<0.001].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Behavioral results from Experiment 1.

The left panel (Exp1a) shows results with only harmonics 6–10 present in the target; the right panel (Exp1b) shows results with the bandpass filtered target, with harmonics 6–10 in the passband. Large filled circles and error bars indicate the average F0DL and ±1 standard error of the mean (SEM). The small filled circles and error bars indicate individual F0DLs and ±1 SEM for each participant.

https://doi.org/10.1371/journal.pcbi.1009889.g002

Our initial hypothesis was that the addition of a masker (i.e., the GEOM condition) would reduce the availability of rate-place cues and so would result in particularly poor F0 discrimination at high frequencies. In fact, contrary to our hypothesis, an interaction contrast test revealed that the difference in performance between the ISO and GEOM conditions was larger at low frequencies than at high frequencies [estimated ratio of ratios = 2.40, F(1, 22.29) = 42.71, p<0.001]. There are many possible explanations as to why the GEOM masker worsened performance more at low frequencies than at high frequencies. One possibility is that the smaller effect of the GEOM masker at high frequencies reflects a ceiling effect for pitch discrimination in our task. That is, for large F0 differences, listeners may have relied on changes in gross spectral cues instead of on TFS information or spectral details. As a result, there may have been an upper limit to how poor F0DLs could be in our stimulus conditions, and if listeners reached such a limit in the high-frequency GEOM condition, the ISO-GEOM ratio in the high-frequency condition may have underestimated the true impact of the GEOM masker at high frequencies. Another possibility is that the GEOM masker did not achieve its intended goal of eliminating representations of resolved harmonics in the neural response to the stimulus; this possibility is explored further via modeling described below.

Somewhat different stimuli were used in Experiment 1a and Experiment 1b. Specifically, the strong spectral edges cues present in the stimuli for Experiment 1a (which were systematically related to the F0 on each trial) were replaced with sloping spectral edges in Experiment 1b (which were not systematically related to the F0 on each trial) by using a bandpass filter. As indicated by the significant interaction between masker and experiment, this difference affected performance, with the effect of the masker being approximately 1.4 times larger on average in Experiment 1a than in Experiment 1b [estimated ratio of ratios = 1.4, F(1, 130.42) = 9.02, p = 0.016]. From visual inspection of Fig 2, the difference between Experiments 1a and 1b appears constrained to the high-frequency GEOM condition. However, the three-way interaction between F0, masker, and experiment was not significant, and a series of pairwise contrasts between Experiments 1a and 1b for each of the conditions did not reveal any significant differences after correction for multiple comparisons (all p>0.054). Collectively, the small size of the observed differences between Experiment 1a and Experiment 1b, and the fact that performance was, if anything, better in Experiment 1b than in Experiment 1a, suggests that listeners were not using spectral edge cues in Experiment 1a.

Experiment 2

It is possible that the GEOM masker may not have achieved its intended goal of eliminating rate-place representations of resolved target harmonics in the neural response. Experiment 2 attempted to address this possibility by measuring the target-to-masker ratio (TMR) that listeners required to discriminate the F0 of HCTs presented concurrently with two spectrally overlapping HCT maskers. The test tones (reference and target) had F0s that were separated by 1.5 or 2.5 times the F0DL measured for each participant individually without a masker. The masker tones had F0s that were below and above the F0 of the test tones (by between 5.25–7.25 semitones, selected randomly on each trial with a uniform distribution), and auditory-nerve simulations (see below) confirmed that the target harmonics were unlikely to be spectrally resolved at a TMR of 0 dB (equal-amplitude target and masker components). The targets and maskers were both synthesized in the same way as the tones in Experiment 1b (i.e., containing all harmonics of their F0 but bandpass filtered to attenuate all but harmonics 6–10 of the target or 5–11 of the maskers).

The results of Experiment 2 are shown in Fig 3. An ANOVA revealed significant main effects of F0 (low or high) [F(1, 10.00) = 11.78, p = 0.013] and interval size (1.5 or 2.5 F0DL) [F(1, 10.00) = 17.43, p = 0.0057] as well as a significant interaction between F0 and interval size [F(1, 10.00) = 6.77, p = 0.026]. Listeners achieved considerably lower TMRs in low-frequency conditions than in high-frequency conditions [estimated difference = −3.66 dB, F(1, 10.00) = 11.78, p = 0.019]. This frequency effect was present even though the difference in F0 was set for each listener based on their own F0DLs from the corresponding ISO condition in Experiment 1, thus nominally equating difficulty across low- and high-frequency conditions in the absence of the masker. In other words, the presence of two HCT maskers interfered more with pitch discrimination at high frequencies than at low frequencies. Under the assumption that rate-place cues for F0 were successfully eliminated by the DBL masker, this finding is qualitatively consistent with our hypothesis that F0 discrimination at high frequencies is based on a rate-place code and so should be more strongly disrupted by the reduction or elimination of spectrally resolved harmonics. This conclusion, and the assumptions underlying it, are considered in more detail in the following section.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Behavioral results from Experiment 2.

Results from Experiment 2. Large filled circles and error bars indicate the average TMR and ±1 standard error of the mean (SEM). The small filled circles and error bars indicate individual F0DLs and ±1 SEM for each participant.

https://doi.org/10.1371/journal.pcbi.1009889.g003

As expected, listeners generally performed better with the larger interval size (2.5 F0DL) than with the smaller interval size (1.5 F0DL), as confirmed by a contrast test [estimated difference = −1.52 dB, F(1, 17.43) = 17.43, p = 0.0076]. However, an unexpected interaction between F0 and interval size revealed that the larger interval size yielded better performance at low frequencies [estimated difference = −2.60 dB, F(1, 1.00) = 21.21, p = 0.0049] but not at high frequencies [estimated difference = −0.44 dB, F(1, 10.00) = 0.68, p = 0.43]. An interaction contrast test comparing the size of the interval effect at low and high frequencies was, after correction, marginally significant [estimated difference of differences = 2.16 dB, F(1, 10.00) = 6.77, p = 0.053], providing modest evidence that the size of the interval effect differed between low and high frequencies, although in both cases the trend was in the same direction, with a larger interval producing a lower TMR at threshold.

Auditory-nerve modeling highlights the complexities of HCT mixtures

In the development and interpretation of Experiments 1 and 2, we assumed that the addition of concurrent HCT maskers would reduce or eliminate rate-place cues for target components in the auditory-nerve response. Under this assumption, we expected that the HCT maskers would impact performance more at high frequencies (where we hypothesized that rate-place cues are necessary for good performance) than at low frequencies (where TFS information might still convey useful information even in the presence of the maskers). Instead, we found that the maskers had a larger negative effect at low frequencies in Experiment 1 and a larger negative effect at high frequencies in Experiment 2. Our observed pattern of data can be qualitatively explained by assuming that (1) rate-place cues were not eliminated by the GEOM masker in Experiment 1 (resulting in better-than-expected performance at high frequencies), (2) rate-place cues were eliminated (or heavily degraded) by the DBL masker in Experiment 2 (resulting in poorer than expected performance at high frequencies), and (3) temporal cues were relatively less affected by the maskers than were rate-place cues (explaining why the DBL masker had less effect at low than at high frequencies). To explore the validity of these assumptions, we simulated auditory-nerve responses to our HCT mixture stimuli from Experiments 1 and 2 and visualized the availability of different cue types for each masker type and F0.

For Experiment 1, we simulated responses to the targets in alone and in the presence of the GEOM masker for populations of high-spontaneous-rate and low-spontaneous-rate auditory-nerve fibers and visualized the average-rate responses (i.e., excitation patterns). As can be seen in Fig 4, for high-spontaneous rate fibers, resolved harmonics were only barely visible in the average-rate response for ISO stimuli at both low and high frequencies, as the sound levels were high enough to saturate most simulated high-spontaneous-rate fibers). However, for low-spontaneous-rate fibers, resolved harmonics were more clearly visible in the average-rate response to the ISO stimulus at both low and high frequencies, with responses at high frequencies in particular showing strong peaks and valleys corresponding to the frequencies of the target harmonics (this difference at low and high frequencies is consistent with sharper relative tuning in the model at high frequencies, in line with estimates from humans and other species; [50,51]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of the availability of spectrally resolved harmonics at low and high frequencies in an excitation pattern simulation.

Excitation patterns (average firing rate versus CF) for high-spontaneous-rate (HSR) auditory-nerve fibers (left) and low-spontaneous-rate (LSR) auditory-nerve fibers (right) responding to the stimuli in Experiment 1b based on a computational model of the auditory periphery [52]. The solid curve indicates the average excitation pattern while the filled area around the curve indicates ±1 standard deviation (over 10 simulations with different samples of masking noise and level roving). Vertical dashed lines indicate the frequencies of target harmonics. The F0 difference between the target and masker was 3%. See Materials and Methods for more details on the simulations.

https://doi.org/10.1371/journal.pcbi.1009889.g004

At least for the low-spontaneous-rate fibers, the GEOM masker seems to have elicited broad peaks reflecting a combination of target and masker components. These combined peaks evoked by pairs of target and masker harmonics may still be useful for coding F0, insofar as they are still systematically related to the target component frequencies. F0 discrimination based on identifying peaks in excitation patterns might worsen only to a small extent when using these blurred peaks instead of using sharper peaks associated with target components alone [45]. Regardless, the central insight from Fig 4 is that the GEOM masker likely did not achieve its intended goal of eliminating rate-place cues at high frequencies, making interpretation of the resulting behavioral results challenging. Future psychophysical studies could attempt to overcome these limitations in part by exploring other types of maskers, such as (possibly inharmonic) complex tones or filtered noise stimuli tailored to “fill in” the gaps between resolved target components in each interval. Such maskers could also potentially overcome other limitations of the present masker stimuli, such as the potential for strong beats between the harmonics of the target and the harmonics of the maskers (which were often very close in frequency for listeners that achieved good thresholds).

For Experiment 2, the key uncertainty in interpreting the results is whether the DBL masker achieved its intended goal of eliminating rate-place cues for resolved target components, and to what extent the masker impacted temporal cues at low frequencies. To probe these questions, we simulated auditory-nerve responses to the DBL stimulus as a function of TMR and visualized the availability of temporal and rate-place cues. Fig 5 shows autocorrelograms (i.e., autocorrelations of auditory-nerve responses over a range of CFs; [48]) of auditory-nerve responses (temporal coding) at both low and high frequencies while Fig 6 shows excitation patterns (rate-place coding) at both low and high frequencies for the DBL stimulus.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Comparison of simulated autocorrelograms at low and high frequencies for the DBL stimulus.

(A) Autocorrelograms for simulated high-spontaneous-rate auditory-nerve-fiber responses [52] for the low-frequency DBL stimulus (left column) and the high-frequency DBL stimulus (right column). Here, one masker F0 was set to 5.5 ST below the target F0 while the other masker F0 was set to 6 ST above the target F0. The figure was constructed by fixing the target F0 and then varying the simulated CF (y-axis) linearly over a fixed range. For each CF, the instantaneous firing rate of the auditory-nerve model was computed and then the autocorrelation of that response was computed. Color (from purple/blue [low] to yellow/green [high]) indicates the autocorrelation value at each lag-CF point, analogous to color indicating intensity in a spectrogram. In each column, the first four panels show autocorrelograms for varying TMRs and the bottom panel shows the summary autocorrelation function (sACF; the sum of the autocorrelograms along the CF axis). Red boxes and labels indicate the zoomed-in views plotted in the following subfigures. (B) A zoomed-in view of autocorrelograms and the sACF for 0, 5, and 10 dB TMR at low frequencies for lags of 3.5 to 4.5 times the period of the target F0. Vertical lines indicate lags corresponding to the periods of each tone in the stimulus or to multiples of those periods (marked using the shorthand “2F0” to refer to the lag twice as long as the period of the tone). Lags corresponding to multiples of the target period are indicated with “XF0”, while lags corresponding to multiples of the masker periods are indicated with “XF0_L” and “XF0_U” for the lower and upper maskers, respectively. (c) A zoomed-in view of autocorrelograms and the sACF for 5 and 10 dB TMR and without maskers at high frequencies for lags of 1.5 to 2.5 times the period of the target F0. Vertical lines indicate the same as in the previous subfigure. Only vertical lines corresponding to the target period are plotted in the last subpanel because this simulation was conducted without the maskers. See Materials and Methods for more details on the simulations.

https://doi.org/10.1371/journal.pcbi.1009889.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison of excitation patterns at low and high frequencies for the DBL stimulus.

Excitation patterns (average firing rate versus CF) for simulated high-spontaneous-rate and low-spontaneous-rate auditory-nerve fibers [52] responding to the low-frequency DBL stimulus (left column) and the high-frequency DBL stimulus (right column) in Experiment 2. The solid curve indicates the average excitation pattern while the filled area around the curve indicates ±1 standard deviation (over 10 simulations with different samples of masking noise and level roving). Vertical dashed lines indicate the frequencies of target components and masker components. Color of the lines indicates which F0 they corresponding to (orange for the lower masker F0, gold for the target F0, and pink for the upper masker F0). See Materials and Methods for more details on the simulations.

https://doi.org/10.1371/journal.pcbi.1009889.g006

In Fig 5, strong evidence for the target F0 consists of vertical ridges at lags corresponding to the F0 (integers on the x-axis) spanning along the range of CFs (y-axis). As can be seen, temporal coding of the target F0 was weak at 0 dB TMR at both low and high frequencies. At low frequencies and at 0 dB TMR, vertical ridges in the autocorrelogram often reflected synchronization to masker components instead of target components (Fig 5B). At more favorable TMRs, vertical ridges corresponding clearly to the target period emerged and the most prominent peaks in the summary autocorrelation (sACF) were located at integer multiples of the target period. By 10 dB TMR, the low-frequency autocorrelogram closely resembled the autocorrelogram for the stimulus without maskers. Coding of the target F0 was much weaker at high frequencies, consistent with the loss of phase locking to the TFS of individual components at high frequencies. Even at 10 dB TMR, temporal coding of the target F0 was considerably less salient than in the absence of the maskers. Fibers tuned near stimulus components conveyed little temporal information about F0 (because they are saturated and do not phase lock to the high-frequency TFS; Fig 1). Fibers tuned between stimulus components still conveyed some temporal information (because they respond to modulations at F0 elicited by beating between adjacent target components in the stimulus; Fig 1), but the resulting autocorrelation peaks were broad and shallow (Fig 5C).

In Fig 6, strong evidence for the target F0 would consist of prominent peaks at each target harmonic (integers on the x-axis). High-spontaneous-rate fibers showed few prominent peaks at any TMRs at either low or high frequencies because the sound levels used in the simulations saturated the high-spontaneous-rate fibers. In contrast, low-spontaneous-rate fibers showed many prominent peaks in the excitation patterns. At 0 dB TMR, these peaks corresponded to masker components or mixtures of target and masker components. At more favorable TMRs, these peaks more clearly reflected target frequencies. However, even with masker components attenuated by 10 dB (i.e., 10 dB TMR), the peaks elicited by target components still had smaller peak-to-valley ratios than corresponding peaks in the excitation pattern without maskers (bottom row), masker components outside the stimulus passband still elicited strong peaks, and peaks near masker components within the stimulus passband still influenced the shape of peaks near target components. This simulation demonstrates that the notion of “resolved harmonics” in the context of mixtures of HCTs, where auditory neurons are responding to mixtures of components in complicated ways, is problematic and lacks a clear definition [44].

Figs 5 and 6 emphasize the complexities inherent in interpreting the results of Experiment 2. Both temporal and rate-place cues were likely strongly impacted by the presence of the DBL masker. If we assume that the same mechanism is used at low and high frequencies to decode the target F0, then predictions are relatively clear. If a temporal code is used at low and high frequencies, thresholds with the DBL masker ought to be better at low frequencies than at high frequencies, because temporal cues at low frequencies are much more robust to the presence of the DBL masker than are temporal cues at high frequencies (Fig 5). If instead a rate-place code is used at low and high frequencies, thresholds with the DBL masker ought to be better at high frequencies than at low frequencies, because sharper relative tuning at high frequencies leads to clearer rate-place cues for F0 at high frequencies than at low frequencies (Fig 6). In our behavioral data, we observed that performance was better at low frequencies than at high frequencies with the DBL masker (Fig 3), a pattern of results that is nominally consistent with the temporal-code prediction. However, if different mechanisms are used at low and high frequencies to decode target F0, then predictions are less clear because comparing the relative fidelity of temporal and rate-place cues is non-trivial.

Predicting performance with an ideal-observer model

The behavioral experiments provide novel data on how concurrent and spectrally overlapping HCT maskers affect pitch discrimination at both low and high frequencies. However, the results are equivocal in their support of the hypothesis that pitch discrimination is based on a rate-place code at high frequencies. In Experiment 1, adding a single HCT masker impaired pitch discrimination more at low frequencies than at high frequencies (contrary to our hypothesis), whereas in Experiment 2, adding a double HCT masker impaired pitch discrimination more at high frequencies than at low frequencies (consistent with our hypothesis). Moreover, the simulations illustrated in Figs 4–6 emphasize that adding HCT maskers to an HCT target is likely to affect both temporal and rate-place cues for F0 in complicated ways, making it difficult to interpret the effect of masker in terms of either set of cues.

An alternative approach to probing the neural underpinnings of high-frequency F0 discrimination is to focus on the difference in performance at low and high frequencies for isolated HCTs (which evoke less complicated responses than HCT mixtures) and attempt to determine whether the magnitude of that difference is consistent with the predictions of either temporal or rate-place models of pitch. To this end, we employed three existing computational models of the human auditory periphery [27,52,55] to predict FDLs for pure tones and F0DLs for the isolated HCTs from Experiment 1a using ideal-observer analysis. Simulations of pure-tone FDLs were included to provide direct comparisons with prior work using similar methods [27,28,56]. Thresholds were based on simulated responses from high-spontaneous-rate fibers at 40 CFs spanning a 1.5-octave range around the target frequency (FDLs) or 40 CFs spanning a ~1.1-octave range around the stimulus passband (F0DLs; see Materials and Methods for details). Thresholds were generated for two observer types: an all-information observer and a rate-place observer. Whereas the all-information observer utilized all information in the simulated auditory-nerve response optimally, the rate-place observer averaged the simulated auditory-nerve response over time and then utilized that time-averaged information optimally (see Materials and Methods for details). These two observers allowed us to establish an optimal bound on performance in the modeled tasks (all-information observer) and to determine how much predicted performance was affected by discarding all temporal information from the simulated auditory-nerve responses (rate-place observer).

Frequency discrimination.

Ideal-observer predictions for pure-tone frequency discrimination are shown in Fig 7. The predicted FDLs generally align well with those from previous work using similar methods [27,28,56]. All-information FDLs are best near 2 kHz and then steadily degrade as the tone frequency increases (Fig 7A) due to degraded coding of TFS. However, the rate at which performance degraded with increasing frequency varied between the underlying nerve models (Fig 7C). For example, between 2 and 8.5 kHz, the Verhulst et al. [55] model predicted a factor of 5 decrease in performance, whereas the Zilany et al. [52] model predicted a factor of 25 decrease in performance, and the Heinz et al. [27] model predicted a factor of 17 decrease in performance. The onset and slope of the degradation in performance with increasing frequency in each model was directly related to the cutoff frequency and slope of phase-locking roll-off in each model’s nerve fibers (Fig 7B and S1 Text). Higher levels generally resulted in lower (better) thresholds for the all-information model. As demonstrated quantitatively over a wider range of levels in prior work [27], this trend is consistent with trends in behavioral data for frequency discrimination in quiet [57].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Ideal-observer predictions for pure-tone frequency discrimination.

Results of the frequency discrimination simulations. (A) Simulated FDLs versus frequency for a pure tone in each auditory-nerve model. Simulations in this panel include no parameter roving. Points indicate the simulated FDLs at a particular frequency while lines indicate a locally estimated scatterplot smoothing (LOESS) fit to the simulated FDLs. The solid black line indicates the predicted FDLs from Micheyl et al. [58] scaled by a factor of 0.002 (to roughly match the low-frequency side of the curve to the best-performing model predictions from the present study). The axis on the right-hand side corresponds to the unscaled FDLs predictions from Micheyl et al. [58]. (B) Simulated all-information FDLs and vector strength (top row) and simulated rate-place FDLs and Q10 (bottom row) versus frequency with a double y-axis. To choose the warping on the y-axis for vector strength and Q10, linear models were fit to predict log-transformed FDLs as a function of log-transformed reciprocals of vector strength (Q10) for the all-information FDLs (rate-place FDLs). The fitted regression equations were then used to warp the y-axes. In other words, we warped the y-axes for vector strength and Q10 to maximize overlap with the FDL predictions (across all three models) in order to visually demonstrate the relationship between vector strength and Q10 and the simulated FDLs. (C) Ratio of simulated FDLs at 8.5 kHz and 2.0 kHz in the non-roved simulation at 30 dB re: threshold for each model (left) and ratio of behavioral estimates of FDLs at 8.5 kHz and 2.0 kHz from various studies (right). Simulated FDLs were interpolated using LOESS while behavioral FDLs were linearly interpolated on log-log coordinates.

https://doi.org/10.1371/journal.pcbi.1009889.g007

In contrast to the all-information observer, the rate-place observer predicted thresholds that were poorer overall and were relatively flat, or improved slightly with frequency, as expected based on sharper tuning at high frequencies (Fig 7A and 7B and S1 Text). Changes in level over the tested range appeared to have minimal impact on the rate-place observer (except at the highest frequencies in the Zilany et al. [52] model, where increased levels resulted in somewhat poorer performance). At much higher levels, rate-place thresholds degraded significantly (S3 Text). The poorer performance of the rate-place observer at high levels is not consistent with behavioral data for pure-tone frequency discrimination in quiet, which shows improved performance at high levels [57]. Consistent with expectations, performance of the rate-place observer was correlated with the sharpness of tuning in the underlying auditory-nerve model, with sharper tuning resulting in better FDLs (Fig 7B). The Zilany et al. [52] and Verhulst et al. [55] models had slightly better rate-place thresholds than the Heinz et al. [27] model, consistent with these models having sharper peripheral tuning, particularly at high frequencies.

Consistent with behavioral data [59] and the simulations of Heinz et al. [27], level roving (i.e., randomizing the level from stimulus to stimulus) had little impact on frequency discrimination for either the all-information or rate-place observers (S2 Text). In contrast, phase randomization (i.e., randomizing the starting phase from stimulus to stimulus) appeared to impair the all-information observer by a small but consistent amount (approximately a factor of two) at least up to approximately 10 kHz (S2 Text). This increase in thresholds by a factor of two closely matches the theoretical predictions made by Siebert [28] but is not observed in humans, who are insensitive to the starting phase of pure tones or the phase relationships between spectrally resolved components [60].

We compared the model results to existing behavioral data, using a function derived from multiple studies of pure-tone frequency discrimination [58]; this function’s FDLs were always larger (poorer) than the predictions of both the rate-place and all-information observers, suggesting that both models contained sufficient information to explain human behavior. However, the dependence of FDLs on frequency was better captured by the all-information model, as shown by the black curves in Fig 7A, which were scaled down by nearly three orders of magnitude in order to more closely match the absolute values of the all-information observer. In addition, we extracted FDLs from four individual behavioral studies that tested a sufficiently wide range of frequencies [21,22,36,38] and then estimated the FDL ratio between 8.5 kHz and 2 kHz for each study (Fig 7C; see Materials and Methods for details). Using the ratio of two FDLs emphasizes the effect of frequency in each dataset rather than absolute differences in FDLs between datasets or between behavioral data and the ideal-observer thresholds.

Both the behavioral data and the computational results showed a wide range in ratios of performance between 8.5 kHz and 2 kHz. Some insights can nevertheless be gained. Notably, with the exception of the Moore and Ernst [21] results, the ratios from the behavioral studies are essentially bound below by the all-information observer predictions from the Verhulst et al. [55] model (which has the smallest ratio between 8 kHz and 2.5 kHz and the slowest rolloff of phase locking) and above by the Zilany et al. [52] model (which has the largest ratio between 8 kHz and 2.5 kHz and the fastest rolloff of phase locking). In other words, despite variations in both the behavioral results and model predictions, the degradation in behavioral performance with increasing frequency is in relatively good quantitative agreement with trends observed in optimal frequency discrimination.

F0 discrimination

Ideal-observer predictions for F0DLs with harmonics 6–10 are shown in Fig 8. The all-information observer predicted that F0 discrimination should be best for F0s in the range of 200–500 Hz (with the precise range depending on the underlying auditory-nerve model) and then steadily worsen with increasing F0 before plateauing around 600–1000 Hz (Fig 8A), at least in the Heinz et al. [27] and Zilany et al. [52] models. For the present stimuli, the 6th harmonic was the lowest present, so that an F0 of 600 Hz corresponded to a lowest harmonic of 3600 Hz, whereas an F0 of 1000 Hz corresponded to a lowest harmonic of 6000 Hz. The model fibers do not phase lock strongly to frequency components above about 4 kHz (S1 Text); thus, this pattern of results is suggestive of a transition from more accurate TFS-based coding at lower F0s to less accurate envelope-based coding at higher F0s in the all-information observer (see no-masker simulations in Fig 5C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Ideal-observer predictions for F0 discrimination.

Results of the F0 discrimination simulations. (A) Simulated F0DLs versus F0 of the ISO HCT stimulus in each auditory-nerve model. Simulations in this panel include no parameter roving. Points indicate the simulated F0DLs at a particular F0 while lines indicate a locally estimated scatterplot smoothing (LOESS) fit to the simulated F0DLs. (B) Simulated all-information F0DLs and vector strength (top row) and simulated rate-place F0DLs and Q10 (bottom row) versus frequency with a double y-axis. To choose the warping on the y-axis for vector strength and Q10, linear models were fit to predict log-transformed F0DLs as a function of log-transformed reciprocals of vector strength or Q10 for the all-information F0DLs or rate-place F0DLs, respectively. The fitted regression equations were then used to warp the y-axes. In other words, we warped the y-axes for vector strength and Q10 to maximize overlap with the model predictions (across all three models) in order to visually demonstrate the relationship between vector strength and Q10 and the simulated F0DLs (C) Ratio of simulated F0DLs at 1.4 kHz and 0.28 kHz in the non-roved simulation at 30 dB re: threshold for each model (left) and ratio of behavioral estimates of F0DLs at 1.4 kHz and 0.28 kHz from various studies (right). Simulated F0DLs were interpolated using LOESS while behavioral F0DLs were linearly interpolated on log-log coordinates.

https://doi.org/10.1371/journal.pcbi.1009889.g008

As expected, and consistent with the results for frequency discrimination, the rate-place observer thresholds were higher than the all-information observer thresholds. Nevertheless, as with the pure-tone FDL predictions, the rate-place thresholds were still better than the average behavioral thresholds, suggesting that, in principle, sufficient rate-place information is available at the level of the auditory-nerve to account for the accuracy of behavioral F0 discrimination at both low and high frequencies. Performance for the rate-place observer was generally flat with frequency or slightly improved with higher frequencies, depending on the exact model in question. These inter-model differences are a direct result of differences in peripheral tuning between the models (Fig 8B and S1 Text). The rate-place observer tended to be more sensitive to changes in level than the all-information observer (Fig 8A). Generally, rate-place thresholds were poorer at higher levels, consistent with the fact that higher levels degrade representations of stimulus harmonics in the average rate responses of auditory-nerve fibers [61]. At much higher levels, rate-place thresholds worsened significantly, consistent with saturation of the simulated high-spontaneous-rate fibers at high levels (S3 Text). There is some evidence that F0 discrimination (in noise) worsens with increasing level [62]; however, this effect was observed at higher levels behaviorally than those tested in the present model simulations.

Both the all-information observer and the rate-place observer models were affected by parameter roving. Level roving typically had a negative impact on the rate-place observer at higher frequencies but little impact on the all-information observer (S2 Text). In comparison, phase randomization (i.e., randomizing the relative phases of the components from stimulus to stimulus instead of always synthesizing components in sine phase) had no impact on the rate-place observer but had a negative impact on the all-information observer (S2 Text). Generally, across a fairly wide range of levels and frequencies, phase randomization elevated all-information thresholds by a factor of two, consistent with the predictions of Siebert [28] and the frequency discrimination results. In behavioral data, phase randomization only affects thresholds when all the stimulus components present are unresolved, as expected [31,32].

To compare the model results to behavioral data, we extracted discrimination thresholds from three behavioral studies that tested the same F0s as in our study [36–38] and then calculated the ratio between F0 discrimination thresholds for F0s of 1.4 kHz and 0.28 kHz for each study as well as the present study (Fig 8C; for the present study, data was pooled across Experiment 1a and Experiment 1b, see Materials and Methods for details). In contrast to the varying stimuli and methods of the selected pure-tone frequency discrimination studies, the selected F0 discrimination studies all used essentially the same stimuli and methods. The ratios between F0 discrimination thresholds at 1.4 kHz and 0.28 kHz for each study ranged from approximately 3 [36] to approximately 5 [37]. These values are within a range that can be plausibly explained by the degradation of the all-information observer with increasing frequency, with the same ratio for the all-information observer ranging from approximately 3 for the Verhulst et al. [55] model to over 10 for the Heinz et al. [27] model. In contrast, the rate-place observer model again predicted threshold ratios of around 1, which is outside the range observed in behavioral data.

As was the case for frequency discrimination, the all-information observer provided the best overall match to human data in terms of relative performance at low and high frequencies. The plateau in the predicted F0 discrimination thresholds of the all-information observer above about 600 Hz in the Heinz et al. [27] and Zilany et al. [52] models (in contrast to the progressive degradation of predicted pure-tone frequency discrimination thresholds at high frequencies; Figs 7 and 8) suggests that, at higher F0s once phase locking to TFS becomes unavailable, the all-information observer relies on temporal cues produced by peripheral interaction between adjacent harmonics rather than phase locking to TFS at the component frequencies (Fig 1B).

The correspondence between all-information trends and human behavior could be interpreted as support for the use of temporal-envelope cues at high F0s by humans. However, as discussed earlier, several lines of evidence from pitch psychophysics pose challenges to this interpretation. Specifically in the case of the high-F0 HCTs with harmonics in the range of the 6^th to the 10^th, pitch perception is robust to manipulations in which the even and odd harmonics are presented to different ears (which would decrease modulation depth and double the modulation rate in each ear [38]) but is impaired by harmonicity manipulations in which the frequencies all components are shifted up by a constant amount in Hz (which preserves modulation rates while rendering the tone inharmonic [38,39]). Although beyond the scope of this paper, auditory-nerve simulations and ideal-observer analysis could help further clarify our understanding of these conditions. More generally, for HCTs composed of unresolved harmonics and amplitude-modulated noises (for which pitch is thought to be exclusively conveyed by temporal-envelope cues), humans are not able to achieve accurate pitch discrimination for F0s (or modulation rates) beyond about 600–700 Hz [40,42]. These lines of evidence seem to rule out a class of simple models wherein listeners compare temporal-envelope rates to discriminate F0 in the present task. However, there are several notable differences between the present HCT stimuli and the stimuli used to probe temporal-envelope pitch (unresolved HCTs and modulated noises) that are worth considering further. First, whereas for unresolved HCTs and modulated noises the only reliable pitch cue is the temporal-envelope rate, the present HCT stimuli consist of partially resolved harmonics, which provide rate-place cues in addition to any temporal cues elicited by peripheral interaction between stimulus components. Second, whereas for unresolved HCTs and modulated noises, all auditory-nerve fibers tuned to the stimulus should have strongly modulated responses, in the present HCT stimuli, modulation power varies substantially over the range of CFs, with deeper modulations in channels tuned between stimulus components (Figs 1 and 5). As a result, for unresolved HCTs or modulated noises, the auditory system is relegated to comparing temporal-envelope modulation rates to perform discrimination. In contrast, for the present HCT stimuli, the auditory system could derive F0 estimates by combining information provided by average ANF rates (the excitation pattern) with information provided by the distribution of envelope modulation power across CFs (the so-called “fluctuation profile”; [63]). Recent modeling studies suggest that such a strategy, based on decoding of simulated fluctuation profiles at the level of the midbrain, can in principle account for behavioral F0 discrimination in stimuli similar to the present low-frequency HCT stimuli [64] as well as performance in other psychoacoustical tasks [65]. At the same time, these prior modeling studies have focused on midbrain neurons tuned to modulations on the order of around 100 Hz, while stimulus-envelope modulations in the present high-frequency HCT stimuli are at much higher rates (> 1 kHz) where sensitivity to amplitude modulations is thought to be limited [66–68]. Further investigation will be needed to determine whether the fluctuation-profile approach can be successfully extended to our high-frequency stimuli and reconcile the trends in the ideal observer with what is known from pitch psychophysics.

Effect of maskers

In theory, the ideal-observer model could be applied to the Experiment 1 GEOM stimulus and the Experiment 2 DBL stimulus to investigate how HCT maskers affect F0 discrimination. However, this was not done for the present analyses for reasons discussed in Supporting Information (S4 Text). The primary issue was that preliminary attempts at modeling these conditions usually resulted in the ideal observer performing better in the presence than in the absence of a masker (Fig A in S4 Text). Although initially surprising, further consideration of the nature of the ideal observer makes it clear why this could happen. As shown previously [27], the lower bound on the precision of estimating a stimulus parameter in our model, as specified by the Cramér-Rao lower bound, is equivalent to the performance achieved by a maximum likelihood estimator (at least for the one-parameter case). The maximum likelihood estimator examines each spike in the auditory-nerve response and determines whether it is more probable that the spike came from a baseline interval (f) or an incremented interval (i.e., f + Δf) and then combines those probabilities across all observed spikes. Intuitively, this means that the observer has perfect templates of what the baseline and incremented responses should look like and compares the response against those templates according to an optimal rule. In the case of stimuli containing maskers, this means that the ideal observer can exploit any feature of the responses in decoding. As observed previously for noise maskers [69], interactions between targets and maskers that would never be usable by human observers (because they are completely unpredictable from trial to trial) can be exploited by the ideal observer, leading to unrealistic thresholds. To resolve these limitations of the present ideal-observer model, future work could explore the application of quasi-ideal observers [69,70], deep neural network models [71], or other modeling frameworks to our stimuli to better understand how F0 cues may be more realistically decoded in HCT-mixture stimuli.

Ethics statement

The behavioral data collected for this study was collected according to protocols approved by the University of Minnesota Institutional Review Board. Informed consent was obtained from participants in written form prior to initiation of the experimental protocol.

Behavioral data collection

Participants

Participants in the present experiments were all students at the University of Minnesota with audiologically normal hearing. Of the 36 participants that passed the screening procedures described below (29 female, 7 male), the mean age was 22.8 years (SD = 1.9, minimum = 20, maximum = 28). Participants were recruited either through a Department of Psychology research participant pool or through an in-house participant database. Participants were paid for their participation.

Procedures

All experiments began with a three-stage screening process. The first stage consisted of a standard audiogram to ensure normal-hearing status (i.e., audiometric thresholds ≤ 20 dB hearing level [HL] at octave frequencies from 0.25 to 8 kHz). The second stage consisted of measuring detection thresholds for pure tones embedded in the same broadband TEN used in the main experiments. The purpose of this stage was to ensure that participants would be able to hear all the target stimuli in the presence of the masking noise during the experiments. Detection thresholds were measured using a two-interval two-alternative forced choice (2I2AFC) method combined with a 3-down 1-up staircase procedure [75]. Both intervals contained the TEN and one interval, chosen at random on each trial, contained the target tone. The participants’ task was to select the interval containing the tone. The tones were 350 ms in duration, including 20-ms raised-cosine onset and offset ramps. The noise was 500 ms in duration, including with 20-ms raised-cosine onset and offset ramps, and the tones were temporally centered in the noise. The noise had a level of 40 dB SPL in the estimated ERB of the human auditory filter centered on 1 kHz. The adapted variable was the rms level of the target in dB SPL. The starting value of target level was 55 dB SPL. The initial step size of the adaptive procedure was 3 dB. After two reversals in the direction of the adaptive procedure, the step size was reduced to 2 dB. After another two reversals this was reduced to 1 dB (the smallest step size used). The procedure was terminated after 6 reversals at the smallest step size and the threshold was defined as the mean of the target levels at the last six reversals. Tone frequencies of both 14 kHz and 16 kHz were tested. In order to pass this stage, participants needed thresholds better than 45 dB SPL at both frequencies, averaged across three runs per frequency. This value was 2 dB lower than the lowest possible presentation level of individual harmonics in the experimental stimuli, and so it was thought that participants who could pass this criterion should be able to detect the stimuli during the experiment. Participants who failed this stage did not complete any other part of the experiment. Of the 79 participants who attempted the audibility screening, 44 passed (pass rate = 56%) and were invited to participate in following stages of screening.

The third stage consisted of measuring F0DLs for the two nominal F0s tested in the experiments described below (280 Hz and 1400 Hz). The purpose of this stage was to ensure that participants had the ability to accurately label the direction of a change in F0. The F0DLs were measured using a 1I2AFC task and a 3-down 1-up adaptive staircase procedure. In each trial, three isolated HCTs with the same F0 (the reference F0) were presented in sequence as a precursor, followed by a target HCT with a different F0 (the test F0) with a higher or lower F0 than the precursor tones (selected at random on each trial with equal probability). All the precursor and target tones were separated from each other by 50-ms silent intervals. The listener’s task was to judge whether the target F0 was higher or lower than the reference F0. Visual feedback was provided immediately after each trial. The starting value of DF0 (the difference between the reference and target F0s) was 10% of the lower F0. The initial step size was a change in DF0 by factor of 2. After two reversals this was adjusted to a factor of 1.41. After another two reversals this was adjusted to a factor of 1.19 (the smallest step size used). The procedure was terminated after 6 reversals at the smallest step size and the threshold was defined as the geometric mean of the DF0 at the last six reversals. The maximum permitted DFO was approximately 30%; if a larger DF0 was called for by the adaptive procedure on more than 6 consecutive trials then the run was terminated early and no threshold was recorded. The reference and target F0s were geometrically centered around an F0 selected randomly on each trial from a uniform distribution of either 280 ±10% Hz (low-frequency condition) or 1400 ±10% Hz (high-frequency condition). Each tone was 350 ms in duration, including 20-ms raised-cosine onset and offset ramps. The tones were composed of harmonics 6–10, presented in random phase (Experiment 1a and Experiment 2) or presented with all harmonics up to the Nyquist frequency in sine phase and then zero-phase bandpass filtered with a 12^th-order Butterworth filter with cutoffs at 5.5 to 10.5 times the nominal F0 (Experiment 1b). The level of each harmonic was 50 dB SPL. In order to pass this stage, participants were required to obtain thresholds better than 6% and 12% for the low- and high-F0 conditions, respectively, averaged across three runs per nominal F0. Participants who failed this stage were given two additional chances to pass the screening under an identical procedure. Participants who passed on any attempt moved on to complete the experiment while participants who failed all attempts did not complete any other part of the experiment. Of the 44 participants who attempted the pitch screening, 36 passed (pass rate = 82%) and went on to complete one or more experiments.

Behavioral data analysis

Computational model

We used three previously published auditory-nerve models to conduct our simulations of auditory-nerve firing rates. The use of multiple nerve models affirmed that the key trends were not artifacts of assumptions or features that were specific to a particular model. First, we used the auditory-nerve model described in Heinz et al. [27]. Although simpler and less accurate than the other nerve models we tested, this model had the advantage of being much faster than the other models and it allowed for a more direct comparison between the present results and previous auditory-nerve ideal-observer results based on this model [27,56]. This model was implemented in Python and run at a sampling rate of 1250 kHz (higher than the other models to reduce distortion at high stimulus frequencies and levels). Second, we used the auditory-nerve model of Zilany et al. [52]. The model was run in an adapted version of the Python interface to the original C implementation of the model provided by the cochlea package [82]. The model was run with parameters designed to replicate the human peripheral tuning estimates of Shera et al. [51] at a sampling rate of 200 kHz (for the basilar-membrane and inner-hair-cell stages) and 50 kHz (for the synapse stage). Finally, we used the auditory-nerve model of Verhulst et al. [55]. The model was run via the authors’ published Python code at a sampling rate of 300 kHz.