2.1. fMRI BOLD responses to snakes are larger than to gratings

We first consider an observation about fMRI responses to two classes of simple, grayscale, band-passed, static images. For one class, the stimuli contain several curved contours, which we refer to throughout as snakes. For the other class, the stimuli contain several straight, parallel contours, which we refer to as gratings. We refer to these classes together as the target stimuli. The surprising observation is that for V1, V2 and V3, the fMRI responses are substantially larger for the snakes than for gratings (Fig 2). The responses to the gratings, irrespective of density, are only about as high as the lowest response to the snakes. We confirm this pattern with three additional fMRI data sets, which also show larger responses to snakes than gratings in V1, V2 and V3 (Figs A1-A4 in S5 Appendix).

To check whether this pattern of results is specific to fMRI, we replot published intracranial data from Hermes et al., 2019 [13], in which human subjects with ECoG recordings viewed a similar set of stimuli (Fig 2, bottom panel). The ECoG data, plotted as the percent increase in broadband power over baseline, show the same general effect as the fMRI data: The responses to snakes are much larger than to gratings, irrespective of texture density, indicating that the effect is not limited to the fMRI BOLD response. (See Hermes et al., 2019 [13] for methodological details.)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. FMRI responses are larger for curved patterns than for straight patterns.

The mean fMRI responses within a visual area are plotted for V1, V2, and V3 in Data set 2 of Table A in S1 Appendix. For both curved stimuli (snakes, dark bars) and straight stimuli (gratings, light bars), data are plotted in order of increasing texture density from left to right. Examples of the 5 densities are shown above. Error bars are the standard deviation of the mean, bootstrapped across fMRI runs. Responses are larger for the snakes than gratings. The same effect is also observed for Data sets 1, 3 and 4 of Table A in S1 Appendix (Fig A in S5 Appendix). Bottom panel: For comparison, we replot data from intracranial recordings (ECoG) from Hermes et al 2019 [13], which also show larger responses for snakes than for gratings. The full set of stimuli is shown in S2 Appendix. Data plotted from the function s4_visualize(’figure 2’) in the GitHub code repository.

https://doi.org/10.1371/journal.pcbi.1011704.g002

In the next four sections, we describe the four models that are fit to the data. For tractability, we fit each model’s parameters to the aggregate (average) data within a visual area, rather than to each voxel individually. All the stimuli are texture-like, meaning that they have similar properties across the whole image aperture, and for each stimulus class, nine different exemplars were shown per 3-s trial. The different exemplars have the same higher-order statistics but vary in their precise spatial distributions. Hence, model variables, such as contrast energy, would have similar values for spatially localized portions of the image (as one would compute for an individual voxel) as for the whole image (as we compute to model the aggregate response of a visual area). For this reason, we did not include model parameters for the spatial location or spatial extent of the receptive fields for individual voxels. We first describe models fit to the target stimuli. We then summarize fits to the larger set of stimuli viewed by the subjects.

All four models consist of three components, filtering, spatial pooling, and a power-law nonlinearity. In the filtering stage, we computed the contrast energy of the stimuli at 8 orientations and 4 spatial frequencies (see Methods for details). Although stimuli were designed to have power concentrated close to 3 cycles per degree, there is some spillover to lower and higher frequencies, which is why we use multiple spatial frequency bands in our models. The spatial pooling stage pools the contrast energy to yield a total contrast energy. In some models, the pooling stage also includes divisive normalization. The output of the pooling is a scalar, which is then passed through a power-law nonlinearity to predict BOLD amplitude in units of percent signal change. The power-law nonlinearity achieves compressive spatial summation [10]. All models have the same filtering (first stage) and output non-linearity (third stage). They differ in the spatial pooling stage.

2.2. The larger response to snakes is not captured by a simple contrast energy model

A standard contrast energy model pools the contrast energy by simply summing it across space, orientations, and spatial frequencies to give a total contrast energy. It predicts that responses should increase with both stimulus contrast and with density of the pattern. This model does not predict a larger response to the snakes than to gratings, contrary to the data (Fig 3). In fact, the cross-validated variance explained is low (V1, V2) or even negative (V3) in the example data, meaning that the model prediction is less accurate than it would have been if it simply predicted the mean response across all stimuli. (The data are cross-validated, which is why the variance explained can be negative). In short, the contrast energy model provides a poor fit to the fMRI data in V1-V3 for these classes of stimuli. It is also a poor fit to the target stimuli in the other three data sets (Table A in S3 Appendix and Fig A1 in S5 Appendix). This does not mean that contrast energy models are always poor fits to fMRI responses in V1-V3. For example, when stimuli vary in how contrast energy is distributed across space, a contrast energy model can capture a lot of the variance in the responses across images, as shown by Kay et al., 2013 [9].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The contrast energy model does not account for V1-V3 responses to snakes and gratings.

(A) Schematic representation of the contrast energy model. For simplicity, we show E summed across spatial frequency bands. (B) Mean fMRI responses in V1, V2, and V3 to snake and grating stimuli that vary in density and contrast (Data set 2). Bars: mean and standard error of the responses. Dark bars represent snake stimuli and light bars represent grating stimuli. Each group of stimuli is arranged in increasing order of either density or contrast. Dots: cross-validated predictions from the contrast energy model. See Fig A1 in S5 Appendix for fits to all 4 data sets. See S4 Appendix for model parameters. Data and model fits plotted using the function s4_visualize(’figure 3’) in the code repository.

https://doi.org/10.1371/journal.pcbi.1011704.g003

2.3. The larger response to snakes is not captured by an untuned normalization model

We then add divisive normalization to the model. After computing contrast energy, we normalize the outputs by dividing the contrast energy at each pixel by the contrast energy of a normalization pool (Fig 4, upper panel). The normalization pool includes nearby locations, all spatial frequencies, and all orientations, giving it the name untuned normalization model (Fig 4, lower left panel).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Two divisive normalization models.

(A) Schematic of divisive normalization models. (B) The weights used to calculate normalization for the untuned (left) and tuned (right) models, plotted using the function s4_visualize(’figure 4’) in the code repository.

https://doi.org/10.1371/journal.pcbi.1011704.g004

The untuned normalization model and the contrast energy model make similar predictions and explain a similar proportion of the variance in the data. The normalization model results in more saturation at high contrast, as expected from divisive normalization [7]. This is especially evident in V2 and V3 at the highest stimulus contrast. The reason that the normalization model and the contrast energy model have a similar overall pattern of predictions is that the power law output nonlinearity, included in all models, can partially mimic the effects of normalization [10, section “Relationship to Divisive Normalization”].

Like the contrast energy model, the untuned normalization model predicts a similar BOLD amplitude for snakes and gratings, thereby failing to account for the data (Fig 5, upper panel). The model is not sensitive to heterogeneity across orientations because the normalization pool equally weights all orientation channels. Therefore, the output does not depend on whether the contrast energy is concentrated in one orientation channel, as in the gratings, or spread across many channels, as in the snakes. The untuned normalization model’s failure to account for the greater response to snakes is reflected by low variance explained for the target stimuli in each of the four data sets (Table A in S3 Appendix and Fig A2 in S5 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The tuned normalization model is much more accurate than the untuned normalization model.

The mean fMRI responses from V1, V2, and V3 are replotted from Fig 3. The dots are the cross-validated predictions from the untuned normalization model (above) or the tuned normalization model (below). Data and model fits plotted from the function s4_visualize(’figure 5A’) and s4_visualize(’figure 5B’) in the code repository. See Figs A2-A3 in S5 Appendix for fits to all 4 data sets.

https://doi.org/10.1371/journal.pcbi.1011704.g005

2.4. The larger response to snakes is captured by an orientation-tuned normalization model

The untuned normalization model implements a surround suppression that includes energy at all orientations. Findings from electrophysiology [35–39], psychophysics [38,40–42], neuroimaging [26,27,43–45], and theory [46], however, suggest that surround suppression is orientation-tuned. For example, Cavanaugh, Bair, and Movshon, 2002 [35] reported that the response of a neuron to a stimulus at its preferred orientation in its receptive field is suppressed more when the surrounding region contains contrast at the same orientation compared to different orientations. Because our grating stimuli have contrast energy concentrated at a single orientation, and the snake stimuli do not, one might surmise that an orientation-tuned normalization model would show greater suppression for the gratings, where the RF centers and surrounds will have matched orientations, than for the snakes, where the orientations are more likely to differ between center and surround. If so, this could then account for our observed effect.

The untuned normalization model is the same as the tuned model except for one difference: At each pixel in the oriented contrast energy images, the tuned model normalizes the contrast energy across nearby locations only at the preferred orientation (orientation-tuned surround) (Fig 4, lower right panel, diagonals). Within a single location (i.e., at each pixel), the normalization is untuned (off-diagonals in the same panel), also called cross-orientation suppression [47].

The orientation-tuned normalization model captures the large and systematic differences in response amplitude between gratings and snakes (Fig 5, lower panel). In the example data set, the BOLD amplitude and the orientation-tuned normalization model predictions for the gratings are about half of those to the snakes, for both the density and contrast manipulations, and for all three visual areas. In addition to capturing this difference in the means between the two stimulus classes, the model also captures the difference in slope. As the density or contrast increases, the model predicts steeper slopes for snakes than gratings. These patterns in the model fits are found across the four data sets (Fig A3 in S5 Appendix). The orientation-tuned normalization model is more accurate than the previous two models in all cases (Table A in S3 Appendix, four data sets and three ROIs). This result holds up for different size surrounds–what mattered was whether the surround was tuned or not tuned, not its size. The indifference to the size of the surround almost certainly reflects the properties of the stimulus set, not neural tuning: the stimuli are all textures, with similar properties across the image. Had the stimuli varied systematically across location, the size of the surround would likely have had a large effect on model accuracy. Note that the data sets sometimes show a negative slope with increasing density (e.g., V3, gratings varying in density). The model is unable to capture this effect. The study in which these sparse stimuli were first used [11] showed that a second-order contrast model could account for the decreased response with increasing sparsity, as the sparser stimuli have more second-order contrast. It is possible that the orientation-tuned normalization model would also be able to do so if it included spatial receptive fields per voxel that were small relative to the image.

2.5. Normalization by orientation anisotropy

The large advantage in prediction accuracy of the tuned over the untuned normalization model supports the idea that suppression is feature-specific. As with any model, we chose a specific instantiation of a more general idea, namely feature-specific suppression. The specific instantiation entailed a minimal change from the untuned normalization model, requiring only a change in normalization weights, and builds on the tradition of feedforward, filter-based models. Feature-specific tuning can also be implemented in other ways, for example based on more abstract ideas like predictability or redundancy in the image. There is some evidence for models like these [22,48]. We implemented a second method of achieving orientation-tuned normalization, in which normalization was proportional to orientation anisotropy (Fig 6, Normalization by orientation anisotropy, “NOA”). Normalization in this model is most pronounced when an image patch has a single orientation, without a specification in terms of a match between center and surround (See 3.3 What is the tuning in orientation-tuned normalization?). Specifically, in the normalization by orientation anisotropy model, the contrast energy is normalized by the standard deviation across the outputs of the orientation channels. This normalization by anisotropy model applies greater normalization when the contrast energy is concentrated in a single orientation channel, resulting in a lower response for gratings. There is no explicit representation of centers, surrounds, or feature matching in the normalization pool. This implementation is consistent with the idea that responses are reduced by the amount of redundancy in the image.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The normalization by orientation anisotropy model also accounts for the responses in V1-V3.

(A) Schematic of the normalization by orientation anisotropy model. (B) The mean fMRI responses from V1, V2, and V3 are replotted from Fig 3. The red dots are the cross-validated predictions from the normalization by orientation anisotropy model. Data and model fits plotted from the function s4_visualize(’figure 6’) in the code repository. See Fig A1 in S5 Appendix for fits to all 4 data sets.

https://doi.org/10.1371/journal.pcbi.1011704.g006

The normalization by anisotropy model exhibits similar predictions to the orientation-tuned normalization model, capturing the larger response to the snakes (Fig 6). Both models predict that the responses to snakes are about double the responses to gratings, similar to the data. It also predicts a higher slope for the snakes than the gratings, both as a function of density and contrast. The success of these models validates the idea that normalization depends on how contrast energy is distributed across orientations, not just on the overall contrast energy.

2.6. The two normalization models that are sensitive to orientation accurately predict responses to a wide range of stimuli

Both models without orientation sensitivity fail to account for the higher responses to snakes than gratings (Fig 7), and have low variance explained (Fig 8). Models that include orientation sensitivity capture the higher responses to snakes (Fig 7), and fit the data accurately (Fig 8). This suggests that sensitivity to orientation should be incorporated into normalization models of visual cortex. The accuracy differences across models are not due to the number of free parameters: the untuned normalization model has the same number of free parameters (three) as the tuned normalization model and the anisotropy model. Moreover, the prediction accuracy was computed using cross-validation, so that having more parameters does not necessarily lead to better predictions. The tuned normalization model has a numerically higher accuracy than the anisotropy model for nearly all data sets in all conditions (S3 Appendix), but the advantage of the two models with orientation sensitivity dwarfs the slight difference between these two models. Interestingly, for the two untuned models, prediction accuracy declines from V1 to V2 to V3, whereas for the two tuned models, accuracy increases (target stimuli) or stays flat (all stimuli) from V1 to V2/V3 (black lines in Fig 8).This pattern is consistent with the notion that along the visual hierarchy, neural responses become increasingly sensitive to statistical regularities, such as similarity in features across the image (see Discussion section 3.4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The orientation dependent models account for the higher responses to snakes than gratings.

For each visual area, V1-V3, we averaged the response to snakes and to gratings across stimuli in the target set and across the 4 data sets to compute the ratio of snakes to gratings: mean(snakes) / mean(gratings). We computed this value for each data set and plotted the average and standard error across the four data sets. In the data, the response to snakes is about double to gratings. This is matched in the two normalization models that are sensitive to orientation, but not the other models. Data and model fits plotted from the function s4_visualize(’figure 7’) in the code repository. CE = contrast energy; DN = untuned normalization; OTN = orientation-tuned normalization; NOA = normalization by anisotropy.

https://doi.org/10.1371/journal.pcbi.1011704.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Cross-validated variance explained for 4 models across all ROIs and data sets.

(A) R² for the target data set, comprising 18 stimuli for data set 1 (DS1) and data set 2 (DS2) and 17 for data set 3 (DS3), data set (DS4). The number of fitted model parameters (degrees of freedom) is indicated in parentheses for each model type. The four bars in each group correspond to data sets 1–4. The black horizontal lines are the means across the 4 data sets. (B) Same as panel A, but for a larger set of stimuli (50 for data set 1; 48 for data set 2; 39 for data set 3 and 4). The R² values are also reported in S3 Appendix. Data plotted from the function s4_visualize(’figure 8’) in the code repository. Abbreviations as in Fig 7.

https://doi.org/10.1371/journal.pcbi.1011704.g008

Because the two orientation-sensitive model were motivated by the need to explain the greater response to snakes than gratings, it is important to test the models on other stimuli as well. We refit all 4 models to the full data sets, which consisted of 50 (data set 1), 48 (data set 2) and 39 (data set 3, data set 4) stimuli, spanning a variety of texture types. In addition to the snakes and gratings, there are textures we refer to as noise bars, waves, plaids, and circular (S2 Appendix and Table C in S1 Appendix).

Just as with the target stimuli, across the full sets of stimuli, the tuned normalization model and the anisotropy model made accurate predictions, explaining 63%-77% and 49%-66% of the cross-validated variance in V1-V3 for the example data set (Fig 9). These two models also provide good fits to the other three data sets, shown in S5 Appendix. The fits to the larger stimulus sets, like the fits to the target stimuli alone, capture the observation about the two stimulus classes, meaning a larger predicted response for snakes than gratings. The two models also accurately predict lower responses to waves (one dominant orientation) than noise bars (many orientations). The two models also predict increasing response amplitudes from gratings (one orientation) to plaids (two orientations) to circular (16 orientations), as evident in stimulus sets 3 and 4 (Figs D3-D4 and E3-E4 in S5 Appendix). This pattern of predictions matches the data. The untuned models do not differ in their predictions for these three stimulus categories.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The orientation dependent models fit a wide variety of stimuli.

The mean fMRI responses from V1, V2, and V3 are shown for the full set of stimuli from data set 2 (See S2 Appendix). The data for snakes and gratings are replotted from Figs 3, 5 and 6, and again shown in dark and light gray. An even lighter gray is used for all other stimulus classes. The red and yellow dots are the cross-validated predictions from the orientation-tuned normalization model and the normalization by orientation anisotropy model. See Figs B-E in S5 Appendix for similar plots for data sets 2–4. Data and model fits plotted from the function s4_visualize(‘figure 9‘) in the code repository.

https://doi.org/10.1371/journal.pcbi.1011704.g009

Across ROIs and data sets, the orientation-tuned normalization model accounts for the highest variance, with R² ranging from 55% to 77% (Fig 8, bottom; S3 Appendix, the third row). The anisotropy model ranks second in all cases, substantially outperforming the two baseline models. Similar to the pattern with the target stimuli, when fitting to all stimuli the two untuned models show substantially decreasing accuracy from V1 to V2 to V3. The tuned normalization model and the anisotropy model decrease only slightly in accuracy from V1 to V2 to V3, meaning that the advantage for the tuned models is largest in extrastriate areas.

3.1 Why do some models fail to account for the much larger response to snakes?

We began with the observation that the BOLD response is about twice as large for patterns with curved contours (snakes) as for similar stimuli with straight, parallel contours (gratings). The difference in the responses is not a peculiarity of the BOLD signal, as a similar pattern was also observed in human intracranial measures of the field potential (broadband power from ECoG electrodes). The contrast energy model without normalization and the contrast energy model with untuned normalization did not predict large differences between the responses to the two stimulus classes. Both models pool contrast energy over space and orientation without an orientation-specific normalization or other orientation specific non-linearity. If two images have the same total contrast energy, then the way the energy is distributed across orientation channels will not matter, either for the measure of total energy (contrast energy model) or for the amount of normalization (untuned normalization model).

The implementation of the orientation-tuned normalization model was motivated in part from electrophysiology results showing that in V1, surround suppressive fields tend to be tuned to orientations close to the RF center’s preferred orientation [35]. Psychophysical experiments also show interactions between a target and surround that depend on matched orientation [e.g., 42]. While there is a lot of evidence from electrophysiology, fMRI, and psychophysics in support of feature tuning in normalization, here we explicitly compared image-computable models with and without feature-tuned normalization. Implementing the model and fitting it to data enabled us to assess whether (1) it quantitatively accounts for two-fold difference in response to snakes vs gratings (2) whether it also provides good fits to a wide range of other stimuli that it was not explicitly implemented for and (3) whether it can capture other more subtle effects, like the difference in the slope of the contrast response function between snakes and gratings. In all three cases, the answer was yes. We also implemented a second model with tuned normalization (the normalization by orientation anisotropy model), and this model also provided an affirmative answer to these questions, confirming the importance of including orientation dependence in the normalization computation. We discuss similarities (3.2) and differences (3.3) between the two models below.

3.2 Model behavior: Models with orientation dependent normalization capture the differences in both mean and in slope between snakes and gratings

The two models with orientation dependent normalization predict larger outputs, on average, for snakes than gratings, even when the contrast energy of the stimuli is approximately matched. This is due to how the normalization is computed. The grating stimuli elicit large outputs in the orientation channels that are matched to the stimulus, moderate outputs in adjacent orientation channels, and little to no response in other channels. As a result, there is high anisotropy (standard deviation across channel outputs), resulting in more suppression in the normalization by orientation anisotropy model. There is also high suppression from the surround in the orientation-tuned normalization model. The two models were implemented to capture the difference in mean between the two stimulus classes, so perhaps it is not so surprising that they do so.

The two models, unlike the contrast energy model and the untuned normalization model, also accurately predict steeper slopes for snakes than gratings (with respect to both contrast and density), a pattern that the models were not explicitly motivated to capture. They predict the difference in slope because at low total contrast energy for the image, there is little normalization, and hence the response to a snake and a grating stimulus will be comparable. This explanation applies to both low contrast stimuli and low-density stimuli, because in both cases the summed contrast energy is low, meaning the normalization term in the denominator is small. Hence, little normalization at low contrast is expected from the model, and is also confirmed by empirical measures of spatial summation and surround suppression at different contrast levels [49,50]. The more nuanced prediction from these models is that for stimuli with high total contrast energy (high stimulus contrast and high density), there is a lot of normalization for gratings and much less for snakes, resulting in a more pronounced difference in predicted response. The difference in predicted responses at high contrast but not at low contrast causes a difference in slope. For the two untuned models, normalization increases with contrast energy, but it increases similarly for the snakes and gratings, hence predicting similar slopes.

Interestingly, the data and the two models with orientation dependent normalization also show a greater slope for “noise bar” stimuli than “waves” (Figs B3-B4 and C3-C4 in S5 Appendix). The noise bars, like the snakes, have many orientations in first order contrast (but unlike the snakes, have only one dominant orientation for second-order contrast). The waves are the complement, with one dominant orientation for first order contrast (like gratings) but many orientations for second-order contrast. The pattern in the data and model predictions is that the stimuli with many orientations (noise bars) increase more steeply as a function of contrast than the stimuli with a narrower range of orientations (waves), supporting the observations made with snakes and gratings, but adding the further nuance that what seems to matter most is the orientation distribution (wide vs narrow) for first order rather than second-order contrast.

The difference in slopes between snakes and gratings in the data is related to, but not identical to, results observed for single unit V1 cells. For a typical V1 cell, the contrast response function has a higher slope for preferred than for non-preferred stimuli [51–53], a pattern also predicted by normalization models [7]. The pattern is predicted because of the difference in the numerator, which is high for preferred stimuli and low for non-preferred. The denominator is about the same for the two stimuli, proportional to the total contrast energy in the stimulus. We attribute the difference in slope between snakes vs gratings to a difference in the denominator of the normalization equation, which is high for gratings, low for snakes. The numerator is about the same for the two classes, proportional to the total contrast energy. In both cases–single unit responses to non-preferred stimuli, and population responses to grating stimuli–the reduced slope is due to a large amount of normalization relative to the driven response.

3.3 What is the tuning in orientation-tuned normalization?

We implemented two models with orientation dependent normalization. Both capture the same tendency for more normalization for homogeneous stimuli like gratings. They differ in implementation, however. The tuned normalization model differs only slightly from more typical normalization models: the only difference is that the normalization weights happen to conform to a specific pattern, such that cells with similar feature tuning (here, orientation) with nearby receptive fields have high weights, and cells with different feature tuning have low weights. As a result, surround suppression is most effective according to this model when the orientation of a surrounding region is matched to the preferred orientation of a cell. This is justified by findings from single unit data in macaque V1 that

“the surround influence was always suppressive when the surround grating was at the neuron’s preferred orientation” [35].

The normalization by orientation anisotropy model is a larger departure from the standard normalization model, since the computation of anisotropy is not a simple weighting of the outputs of nearby cells. Unlike the tuned normalization model, it has the greatest suppressive effect when the features of a stimulus are anisotropic (like a grating) irrespective of whether the stimulus orientation matches the center tuning of a cell. Interestingly, there is also empirical support for this pattern from the same study by Cavanaugh et al., 2002 [35]: specifically, evidence for “the tuning of the surround being dependent to some degree on the stimulus used in the center—suppression was often stronger for a given center stimulus when the parameters of the surround grating matched the parameters of the center grating even when the center grating was not itself of the optimal direction or orientation.” This pattern has been found in multiple studies, indicating that for macaque and cat V1 cells, surround suppression is often maximal when the surround and center stimulus orientation match, independent of the orientation preference of the cell [37,54,55]. Put another way, the tuning of the surround changes as the center orientation or direction changes. As shown by simulation, our normalization by anisotropy model can capture this observation from single units, but our tuned normalization model cannot (Fig 10). In this regard, the Normalization by orientation anisotropy model is more like Coen-Cagli et al’s [22] model of single-cell data than is the Orientation-tuned surround model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Center surround interactions in single units and computational models.

(A) Data replotted from three example cells from macaque V1 [35]. The thin lines are tuning curves for drifting gratings in the cell’s RF center, with 0 deg indicating the preferred direction. The thick lines and open circles show responses to stimuli including a center and surround. The center direction is indicated by the large black dot, and the surround direction is indicated by the values in the x-axis. The important observation is that the greatest suppression (indicated by the asterisks) occurs when the surround direction matches the center direction, not when the surround direction matches the center’s preferred direction. (B) Simulations for the orientation-tuned normalization model for stimuli similar to those in panel A (but static rather than drifting). Because the surround suppression is matched to the center’s preferred tuning, the largest suppression occurs at 0 deg, differing from the single unit data. (C) Simulations for the normalization by orientation anisotropy model for the same stimuli as in panel B. Here, the greatest suppression is when the surround orientation matches the center orientation, similar to the single unit data. Data and simulations plotted from the function s4_visualize(‘figure 10‘) in the code repository.

https://doi.org/10.1371/journal.pcbi.1011704.g010

The two models we implemented leave us in an unusual position. We have one model that has slightly higher prediction accuracy and is more in line with standard normalization models (orientation-tuned normalization), and a second model with slightly lower (but still high) prediction accuracy, but a closer fit to some data from single units (normalization by orientation anisotropy). The simplest conclusion is that a contrast energy model incorporating some form of orientation-dependent normalization greatly outperforms models that do not have tuned normalization. Given the diversity of cells in visual cortex, it is not likely that a single, simplified model will be sufficient to capture the behavior of all cells or cell populations for all stimuli. A neural network model with recurrence, temporal dynamics, and with feedforward, feedback and lateral connections might provide insight into the specific ways in which the surround modulation emerges [54,56], particularly given evidence that normalization changes as the response to a stimuli unfolds [57–60].

Interestingly, the denominator in the normalization by anisotropy model (the normalizer) is almost the same as the numerator of an orientation variance model used to predict the amplitude of gamma oscillations measured with ECoG electrodes in human subjects [13]. The fact that the same term is used in the numerator to predict gamma oscillations and the denominator to predict BOLD is consistent with the empirical observation that many of the stimuli that are most effective for driving large gamma oscillations (high contrast luminance gratings) are relatively ineffective for eliciting BOLD signals [61,62], and multiunit action potentials [63,64]. The link may be that gamma oscillations, rather than being the fundamental mechanism for perception and long-range cortical communication [65,66], are rather a result of the normalization process [13,67]. An open question is whether gamma oscillations might be predicted as accurately, or perhaps more accurately, by the normalization pool used in our orientation-tuned normalization model than the normalization pool of the normalization by anisotropy model.

3.4 Orientation dependent normalization and image statistics

The normalization by orientation anisotropy model and the orientation-tuned model showed larger advantages over untuned models for V2 and V3 compared to V1. This pattern is consistent with findings from electrophysiology and computational modeling of behavior. First, evidence from the non-human primate visual system suggests that the orientation tuning of surround suppression in V1 arises in large part from feedback from extrastriate areas, especially for the more spatially distant effects of surround suppression [55,68,69]. If the tuned suppressive effects depend on computations in extrastriate regions, then these regions may also exhibit more tuned suppression than V1.

More generally, tuned suppression, either in the form of our tuned normalization model or the anisotropy model, reflects sensitivity to higher-order image statistics (correlations over space between filter outputs). Models of extrastriate neural responses, especially V2, also tend to be more sensitive to higher-order image statistics than to contrast energy per se, such as models based explicitly on texture statistics [28–30,70,71], or models with multiple subunits that may exhibit heterogenous feature tuning [72–74].

We consider one specific way that our normalization by anisotropy model might be linked to the V2 models proposed by Simoncelli and colleagues. Suppose the V1 population computes contrast energy localized in orientation and space. (For simplicity, we ignore spatial frequency, as we did here experimentally by band-pass filtering our stimuli.) We then suppose that the V2 cells compute various weighted sums of the V1 outputs. Specifically, we assume that, for each spatial location, the weights among V2 cells form a Fourier basis set on the V1 outputs (across orientation). If these weights are arranged in pairs (similar to the odd and even V1 filters), and the outputs are squared and summed across phase (similar to the V1 energy model), then then the summed V2 population output will be proportional to the variance in the V1 response across orientation (David Heeger, personal communication). If this population output is the normalization pool for V2, then we get a normalization term like that in the anisotropy model, which normalizes by the standard deviation across orientation channels. A V2 model based on these principles would also need a similar term in the numerator, which we do not include. Hence the link is only to the denominator of the model.

3.5 Are typical contrast energy V1 models missing something important?

No model is complete. The standard normalized contrast energy model of V1, when fit with appropriate parameters, can capture substantial variance in V1 responses [75], but not all. Our results indicate that the model fails for at least some relatively simple stimulus classes, and that the failure can be large. But whether a more complex model is needed, such as a gated normalization model [22] or either of the orientation dependent normalization models we implemented, will depend on the stimulus set tested and the purpose of modeling. It is reasonably likely that the computations in both the anisotropy and the tuned normalization models are more connected to computations in extrastriate visual maps than in V1, but may also influence the response in V1 via feedback. [55,76].

More generally, since the development of divisive normalization models in the early 1990s, there has not yet been a generally agreed-upon description of exactly which cell populations contribute to normalization, and with what weights. Some attempts have been made, by assuming efficient coding of natural images [46,77] or by fitting model parameters to neural data [78]. And while there is a large literature on divisive normalization, including its tuning, many issues remain unresolved, including whether normalization within receptive field centers is orientation-tuned (in addition to the extra-classical surround being orientation-tuned) [78]. In this sense, a standard model of V1, with parameters set, that is downloadable and executable on arbitrary input images, does not yet really exist.

Both the anisotropy model and the tuned normalization model we presented make some advance but also have some important limits, most notably the lack of spatial receptive fields, as well as a lack of sensitivity to second-order contrast. Hence, they do not supersede other models, but rather provide compact computational summaries of response patterns that are not well captured by other models. Integrating better computational tools for validation [79], model-based stimulus development [80,81], high-quality standardized data sets [82], and theory [83], may offer the best path toward more complete understanding of neural circuits in visual cortex.

4.1 Ethics statement

Participants provided written informed consent. The experimental protocol was in compliance with the safety guidelines for MRI research and was approved by the University Committee on Activities involving Human Subjects at New York University (IRB-FY2016-363).

We analyzed and modeled four fMRI data sets for this paper. Data sets 1 and 2 were collected at NYU. Data sets 3 and 4 are re-analyzed from a previous paper [11], for which the fMRI data and stimuli are freely available online (http://kendrickkay.net/socmodel/).

4.2 Participants

The two NYU participants were both experienced MRI subjects (female; 24, 29 yo). Data were collected at NYU’s Center for Brain Imaging. The experimental protocol was approved by the University Committee on Activities Involving Human Subjects, and informed written consent was obtained from the participants before the study. Both participants had corrected-to-normal vision. The subjects participated in two separate scanning sessions, one for retinotopic mapping and one for the main study on encoding of textures.

4.3 Stimuli

Publicly accessible links to the stimuli, the names of the stimulus classes, and the correspondence between our naming convention and those in the Kay et al., 2013 [11] paper are described in Tables A, B, and C in S1 Appendix. The total spectral power of all images can be visualized using function visualizeStimulusPowerSpectrum in the code repository and is shown for target stimuli from Dataset 2 of Fig A in S1 Appendix.

4.4 MRI

The methods for MRI acquisition and preprocessing for data sets 3 and 4 are described in Kay et al., 2013 [11]. In brief, each data set comes from one subject, who viewed a variety of stimuli in an event-related fMRI design. Data set 3 was collected over two scan sessions and each stimulus was presented 6 times. Data set 4 was collected over one scan session and each stimulus was presented 3 times. (Note that both in this paper and the Kay website (http://kendrickkay.net/socmodel/), these two data sets are referred to as data sets 3 and 4. However, the Kay website refers to the stimuli for these two data sets as stimulus set 2 and the subjects themselves as “subject B” and “subject C”. We do not adopt these latter two conventions.)

After preprocessing the data (slice-time correction, co-registration, spatial unwarping), a general linear model was applied using the GLMdenoise toolbox [11]. The output of this algorithm includes a coefficient (beta weight) for each stimulus for each voxel solved from the whole fMRI session, as well as 30 bootstrapped estimates of each beta weight (bootstrapping across fMRI runs). The publicly available data (http://kendrickkay.net/socmodel/) are already pre-processed, denoised, and organized by ROI. Specifically, the data we used are in the files called “data set03.mat” and “data set04.mat” (http://kendrickkay.net/socmodel/data set03.mat, http://kendrickkay.net/socmodel/data set04.mat). The data sets are described on the website as “Data set 3 (subject B)” and “Data set 4 (subject C),” respectively. Within the MATLAB files, we used the stored 3D array called “betas” (voxels × stimuli × bootstraps), limited to V1, V2, V3 as indicated in the grouping variables “roi” and “roilabels”, and limited to the 39 stimuli indicated in Table B in S1 Appendix. Visual areas were identified by retinotopic mapping in a separate session.

4.5 Model equations

In the Results, we compared the accuracy of four models fit to the data, three of which are based on existing models or empirical findings–a contrast energy model, a untuned normalization model, and an orientation-tuned normalization model–and one new model, which computes normalization by orientation anisotropy. In this section we describe the computation that comprises each model.

All four models consist of three primary steps: (1) computation of oriented contrast energy, (2) pooling across orientation and space, and (3) a power-law nonlinearity. Steps 1 and 3 are identical for all models. Step 2, spatial pooling, varies between models.

1. Contrast energy: We denote by I(x,y) the value of the pre-processed input image at coordinates (x,y). The pre-processing causes the image values to have mean 0 and range from -0.5 to 0.5. The image is projected onto a set of 128 Gabor filters, which comprise 8 orientations θ, spaced every 22.5 deg; 8 spatial frequencies f, with peak spatial frequency log spaced from 0.75 cpd to 6 cpd; and 2 phases ϕ, separated by 90° (i.e., “quadrature”). F(x,y,θ,f,ϕ) indicates the Gabor filter at a spatial location (x,y), orientation θ, spatial frequency f, and phase ϕ. Each filter comprised a cosine or sine function of 4 cycles, windowed by a Gaussian with SD of 1 cycle. The outputs over the two phases are squared and summed to compute the contrast energy and summed across spatial frequencies. Finally, the contrast energy as a function of spatial position (x,y) and orientation θ becomes (1)

We convolve the image I and the filter F. The computation of contrast energy has no free parameters. Prior to convolution, stimuli were padded with uniform gray (mean luminance) on all sides by the width of the largest filter. After convolution, all energy images were downsampled to 12 pixels per degree for computational efficiency. We note that for simplicity, we summed over spatial frequency channels with uniform weights. If one were to fit separate parameters for each voxel, then one might expect spatial frequency tuning to vary with eccentricity. Nonetheless, the simplification of uniform weighting is reasonable given that the spatial frequency content of our stimuli is concentrated in a single octave (~2–4 cpd), and fMRI studies of spatial frequency tuning find a wide bandwidth at the voxel level, std of 2.2 octaves, or full width at half max of 5.1 octaves [88].

2. Spatial pooling. Each model differs in how contrast energy is pooled to yield a scalar value, s: (2) where we use square brackets to indicate a function of a function (also called a functional). We describe the pooling functional Φ, for each model below.

3. Power-law nonlinearity. Finally, the scalar is passed through a power-law nonlinearity to predict the BOLD amplitude r in units of percent signal change: (3)

Where g is the gain and α is the exponent parameter. These are free parameters fit to the fMRI data. The power-law nonlinearity is similar to divisive normalization in the case where each unit in a population is normalized by the same pool [10].

4.5.1 Pooling functional for contrast energy model.

In the contrast energy model, the contrast energy is summed over orientations and space to yield a scalar output, s.

(4)

N_ori is the number of orientation channels (always 8) and N_pixels is the number of pixels per stimulus in the padded images (344², 419², or 342²). There are no free parameters in this pooling functional for contrast energy, so the complete contrast energy model has only two free parameters, g and α, both from the power-law nonlinearity step (Eq 3).

4.5.2 Pooling functional for divisive normalization model.

The contrast energy in the untuned divisive normalization model is normalized before it is summed: Each (x,y,θ) element in the energy image is normalized by a weighted sum of elements at (x′,y′,θ′). The weighting is a Gaussian function of distance from location (x,y) and can thus be expressed as a convolution of the contrast energy E, with a Gaussian, G: (5)

The standard deviation of G is 4% of the padded image size, which is approximately 1 deg. G is identical across the 8 orientations.

The normalized contrast energy is (6) where σ is a parameter to control the strength of normalization. When σ is large, the normalization is low, and the overall expression approximates the contrast energy model. When σ approaches 0, there is strong normalization. We then sum d across space and orientation to result in the scalar, s.

(7)

The pooling functional for divisive normalization introduces one free parameter, σ. As with the contrast energy model, a power-law nonlinearity is applied to s to predict the BOLD response in percent signal change (Eq 3). Hence the complete model has three free parameters.

We note that the complete divisive normalization model has two similar non-linearities, one in the pooling functional (divisive normalization) and one on the final output (power-law). This is consistent with prior work showing that two stages of normalization (a cascade model) improved model accuracy [11].

4.5.4 Pooling functional for normalization by orientation anisotropy.

In the normalization by orientation anisotropy (NOA) model, the pooling step first sums the contrast energy across space within an orientation band, resulting in one value per orientation band, E_ori(θ): (8)

E_ori indicates the oriented energy. This energy at each orientation is then normalized by the standard deviation across the 8 orientations, and then summed to produce a scalar.

(9)

Where calculates the standard deviation of the oriented energy and a non-negative parameter w controls the strength of the normalization. When σ is large, the normalization is low, and the overall expression approximates the contrast energy model. When σ approaches 0, there is strong normalization by the standard deviation across orientation channel outputs. Calculating the standard deviation of oriented energy involves a squaring operation. To keep the parameters comparable across different models, we also square the numerator and the parameter σ. The NOA pooling functional has one free parameter, σ. The complete model, including the power-law non-linearity (Eq 3) has 3 free parameters.

4.6 Optimization

In each model, we fitted the model free parameters using the MATLAB optimization tool fmincon by minimizing the squared error between the model prediction and the corresponding BOLD amplitude. Because each stimulus consisted of 9 exemplars shown to the subject in rapid succession, the model prediction for each stimulus was obtained by averaging the model predictions across the exemplars. To avoid getting stuck in the local minima of the nonconvex landscape, we ran the optimization algorithm with 40 different parameter initializations. Each initialized value was picked randomly. All parameters were unbounded in the search, minimizing human interference in the fitting. Parameter α is passed through a sigmoid function to ensure its value is between 0 and 1.

4.7 Cross-validation scheme

All models were fit using an leave-one-out cross-validation scheme, where n is the number of stimuli. Thus, the BOLD signal prediction for each stimulus was generated by a model fit to all stimuli except that one. Under this scheme, the models are less likely to overfit data sets.

4.8 Accuracy metric

The model accuracy was quantified as the percentage of the explained variance (R²) in the human BOLD data by the cross-validated model predictions, (10) where r_i represents the BOLD amplitude to the i^th stimulus, represents the corresponding model prediction, and is the mean response across stimuli. We can understand this metric as the extra uncertainty reduction brought by the model beyond describing the BOLD data by its mean.