Dear Dr. Bogacz,

Thank you very much for submitting your manuscript "Uncertainty-guided learning with scaled prediction errors in the basal ganglia" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments. The reviewers and I agree that this work is timely and interesting. There are two major issues to address, as detailed below: (1) A scholarship issue, regarding adequacy of the cited literature, and (2) a normative issue, regarding the link to Kalman filtering.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Samuel J. Gershman

Deputy Editor

PLOS Computational Biology

***********************

Reviewer's Responses to Questions

Comments to the Authors:

Reviewer #1: The authors propose in the present paper a neurocircuitry that might underlie the precision-weighting of prediction errors. The paper is well written, and their proposed model is interesting. Some sections constitute less of a novel contribution to the field though. Some rewriting and context would benefit particular sections, to clarify the novelty of the contributions.

The first part of the paper discusses their scale prediction error model, and perform simulations to demonstrate the model outperforms a more simple RW-model. This section is quite expansive, particularly since there have been many models that attempt to deal with uncertainty in prediction errors, and therefore does not constitute a novel contribution to the field. That is, researchers have covered the importance of precision-weighting of prediction errors, and have proposed various computational models that could account for such processes. For example see the work of Matthew Nassar, Chris Mathys, but also Schultz’s group, who worked on the Tobler findings that are central to this paper, who have done much work on precision-weighting of prediction errors in humans and the role of dopamine herein (For example see Diederen et al., 2015, 2016; 2017; Haarsma et al., 2020). I think the paper would benefit from mentioning this work when describing their particular model, to demonstrate what is novel about it, or why it might not be of importance to get into the differences between these models. I realise that later on in the discussion different models are discussed as well, but it’s not clear to me that their present model provides a clear advantage over other models that seem to capture the processes the authors are interested as well, like the hierarchical gaussian filter for example. If it’s the case that this section more acts as a proof of principle that precision-weighting is important, rather than coming up with a novel insight in reinforcement learning, I would suggest shortening this section and emphasise this point.

The second section discusses how the striatum might compute precision-weighted prediction errors by largely expanding on previous work by the same group on learning reward uncertainty. I think the model is of interest and constitutes a novel contribution to the field on how these mechanisms might be implicated in striatal sub-cortical loops. Some more discussion on how their model compares to other approaches might be beneficial here as well.

Reviewer #2: The authors present a new model for learning in noisy environments. The work is mostly focused on presenting a model which is biologically plausible and authors also present a detailed neuronal mechanism of the model in the basal ganglia. I think this is a timely question and the proposed solution is interesting and novel. The paper is well-written. I’m overall positive about this paper with one major caveat that I hope authors can sufficiently address.

The main issue with the paper is about its link to Kalman filtering, which is important within the paper because authors present their model as a variant of Kalman filter. Authors argue that whereas the line of Kalman filtering work in the literature posits that organisms should be able to track observation noise, there is no biologically-realistic computational model for that. But the main idea behind Kalman filtering is optimal tracking, which requires tracking both posterior mean and posterior uncertainty. The proposed model does not track posterior uncertainty. Authors seem to ignore posterior uncertainty altogether, which is technically the most important facet of Kalman filtering (when both process- and observation noise are fixed but are nonzero.) In fact, I believe (but I’m happy to be convinced otherwise) that their model actually recovers the posterior uncertainty but authors mistakenly label it as observation noise. That’s because they assume that zero process noise in their generative process (fixed mu in Eq 1). In Appendix S2, authors show that the model is equivalent to the Kalman filter when the ratio of observation- to process- noise is infinite. That only happens if one assumes that the process noise is zero. These are all fine for a descriptive model, but I think these assumptions ruin the link to the normative learning mechanism of Kalman filter. And the main problem is that it is not clear why we do need to make such a huge sacrifice from the normative point: does tracking posterior uncertainty really require some process that are so different from the current model?

Perhaps the strongest aspect of the proposed model is the basal ganglia mechanism for implementing the model at the neuronal level. I found that part timely and novel, but again I believe the same neuronal mechanism can be told for a model that scales prediction error with posterior uncertainty in the more general case that the process noise is nonzero.

A number of minor points:

- I suggest to move the proof in the “fixed point analysis” section to Methods to enhance readability of the paper for broader audience. I think that part is quite trivial for technical readers, and quite confusing for general psychologists/neuroscientists; so in both case does not help to present it in the forefront of results section.

- I found the discussion on the free energy models too vague. Maybe authors can explain more about those models and their relation to this work, especially given the recent work by Bogacz (2020).

- I also found the discussion on AI models quite speculative. I think there might be a weak connection between these models and deep RL models, but that is very non-specific (i.e. many other biological learning model might have similar connections). I think deep RL models are not competitors of the current model. Also they do not help to understand the current work.

- In the introduction, authors seem to imply that particle filtering is not biologically realistic without providing any reason why that is so (quite in contrary, there are influential papers arguing plausibility of Monte Carlo models for biological learning, eg Courville and Daw 2006; Gershamn et al, 2010). Relatedly, it is stated in the introduction that the model by ref 9 is not suitable to describe biological learning. While it is true that the focus of that work is the computational level, but that does not mean that it is not suitable to describe biological learning on the mechanistic or algorithmic level, especially given that it is a combination of particle filtering and Kalman filtering.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: None

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Dr. Bogacz,

We are pleased to inform you that your manuscript 'Uncertainty-guided learning with scaled prediction errors in the basal ganglia' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Samuel J. Gershman

Deputy Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Reviewer #2: Authors have revised the manuscript sufficiently and have addressed my comments.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: None

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No