Simple foraging example

The full model of foraging under predation risk that we consider below includes a number of detailed components. Therefore, in order to illustrate and provide intuition for the workings of interruption, we start with a simple, stripped-down model.

Consider an animal (e.g., a rat) foraging in a habitat (Fig 1A) in which a predator (e.g., a hawk) arrives with probability 0.01 per unit time. The animal has to choose between three actions: (i) to continue to feed; (ii) to stop feeding to assess whether a predator is present; or (iii) to escape. Opting to feed has the benefit of gaining resources (worth 1 unit of reward per unit time), but has the drawback of providing only poor information (+) about whether the predator is present (e.g., because it restricts the animal’s field of view). Furthermore, if the predator is present, then the animal gets caught with probability 0.1 per unit time, incurring a cost of −100 units of reward, and causing the task to terminate. By contrast, assess provides the animal with improved information (++) about the presence of the predator, but has reward 0 per unit time. The exact details of how the quality of information is varied shall be described below, but the important point here is that feed and assess have distinct informational consequences. Finally, choosing to escape leads to a location that is safe, but does not permit foraging, thus gaining 0 reward; it also causes the task to terminate. The animal’s assumed task is to maximize the expected sum of its undiscounted future rewards (we consider the more conventional case of long run discounted rewards when we amplify this simple example).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The ability to interrupt behaviour promotes temporal commitment.

(A) Simple foraging example. A rat chooses to feed (reward +1 per time step), assess (reward 0), or escape (reward 0), in a habitat where the predator arrives with probability 0.01 per time step and then stays. Although assess is not rewarded, it provides more reliable information about whether the predator has arrived (++) than feed (+). If the predator has arrived, the probability that the rat gets caught if it remains in the habitat is 0.1 per time step. The decision process terminates (indicated by a !) either when the rat is caught (reward −100) or chooses to escape. (B) If the animal makes decisions at a high frequency, for example committing to an action a for only the minimum amount of time τ_min (top), it will be able to respond to unexpected changes (such as the arrival of the predator; grey box) but will also incur large decision costs; each decision incurs a cost of c_d. Committing for a longer duration τ has the advantage of reducing decision costs (middle), but is risky—the predator may arrive before the action terminates—unless the animal has the option to interrupt its behaviour at an earlier time (bottom). (C–F) Optimal policies, comprising an optimal action a* (left) and corresponding optimal duration τ* (right), for four different conditions. (C) No decision cost, c_d = 0. The optimal duration τ* is always as short as possible, meaning that the animal will be best able to respond appropriately if the predator arrives. (D) Decision cost c_d = 0.01, no interrupt. Optimal durations are essentially still as short as possible; the animal pays the greater cost of making high-frequency decisions in order to maintain its responsiveness. (E) Decision cost c_d = 0.01, interrupt. With the capacity to interrupt ongoing behaviour, the animal now selects long durations for feed and assess; the interrupt allows the animal to minimize its decision costs and also remain reactive, since it can always interrupt and make a new decision when required. (F) When the decision cost is increased further, c_d = 1, the shape of the policy changes so that escape is more predominant. (G) Ratio of average total amount of reward per trial (non-interrupt/interrupt) for different decision costs. For each decision cost, optimal non-interruptible and interruptible policies performed 1000 trials of the foraging task. (H) Average number of decisions per time step, and (I) average number of time steps of the optimal non-interrupible (blue circles) and interruptible (magenta squares) policies as a function of decision cost.

https://doi.org/10.1371/journal.pcbi.1005916.g001

The animal here has to negotiate two fundamental tradeoffs. One is between the rewards that can be directly obtained by choosing to feed and the improved information that can be obtained by choosing to assess. The second tradeoff is between future possible rewards that may be obtained by remaining in the habitat, and the sacrifice of those rewards in favour of safety by choosing to escape.

The final detail of the example is that each time the animal makes a choice about what to do, it selects both an action a (i.e., feed, assess, or escape) and a duration τ with which to perform it. Critically, each time the animal makes a decision it may incur a decision cost, c_d ≥ 0, which summarizes the various computational demands associated with such a decision; as discussed above, these may include planning and/or switching costs. If this decision cost is non-negligible, the animal additionally faces a tradeoff between reducing decision costs by making a prolonged temporal commitment to an action—i.e., reducing decision frequency by choosing a long duration τ—and being able to respond quickly if it is likely that the predator has arrived (Fig 1B).

Whether or not the animal is able to interrupt its ongoing behaviour is a critical determinant of its optimal policy. Here, a policy is a mapping between the degree to which the animal believes that the predator has arrived/is present, β(P), and an action-duration pair (a, τ). Fig 1C–1F display optimal policies, comprising optimal actions a* (left) and associated optimal durations τ* (right), for four different cases. The first (Fig 1C) is the optimal policy when there is no decision cost, c_d = 0. In terms of actions, the animal chooses to feed when it is unlikely that the predator has arrived (β(P) low), to escape when this is more likely than not (β(P) > 0.5), and to assess otherwise. More importantly for our purposes is the observation that optimal durations τ* are uniformly chosen to be as short as possible. Since making a decision has no cost, doing so at the highest possible frequency is the best way to ensure that the animal is best able to respond if it thinks the predator may have arrived.

Next, we consider the case where there is a small decision cost, c_d = 0.01, and the animal is unable to interrupt its own activity (Fig 1D). That is, if the animal chooses to engage in an action for τ seconds, it will be fully committed to performing the action for that duration. Here, there is no change in choice of actions, and very little change in duration except a slight increase in τ* for feed when the predator is very unlikely to have arrived. The low durations here, even though this entails a high frequency of decisions, mean that the animal is willing to pay these decision costs in order to remain responsive to possible threat.

In Fig 1E, the decision cost remains the same (c_d = 0.01), but the animal is now able to interrupt its behaviour. Interrupts also occur as a function of β(P), but (a) as noted, the information available to change β(P) is of lower quality during feed than assess; and (b) we assume that interruption itself is free (although there is then a standard decision cost associated with the necessary re-planning). Again, we see no change in which actions are chosen, but now both feed and assess are chosen to have the longest possible duration (in this example, there is no advantage to the animal of choosing to spend longer on escape). Since the animal can always interrupt itself, making such provisional commitments means that it can minimize decision costs by interrupting and making a new decision only when strictly necessary.

Finally, if the decision cost is increased further, changes in the pattern of optimal actions are observed. Fig 1F shows the optimal policy for c_d = 1, showing an increased propensity of the animal to escape. The alternatives are increasingly disfavoured by the animal not because of any change in risk of predation—this is unchanged—but because of the increased expense of making on-going decisions if it remains in the habitat.

The preceding discussion suggests that the ability to interrupt behaviour should be advantageous: it allows an animal to minimize decision costs by making provisional commitments to temporally-extended actions, while maintaining its ability to respond to changes in the environment. Indeed, an advantage in terms of rate of rewards is observed (Fig 1G). When there is no decision cost (c_d = 0), the ratio of reward rates (non-interruptible/interruptible) is 1, reflecting the fact that an animal would perform equally well whether or not it is equipped with an interrupt. However, as c_d increases, this ratio decreases, reflecting a reward advantage when the option to interrupt is available. Beyond a critical value, the ratio reverts to 1, with the animal deciding to escape immediately.

This advantage in terms of reward rate arises from the reduction in the frequency of decisions when the animal is able to interrupt (Fig 1H). This reduced decision rate makes it worthwhile for the animal to spend more time (safely) foraging in the environment (Fig 1I), thereby increasing its haul of rewards. The advantage disappears when planning is so expensive that the animal escapes immediately. However, the cost at which this occurs is lower when interruption is not possible (cf. Fig 1H and 1I).

States

States are determined by the values of three binary variables: 1) location ∈ {refuge, patch}, where refuge affords safety but not food, while patch affords foraging but also possible predation; 2) habitat quality ∈ {good(G), bad(B)}, which determines the current utility of feeding (see below); and 3) predator ∈ {present(P), absent(A)}, which describes whether there is currently a predator in the vicinity or not. We assume that the animal always knows its location, but that the state of predation and habitat quality are hidden variables whose values need to be inferred based on evidence.

Actions

As before, choice involves selection of both an action a and a duration τ for its performance, and we refer to an action-duration pair (a, τ) as an activity.

Possible actions are location-specific. In the refuge, the set is , where rest is a recuperative behaviour; assess specifically aims to increase certainty about whether a predator is currently in the environment from a position of relative safety (e.g., sniffing near the entrance of the refuge); and transit simply means moving to the patch location to forage.

In the patch location, the set of available actions is , where feed refers to the ingestion of food; assess, as before, is aimed at detecting whether a predator is present, though here from a position of possible danger; freeze aims at avoiding predation by decreasing the probability of detection; and escape also aims at evading predation but through returning the animal back to the refuge.

Transitions

The patch quality and predator presence are assumed to obey simple semi-Markov dynamics which are independent of the animal’s actions (we capture the consequences of feeding and predation in the rewards; see Discussion). In particular, we assume that (a) the predator transitions from being absent (A) to present (P) with transition rate γ_AP, and from present to absent with transition rate γ_PA; and similarly, (b) the habitat quality transitions between being good (G) and bad (B) according to transition rates (γ_GB, γ_BG).

In terms of predation risk, a crucial factor is the probability that the animal is detected and subsequently caught if a predator is present, given that the animal is currently engaged in a particular action. One of the more substantial simplifications we make is to assume that being detected inevitably leads to getting caught, but incurs only a fixed, finite, negative reward, rather than having a more extreme sanction. We formulate predation risk directly in terms of a rate of detection, assuming that this is directly mirrored in the predation rate. For rest and assess in the refuge location, we assume that this rate is 0; in the patch location, we assume that the detection rate is a function of the current action, written in abbreviated form as δ_a, where a denotes the current action. We assume that detection rate is lowest when the animal chooses freeze, and more generally assume the ordering δ_escape ≥ δ_transit ≥ δ_feed > δ_assess > δ_freeze. Note that we assume that the risk of being caught is also lower if the animal chooses assess, since assess and freeze may reasonably be seen as lying on a continuum which trades off the degree of immobility with the amount of information garnered through risk assessment [30].

Habitats which are good (G) or bad (B) are defined in terms of the animal’s encounter rate with either rich or poor patches while feeding. A habitat which is good has a relatively high encounter rate with rich patches and low encounter rate with poor patches. Conversely, a habitat which is bad yields a low encounter rate with rich patches, and a high encounter rate with poor patches.

Rewards

Each action is associated with a reward rate r. These are assumed to be negative (indicating an energetic cost) except for feed (net positive) and rest (zero). For net-negative reward actions, we assume the general ordering r_escape < r_transit ≤ r_assess ≤ r_freeze, so that escape is assumed to be most costly, while we are generally agnostic about the relative energetic costs of the other actions. For the feed action, the reward rate depends on whether the currently-encountered patch is rich (rate ) or poor (rate ). A large negative reward r_pred ≪ 0 is associated with being detected/caught by a predator (note that this is not a rate), which may be considered a cost of injury (we return to the issue of modelling predation costs in the Discussion).

As in the simple example above, we assume that making a decision incurs a constant decision cost, c_d ≥ 0, which summarizes the computational, and presumably metabolic, costs associated with deliberation about which activity to pursue.

Observations

The animal’s location is assumed to be directly observed, while the values of the predator and habitat variables are assumed to be only partially-observable. The animal is therefore required to make inferences about the latter which will depend on both prior knowledge of environment dynamics and observations.

We assume that certain observations are more probable when a predator is present rather than absent, provided the animal takes appropriate measures to detect them. Two distinct types of cue are assumed. Firstly, we assume that an indirect, or ‘passive’, cue o_i is available regardless of the activity in which the animal is engaged (e.g., hearing a rustle in the bushes). This is emitted at a rate when a predator is present, and at a rate when a predator is absent. We write ¬o_i for a non-observation of o_i when it could potentially have been observed. Secondly, we assume that a direct, or ‘active’, cue o_d is additionally available, but only if the animal is engaged in the assess activity (e.g., detecting a visual pattern at a particular location in the foliage). This is emitted at a rate when a predator is present, and at a rate when a predator is absent. Again, ¬o_d represents the non-observation of o_d when the latter would have been possible. Therefore, the animal may enter into different ‘information states’ regarding the predator variable depending on its choice of action, with assess providing the most reliable evidence. We assume that neither type of cue is available when the animal is engaged in rest. An absence of information is different from information about absence, as in ¬o_i or ¬o_d.

Information about habitat quality is assumed to be only available when the animal opts to feed. The relevant cue here is the current reward rate experienced while feeding. This will to some degree be informative about (by depending on) habitat quality—rich patches are more commonly encountered in a good habitat—but will not completely disambiguate the quality of the current habitat, since both types of habitat contain rich and poor patches.

Discount factor

The animal is assumed to discount future rewards according to an exponential function with rate α ∈ [0, 1] (i.e., a unit reward received after a delay of τ seconds is treated as having present value e^−ατ, so that a larger value of α leads to more rapid discounting). The effect on behaviour of varying the discount rate is not a primary focus of the current work, and it is set to α = 0.1 throughout.

Belief states

In addition to its current location, the animal’s belief about whether a predator is present () and whether the habitat is good () are jointly a sufficient basis on which to choose its actions. These collectively form the animal’s ‘belief state’—allowing us to solve the induced belief state semi-Markov decision process.

Belief state updates

How the animal’s beliefs change over time depends on both prior knowledge of the environment’s dynamics and any pertinent observations made. Since the predator and habitat quality variables evolve independently, we can consider belief updates for these separately.

In the case where there is no observation (such as when the animal engages in rest within the refuge), belief updates only depend on the environment dynamics. The two components of the belief state change from time t to t + τ according to (1) (2) For convenience, we consider an approximation to this for τ = Δt ≪ 1 (3) (4) When additional information is provided by observations, this needs to be combined with prior expectations according to Bayes rule. If the animal is engaged in an activity for which only indirect observations o_i provide information about the state of predation, then from Bayes rule, the updated belief having received an instance of o_i in the interval (t + Δt) is (5) where , and is given by Eq (3). For an omission, ¬o_i, the likelihood is used instead. If the animal has access to both indirect and direct observations (i.e., when engaging in assess), belief updates follow a similar pattern, e.g., (6) where , and so forth for the possible combinations of values for o_i and o_d.

Relevant information about the quality of the habitat is only available when the animal selects feed, and encounters either rich or poor patches. Thus, the updated belief when encountering a rich patch in the interval (t + Δt) is (7) where , and is given by Eq (4). If the encounter is with a poor patch, the likelihood is used instead.

Fig 2B and 2C illustrate how beliefs β^Q(G) and β^D(P) evolve over time for the various different cases.

Interruption

When available, interruption is defined as the ability to stop an activity prematurely and make a new decision. That is, given an initial commitment at time t to an activity (a, τ), interruption is the capacity to stop that activity at any intermediate time in the interval (t, t + τ), and make a new decision (at a cost of c_d). Interruption can therefore be thought of as an additional, ‘internal’ action; how decisions about this action should be made, as well as issues surrounding its potential cost and mechanisms, are the principal concerns of the following sections.

Basic model behaviour

To examine the basic behaviour of the model, we start by setting the decision cost to zero, c_d = 0. Fig 3A displays the optimal actions a* as a function of the animal’s location, refuge (left) or patch (right), and belief state {β^D(P), β^Q(G)}. In the refuge, the animal opts to transit to the patch only if a predator is unlikely, chooses to rest if a predator is likely, and selects assess when more uncertain. Choice is modulated by the probability that the habitat quality is good: the animal is slightly more likely to tolerate a higher probability that a predator is present in order to transit, while if the habitat quality is probably bad, the animal is increasingly likely to rest, in fact even when the probability of a predator is lower than 0.5. Note that even though the utility of rest was assumed to be 0, there are conditions under which the animal chooses it anyway. This occurs when the cost associated with assess is deemed too high to be worth paying (i.e., when it is highly probable that a predator is present, or when the habitat quality is likely to be bad). The advantage of assess is that it more quickly moves the animal to a state of increased certainty about whether a predator is present or not: if the predator is likely present, it is better to conserve energy by selecting rest; if the predator is likely absent, then the sooner the animal chooses to transit, the better.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Optimal policies in the absence of a decision cost.

(A) Optimal actions a* as a function of current location (left, refuge; right, patch), and current beliefs that a predator is present, β^D(P), and that the current habitat quality is good, β^Q(G). The discretization of the state space leads to the apparently rough solution. (B) If escape does not lead to safety, then freeze will always be selected in the patch instead. (C–E) The distribution of behaviour shown by the optimal policy depends on whether there is a predator in the environment (black bars) or not (white bars). This includes (C) the proportion of time spent in refuge vs. patch, and (D) the proportion of time spent in different activities. (E) Even though the policy selects activity durations to be as short as possible (τ* = 1 s), contiguous periods of a given action (‘bouts’) may be longer, reflecting successive choices of the same action. The bars show mean bout lengths measured over 100 instantiations of a 15-minute period; error bars indicate ±1 standard error. Environment parameters as above; reward rates , , r_rest = 0, r_transit = r_assess = r_freeze = −0.1, r_escape = −1; detection rates δ_feed = δ_transit = δ_escape = 0.05, δ_assess = 0.02, δ_freeze = 0.01, δ_rest = 0; predation punishment r_pred = −100; decision cost c_d = 0.

https://doi.org/10.1371/journal.pcbi.1005916.g003

When in the patch location, the animal selects feed when β^D(P) is low, assess when there is greater uncertainty, and chooses a defensive action ∈ {freeze, escape} when a predator is more likely than not to be present. Again, these tendencies are slightly modulated by the belief β^Q(G). The decision between freeze and escape is controlled by a number of factors in the model. Firstly, there is a difference in the cost of performing these actions, as we assumed that escape is more costly than freeze. Secondly, there is a difference in detectability while performing these actions—it is assumed that the detection rate is higher for escape than for freeze. Finally, there is the fact—which is the reason why escape is selected at all—that a successful escape will get the animal back to safety, while freeze leaves the animal in the patch location. Unsurprisingly, if escape is rendered ineffectual, in the sense that its performance also leaves the animal in the patch, then freeze is always preferred (Fig 3B), consistent with changes in defensive pattern observed in rats and mice when flight is not possible [30, 31].

As in the simple example considered above, when there is no decision cost it is always optimal for the animal to choose τ to be as short as possible (see ‘Supporting information’, S1 Fig). This is because there is no cost to doing so, while, in the absence of an interrupt, there is a potential cost of committing to longer durations (viz., not being able to change course of action if observations indicate a change in the environment). τ becomes relevant when we consider a nonzero decision cost below.

In Fig 3C–3E, we summarize aspects of behaviour when the optimal policy is repeatedly exposed to an environment where a predator is either absent (white bars) or present (black bars). Unsurprisingly, when a predator is present, the animal spends most of its time in the refuge (Fig 3C) engaged in either rest or assess activity (Fig 3D), whereas it spends most of its time feeding in the patch when there is no predator. The adaptiveness of these behaviours is evident, and qualitatively similar reconfigurations of activity patterns in response to predator presence/absence are observed in laboratory-based ethological studies (e.g., [30]). Fig 3E makes the further point that even if the shortest duration τ is always selected, this doesn’t mean that ‘bouts’ of behaviour, defined as continuous periods of performing a single action, will always be of minimal duration. Instead, an external observer would sometimes measure longer behavioural bouts, particularly in the case of rest and feed activities.

The price of responsiveness: Decision cost, without interrupt

As the decision cost rises from zero (c_d > 0), optimal behaviour changes. Fig 4A shows the optimal policy for decision cost c_d = 0.01, now displaying both optimal actions a* (left panels) in each location and corresponding optimal durations τ* (right panels). Optimal actions a* are essentially identical to the c_d = 0 case above (cf. Fig 3A), and optimal durations τ* are generally selected to be as short as possible. The exception engendered by this rather minimal decision cost comes in the case of rest, where longer durations are observed, particularly when the habitat is likely to be bad. These reflect the dynamics of the environment: if habitat quality is currently likely to be bad, and conditions change relatively slowly, then conditions are likely to remain bad in the near future. Thus, rather than making another (costly) decision to rest at that future time, the animal can safely commit to rest for a longer period.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Non-zero decision costs encourage temporally-extended activity when interruption is possible.

In each case, there are four plots: the optimal choices of action a* (left plots) and duration τ* (right plots) are shown as a function of location (refuge, upper; patch, lower) and belief {β^D(P), β^Q(G)}. (A;B) Optimal non-interruptible policies for decision costs (A) c_d = 0.01 and (B) c_d = 0.2. (C;D) Optimal interruptible policies for the same decision costs: (C) c_d = 0.01 and (D) c_d = 0.2. Parameters otherwise set as above.

https://doi.org/10.1371/journal.pcbi.1005916.g004

By contrast, all other actions are associated with short durations. As in the previous case, this is sensible considering the consequences of doing otherwise. For example, when in the refuge, an animal that commits to assess for an extended duration may thereby forego time that could be better spent foraging or resting; in the patch, the same commitment risks foregoing the opportunity to feed or respond defensively (freeze/escape).

If decisions are made even more expensive, e.g. c_d = 0.2, further changes in policy are observed (Fig 4B). In the refuge, rest becomes the most prominent action, with a duration that is uniformly chosen to be the longest possible. This minimizes decision costs when the environment is determined to be unfavourable. Note that under the assumed environment dynamics, beliefs β^D(P) and β^Q(G) will both move towards 0.5 once rest is initiated (recall that no observation is available during this activity), and yet the belief state (0.5, 0.5) yields the selection of rest under this policy. In other words, once this policy initiates rest, it will never do anything else, since the potential benefits of doing anything else are outweighed by the costs (i.e., any other option must have an average reward < 0, since rest has net reward 0). Of course, in reality, various factors would militate against this, including the stochasticity of action choice and progressive starvation (see Discussion). In the patch, it is notable that freeze has all but disappeared from the behavioural repertoire, replaced by escape. This is because its benefits—lower detectability and the avoidance of unnecessary excursions back to the refuge— are now outweighed by the burden of greater future decision costs incurred in the patch. Some subtle increases in τ* are discernible for feed at low levels of β^D(P); but overall, variation in τ* remains limited.

The benefit of interruptibility: Decision cost, with interrupt

What happens if we now allow the animal to interrupt activities prior to their completion? The animal’s optimal policy only recommends the activity (a*, τ*) in initial belief state {β^D(P), β^Q(G)} which is best in expectation over possible future belief trajectories. It is therefore perfectly possible that, having chosen optimally with respect to the initial belief state, experience sends the animal on a particular trajectory where it reaches a belief state in which an alternative activity would be preferable. Described at the level of a meta-decision, interruption should occur exactly when the benefit of interrupting an activity is greater than that of continuing it (cf. [15]).

Fig 4C shows the optimal interruptible policy for the same c_d = 0.01 case as before. The clear difference is that for actions assess, feed, and freeze, it is optimal to set the duration to be as long as possible: τ* = τ_max. Equipped with the (free) option of interrupting itself at any time, the animal can minimize its decision costs by provisionally committing to long activity durations, only interrupting and making another decision when it really needs to.

The exceptions are transit and escape. For transit (i.e., moving from refuge to patch), choosing a longer duration never makes any sense—it would only increase the associated energetic cost and predation risk—and there is no advantage to interrupting this activity in the model. The same reasoning applies to escape. Since beliefs evolve in a predictable manner for rest, there is never a reason to interrupt this activity—the duration is calibrated in the same manner as in the non-interruptible case.

Fig 4D displays the optimal interruptible policy for the more expensive, c_d = 0.2, case. Rest begins to occupy greater regions of belief space when in the refuge. This is similar to the trend in the non-interruptible case (cf. Fig 4B), albeit to a lesser extent. The interruptible policy does not get ‘stuck’ permanently selecting rest, but will rather select assess when uncertainty is greatest (i.e., at (0.5, 0.5)), and at many other points in this region. Note also that in contrast to the uniform choice of the longest duration for rest in the non-interruptible case, there is still some gradation in choice of τ in the interruptible policy (Fig 4D, upper right panel): when it is strongly believed that habitat quality is currently good, it is better to choose shorter durations of rest to be able to take advantage of predator-free foraging conditions in the near future (cf. Fig 4A and 4C).

Equipped with the capacity to interrupt itself, an animal should perform at least as well as when lacking this capacity (cf. [15], Theorem 2). At worst, decisions could be taken at maximum frequency, and the same decision costs incurred as in the non-interruptible case. We expect the interruptible case to do better than this, however: interruption should allow the animal to commit to extended activity flexibly, and so decrease the cost of unnecessary decisions—just as in the simplified example we first considered.

As in that example, we compared performance in a simulated experiment. Here, behaviour is measured over a 6-minute period in which a predator is initially absent (2 min), then present (2 min), then absent again (2 min); habitat quality is allowed to fluctuate randomly. We ran this experiment 1000 times for different settings of the decision cost c_d, ensuring that conditions were exactly matched between policies. We measured the resulting rewards averaged over both episodes and experiments. Fig 5A (inverted triangles; ‘exact’) plots the reward rate achieved by the non-interruptible policy as a fraction of that achieved by the interruptible policy (cf. Fig 1G). As seen before, when c_d = 0, this fraction is 1, reflecting the fact that the optimal strategy here is to make decisions as often as possible, since there is no penalty to doing so. However, as decisions become increasingly expensive (i.e., c_d becomes larger), the fractional utility rate decreases, reflecting the fact that the interruptible policy is achieving higher rewards. Beyond a certain level of decision cost (here, c_d > 0.4), the fraction reverts to 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The ability to interrupt on-going behaviour confers a reward advantage.

(A) Ratio of average total amount of reward per trial (non-interrupt/interrupt) for different decision costs. This is shown both for the exact interruptible policy (black triangles, dashed line), and the approximate interruptible policy (red asterisks) which is based on a linear approximation (as detailed in the Section ‘A cheap interruption mechanism’ below). For each decision cost, optimal non-interruptible and interruptible policies performed 1000 trials. (B) Average number of decisions per time step and (C) average percentage time spent in rest for the optimal non-interrupible (blue circles) and interruptible (magenta squares) policies. (D) Behaviour and beliefs of the optimal non-interruptible (upper) and interruptible (lower) policy for a particular trial when c_d = 0.1. Time points at which decisions are made are indicated by vertical dashed lines. Model parameters otherwise set as above.

https://doi.org/10.1371/journal.pcbi.1005916.g005

Again, the reward advantage comes from the reduction in the frequency of decisions that is possible when the animal has the capacity to interrupt (Fig 5B). Reversion of the fractional reward rate to 1 in this case corresponds to the point at which simply spending all of its time engaged in rest is the optimal course of action for the animal, regardless of the capacity to interrupt. If we explicitly compare the average time during the experiment spent at rest for non-interruptible and interruptible policies, both are eventually driven to exclusive choice of this action as decisions become more expensive. However, this occurs for lower values of c_d when interruption is unavailable (Fig 5C).

Fig 5D compares the behaviour and beliefs of the non-interruptible (upper) and interruptible (lower) policies for a particular run of the experiment at an intermediate decision cost, c_d = 0.1. This clearly illustrates the fact that the non-interruptible policy makes decisions at a much higher frequency (vertical dashed lines), but also highlights differences in choices. Most notably, in both cases the animal returns to the patch after the predator has been removed from the environment, but when faced with observations that may indicate a potential threat, their behaviour differs. In the non-interruptible case, the animal opts to escape to the refuge and engage in rest, since the costs of foraging are deemed too high when a predator is thought likely and habitat quality is low (Fig 5D, upper). By contrast, in the interruptable case, the animal opts to assess whether there is a real threat, and continues to feed when it transpires that no predator is around—it is sufficiently flexible in its behaviour for it to be worthwhile to continue in the patch and intermix periods of both feeding and assessment (Fig 5D, lower). Note also the shorter bouts of rest in the interruptible case, leading to a greater frequency of assess and so, at least in this case, a marginally earlier return to the patch.

A cheap interruption mechanism

Interruption is evidently useful; however, we have not considered the costs of the calculations associated with this operation. Since interruption can be thought of as another layer of decision-making, we might just have increased the computational burden. One answer is to conceive of a cheap and ‘light-weight’ interruption process.

The points at which interruption should be triggered effectively form a boundary on belief space. We might therefore consider that at the outset of an activity, one could ‘construct’ such a boundary, or threshold, and then have interruption occur whenever this threshold is breached. Belief-monitoring would still be required to recognize if and when the animal enters the termination set of beliefs. This still implies a decision—whether or not this threshold has been reached—but of a particularly simple kind.

Fig 6A–6C (symbols) plot the optimal interruption thresholds θ, i.e., levels of belief β^D(P) at which activities should be interrupted, as a function of decision cost c_d and three fixed levels of belief β^Q(G). The thresholds are plotted for feed and for assess ∈ {patch, refuge}, which are activities that show a robust increase in duration with nonzero decision costs (cf. Fig 4C and 4D above); since freeze tends to be disfavoured at higher decision costs (cf. Fig 4D), data points are much more sparse, and so we do not consider it here. Note that feed only has an upper boundary on β^D(P), while assess has both upper and lower boundaries. Also, for some combinations of c_d and β^Q(G), the set of beliefs {β^D(P)} for which assess in the refuge location is optimal (i.e., the ‘initiation set’) is empty, and so interruption thresholds are omitted in these cases (e.g., for β^Q(G) = 0.2, c_d > 0.2; Fig 6C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Optimal and approximate interruption thresholds.

(A–C) Optimal thresholds θ (symbols) for degree of belief β^D(P) as a function of decision cost c_d and (fixed) belief level β^Q(G). These are shown for (A) feed, (B) assess (in patch), and (C) assess (in refuge). The lines show a linear approximation to the thresholds. (D–F) Optimal and approximate thresholds for the same activities as a function of c_d and cue reliability (i.e., higher or lower true positive rate for indirect observations, , or direct observations, ); β^Q(G) = 0.5. Unless otherwise indicated, the default reliabilities were , , , . Other parameters as previous.

https://doi.org/10.1371/journal.pcbi.1005916.g006

As one might expect from the example optimal policies seen previously (cf. Fig 4C and 4D), the interruption threshold for feed increases for higher β^Q(G), i.e., as the payoff for feeding increases (Fig 6A). With assess in the patch location, the upper and lower thresholds move upwards as β^Q(G) increases, respectively indicating a greater willingness to interrupt and initiate feeding (lower boundary), and a greater reluctance to trigger defensive behaviour (upper boundary) (Fig 6B). For assess in the refuge, the upper boundary moves upwards as β^Q(G) increases, indicating greater reluctance to interrupt and initiate rest when habitat quality is likely to be good; the lower boundary shows less variation—the animal requires β^D(P) to be very low to interrupt and initiate transit, regardless of β^Q(G) (Fig 6C).

The thresholds θ for the first two cases are well approximated by linear functions of β^Q(G) and c_d, while this is less true of assess in the refuge location (Fig 6A–6C, dashed lines). We can nevertheless ask how well, in comparison to the optimal policy, an animal will do when selecting interruption thresholds according to the linear function which most closely approximates the optimal thresholds. Fig 5A (red asterisks) shows that this simple, approximate way of setting thresholds leads to benefits that are extremely close to that of the exact case.

It is also informative to examine how interruption thresholds change as a function of cue reliability. Fig 6D–6F show these as a function of decision cost c_d and true positive rates for either indirect cues, (Fig 6D), or direct cues, (Fig 6E and 6F), while β^Q(G) is kept constant. Reducing the true positive rate in either case means less reliable information about the predator.

For feed, thresholds decrease with less reliable indirect cues, consistent with greater caution—the animal is increasingly prepared to interrupt feeding in order to assess and gain better information (Fig 6D).

For assess in the patch location, upper thresholds similarly decrease with less reliable direct cues, reflecting a greater willingness to trigger interruption and switch to defensive behaviour (Fig 6E, upper). That lower thresholds actually increase with less reliable cues indicates a greater willingness to interrupt and initiate feed in spite of having less reliable predation cues; this might seem the exact opposite of more cautious behaviour. The reason is that gathering information via assess decreases in marginal value for unreliable cues, making feed an increasingly attractive alternative. Note that the relative reliabilities of direct and indirect cues are important in this trade off, since feed still provides some information via the latter class of cues.

Finally, thresholds for assess in the refuge location follow a qualitatively similar pattern (Fig 6F). With less reliable cues, the animal is quicker to interrupt its behaviour and initiate rest, thereby conserving its energy; with more reliable cues, the animal is more conservative in making this transition (Fig 6F, upper thresholds). The difference in thresholds is less pronounced in the downwards direction, so that a relatively high degree of certainty regarding predator absence is required before initiating transit in all cases.