Royal Society Publishing

Introduction. Modelling natural action selection

Tony J Prescott, Joanna J Bryson, Anil K Seth

Abstract

Action selection is the task of resolving conflicts between competing behavioural alternatives. This theme issue is dedicated to advancing our understanding of the behavioural patterns and neural substrates supporting action selection in animals, including humans. The scope of problems investigated includes: (i) whether biological action selection is optimal (and, if so, what is optimized), (ii) the neural substrates for action selection in the vertebrate brain, (iii) the role of perceptual selection in decision-making, and (iv) the interaction of group and individual action selection. A second aim of this issue is to advance methodological practice with respect to modelling natural action section. A wide variety of computational modelling techniques are therefore employed ranging from formal mathematical approaches through to computational neuroscience, connectionism and agent-based modelling. The research described has broad implications for both natural and artificial sciences. One example, highlighted here, is its application to medical science where models of the neural substrates for action selection are contributing to the understanding of brain disorders such as Parkinson's disease, schizophrenia and attention deficit/hyperactivity disorder.

Action selection is the task of resolving conflicts between competing behavioural alternatives, or, more simply put, of deciding ‘what to do next’. As a general problem facing all autonomous beings—animals and artificial agents—it has exercised the minds of scientists from many disciplines: those concerned with understanding the biological bases of behaviour (ethology, neurobiology and psychology) and those concerned with building artefacts, real or simulated, that behave appropriately in complex worlds (artificial intelligence, artificial life and robotics). Work in these different domains has established a wide variety of methodologies that address the same underlying problems from different perspectives. One approach to characterizing this multiplicity of methods is to distinguish between the analytical and the synthetic branches of the behavioural and brain sciences (Braitenberg 1986). From the perspective of analytical science, an important goal is to describe transitions in behaviour; these can occur at many different temporal scales, and can be considered as instances of ‘behavioural switching’ or, more anthropomorphically, as ‘choice points’. Analytical approaches also seek to identify the biological substrates that give rise to such transitions, for instance, by probing in the nervous system to find critical components—candidate action-selection mechanisms—on which effective and appropriate switching may depend. Beyond such descriptions, of course, a central goal of behavioural science is to explain why any observed transition (or sequence of transitions) occurs in a given context, perhaps referencing such explanation to normative concepts such as ‘utility’ or ‘fitness’. These explanations may also make use of mechanistic accounts that explain how underlying neural control systems operate to generate observed behavioural outcomes. It is at the confluence of these mechanistic and normative approaches that the synthetic approach in science is coming to have an increasing influence. The experimentalist seeks the help of the mathematician or engineer and asks ‘what would it take to build a system that acts in this way?’

Modelling—the synthesis of artificial systems that mimic natural ones—has always played an important role in biology; however, the last few decades have seen a dramatic expansion in the range of modelling methodologies that have been employed. Formal, mathematical models with provable properties continue to be of great importance (e.g. Bogacz et al. 2007; Houston et al. 2007). Now, added to these, there is a burgeoning interest in larger-scale simulations that allow the investigation of systems for which formal mathematical solutions are, as a result of their complexity, either intractable or simply unknown. However, synthetic models, once built, may often be elucidated by analytical techniques; thus synthetic and analytical approaches are best pursued jointly. Analysis of a formally intractable simulation often consists of observing the system's behaviour then measuring and describing it using many of the same tools as traditional experimental science (Bryson et al. 2007). Such an analysis can serve to uncover heuristics for the interpretation of empirical data as well as to generate novel hypotheses to be tested experimentally.

The questions to be addressed in considering models of action selection include: is the model sufficiently constrained by biological data that its functioning can capture interesting properties of the natural system of interest? Do manipulations of the model, intended to mirror scientific procedures or observed natural processes, result in similar outcomes to those seen in real life? Does the model make predictions? Is the model more complex than it needs to be in order to describe a phenomenon, or is it too simple to engage with empirical data? A potential pitfall of more detailed computational models is that they may trade the sophistication with which they match biological detail with comprehensibility. The scientist is then left with two systems, one natural and the other synthesized, neither of which is well understood. Hence, the best models hit upon a good trade-off between accurately mimicking key properties of a target biological system at the same time as remaining understandable to the extent that new insights into the natural world are generated.

In this theme issue, we present a selection of some of the most promising contemporary approaches to modelling action selection in natural systems. The range of methodologies is broad—from formal mathematical models, through to models of artificial animals, here called agents, embedded in simulated worlds (often containing other agents). We also consider mechanistic accounts of the neural processes underlying action selection through a variety of computational neuroscience and connectionist approaches. In this article, we summarize the main substantive areas of this theme issue and the contributions of each article and then return briefly to a discussion of the modelling techniques.

1. Action selection and optimality

When an animal does one thing rather than another, it is natural to ask ‘why?’ A common explanation is that the action is optimal with respect to some goal. For example, when observing the foraging behaviour of a shorebird, one may ask whether the intake of food is being maximized. This view, a direct extension of Darwinian principles, has its more recent roots in behavioural ecology (Krebs & Davies 1997) and optimal foraging theory (Stephens & Krebs 1986). Assessing animal behaviour from a normative perspective has particular value when observations deviate from predictions, because the scientist is now forced to consider the origin of the apparently suboptimal, or ‘irrational’, behaviour.

Many animals behave according to Herrnstein's (1961) ‘matching law’, in which responses are allocated in proportion to the reward obtained from each response. However, as both Houston et al. (2007) and Seth (2007) note, matching is not always optimal. One response to this observation is to propose that suboptimal matching arises as a side effect of some underlying principle of behaviour, such as Thorndike's (1911) ‘law of effect’ which proposes that behaviour shifts towards alternatives that have higher immediate value. Another is given by the notion of ecological rationality—that cognitive mechanisms fit the demands of particular ecological niches and may deliver a predictably suboptimal behaviour when operating outside these niches (Gigerenzer et al. 1999). In line with ecological rationality, Seth (2007) shows that simple decision rules, which lead to optimal foraging in competitive environments with multiple foragers, also lead individual foragers to obey the matching law.

As Houston et al. (2007) discuss, a further possible explanation of apparent irrationality is that we were wrong about what is being optimized. As an example, they consider violations of transitivity, showing that such violations can, in fact, be optimal when decisions are state-dependent and when choice options persist into the future. Another axiom of standard rationality is independence from irrelevant alternatives, the notion that the relative preference of one option over another is unaffected by the inclusion of further options into the choice set. However, as Houston et al. (2007) show, adding a suboptimal option can affect future expectations because, assuming that decision-making errors happen, the suboptimal option is likely to be wrongly chosen in the future. Apparent violations of rationality in human decision making are also discussed by Bogacz et al. (2007, see below).

A related way we can be mistaken about what is optimized is when behaviour reflects compromises among multiple goals. As Crabbe (2007) explains, a compromise action is defined as an action that, while not necessarily best suited to satisfying any goal in isolation, may be the best when all goals are taken into account. For example, a predator stalking two birds might not move directly towards either one of the birds, but in between the two, hedging its bets in case one of the birds elects to fly away. This article provides a review of approaches to compromise behaviour in relevant literatures finding that, while the intuition that compromise is useful is widespread, the data from empirical studies to support this assertion is rather thin. Crabbe (2007) then presents a series of detailed models directly addressing the question of when compromise behaviour is optimal. The models are restricted to simple situations involving both prescriptive (e.g. food) and proscriptive (e.g. danger areas) goals, but are both spatially explicit and open to analytical investigation. The methodology used is distinctive inasmuch as it combines agent-based modelling with a formal analysis of optimal strategies. Perhaps surprisingly, Crabbe (2007) finds that compromise behaviour is usually of little benefit in the scenarios analysed. He suggests the interesting proposal that while this may be true for so-called ‘low-level’ compromise behaviour (e.g. motor actions), it may be less true for ‘high-level’ compromise, in which the compromise is among competing behaviours (e.g. get food and find shelter) that are less easily ‘blended’.

2. Neural substrates for action selection in cortico-basal ganglia loops

A central and largely unsolved problem in the brain sciences is to understand the functional architecture of the vertebrate nervous system. Many questions about this architecture revolve around the issue of action selection. Because it is a fundamental property of neurons to be selective with regard to the patterns of input activity to which they respond, claims that particular brain sub-systems are specifically or preferentially involved in the selection of action, as distinct to other aspects of control, must meet more stringent requirements (see below). It is also by no means inevitable that the functional decomposition of the brain will contain specialist action-selection mechanisms (Seth 2007). Appropriate behavioural switching could be a global property of the functioning of the nervous system, and of its embedding in a body and environment, which cannot be attributed to specific sub-components of brain architecture. In other words, it is plausible that an animal may ‘flip’ from one integrated pattern of behavioural output to another without some identifiable internal ‘switch’ being thrown. On the other hand, theoretical arguments can be presented, based, for instance, on the benefits that accrue from modularity (Wagner & Altenberg 1996; Prescott et al. 1999; Bryson 2005), to suggest that biological control systems may include specialized action-selection components. Hence, one important debate in this field is whether there are specialized mechanisms for action selection in animal nervous systems, and, if so, where these might be found (Prescott 2007, for an evolutionary perspective on this question).

Redgrave et al. (1999) have proposed that, to be considered as a candidate action-selection mechanism, a neural sub-system should exhibit the following properties. First, it should have inputs that carry information about both internal and external cues relevant to decision making. Second, there should be some internal mechanism that allows calculation of the urgency or ‘salience’ that should be attached to each available action. Third, there should be mechanisms that allow for the resolution of conflicts between competing actions based on their relative salience. Finally, the outputs of the system should be configured so as to allow the expression of winning actions while disallowing losers. There is now a growing consensus in the neuroscience literature that the basal ganglia—a group of functionally related structures found in the mid- and forebrain of all vertebrates—meet these criteria and therefore may represent an important neural action-selection substrate.

A key characteristic of the basal ganglia is its connectivity, in the form of multiple parallel loops, with both cortical and sub-cortical systems involved in either the control of movement, planning or the sequencing of thought. Houk et al. (2007) characterize these loops as ‘distributed processing modules’ implementing an action-selection function and contrast them with loops via the cerebellum, whose complementary role is to refine and amplify the neural representation of selected commands. The idea of cortico-basal ganglia loops as a substrate for action selection is a consistent theme in the computational neuroscience models described in this issue (Cisek 2007; Frank et al. 2007; Hazy et al. 2007; Stafford & Gurney 2007). However, there are also important differences in emphasis, and in the precise functional characterization of the role of the basal ganglia, among these contributions. For instance, Frank et al. (2007) and Hazy et al. (2007) share a common starting point in adopting a model of basal ganglia function that stresses the differential role of two intrinsic basal ganglia pathways—one monosynaptic (sometimes termed the ‘direct’ pathway) that inhibits basal ganglia output structures, and another di-synaptic (or ‘indirect’) that has a net-excitatory effect on outputs. According to these authors, the first pathway provides a ‘go’ signal that allows (gates) a desired movement by suppressing basal ganglia inhibition of target structures, while the second pathway is ‘no-go’, preventing the performance of an undesired movement. The same pathways (along with others) are also represented in the Stafford & Gurney (2007) model in this issue; however, these authors emphasize the synergistic operation of intrinsic basal ganglia mechanisms that results in an appropriate balance of basal ganglia output to both winning and losing channels. While the differences between these alternative accounts can seem relatively subtle, the fact that they are fully specified computationally means that they have the capacity to generate different predictions to be investigated and tested against empirical data (Gurney et al. 2004).

A significant focus of current debate is on the balance between intrinsic basal ganglia mechanisms and extrinsic mechanisms elsewhere in the brain in deciding what actions are selected. For Cisek (2007), the selection of visually guided reaching movements within primate fronto-parietal cortex depends largely upon intra-cortical mechanisms, inhibitory and excitatory, with the basal ganglia and prefrontal cortex acting to bias this selection process. Beginning with the traditional distinction between the dorsal (‘where’) and ventral (‘what’) cortical processing streams, Cisek (2007) argues that the dorsal stream contains specialized sub-streams each configured to meet the needs of alternative forms of action. At any one moment several action plans may be active, triggered by the opportunities afforded by the current task/environment context. This view emphasizes that the specification of action, that is the computation of the parameters of movement, can occur alongside the selection of action and possibly within the same neural substrate. Modelling results support this hypothesis that intra-cortical mechanisms may be sufficient to resolve the competition between active action representations to provide a clear winner.

Houk et al. (2007) describe two empirical studies whose inspiration has also been modelling of the neural substrate for action selection in primates. These studies illustrate the interplay between analytical and synthetic approaches in the brain sciences, where biological investigations, predicated on existing models, generate data that match key predictions while raising new questions to be addressed by modellers. In their first study, Houk et al. (2007) use human functional magnetic resonance imaging (fMRI) to show differential patterns of neural processing relating to sensory- and memory-guided movement sequences. Their results show patterns of activity in parts of cortex, basal ganglia and cerebellum consistent with these authors' ‘distributed processing modules’ hypothesis, developed, in part, on the basis of prior modelling investigations. Of particular interest is the evidence of reduced activity in a basal ganglia input area (the caudate), which Houk et al. (2007) propose could be the consequence of presynaptic inhibition—a neuronal selection mechanism that they then demonstrate, using a computational model, to be both effective and energy-efficient. In their second study, an electrophysiological investigation using monkeys, Houk et al. (2007) show that the basal ganglia act to select both the primary movement involved in orienting to a target and the much smaller corrective movements required to precisely locate that target. Interestingly, changes in the basal ganglia output were found to be of similar amplitude for both types of movement. This result suggests that in well-rehearsed tasks, the selection of primary movements may be, in part, ‘exported’ outside the basal ganglia (e.g. to cortex), while corrective movements, which cannot be predicted or learned, remain under full basal ganglia control.

As larger-scale models are developed that encompass both cortical and sub-cortical mechanisms (e.g. Frank et al. 2007; Hazy et al. (2007)), the dynamics of the interactions between these selective processes at different levels of the neuraxis (nervous system architecture) will become better understood. For instance, in mammals, and particularly in primates, basal ganglia loops involving the prefrontal cortex (PFC) have long been identified with a general role in executive function (e.g. Miller & Cohen 2001). For Hazy et al. (2007), loops through PFC provide modulation of working memory representations by taking advantage of a similar circuitry to that which, in more posterior loops, allows the selection of actions. These authors argue that the microanatomy of the PFC suggests the presence of multiple (perhaps 20 000 in humans) local neuronal groups, or ‘stripes’, that could provide the substrate for encoding independent working memories. In their recent computational model (O'Reilly & Frank 2006), intrinsic, recurrent excitatory connections within each simulated stripe allow active representations to be maintained, while disinhibition, via a basal ganglia loop, can bring about the rapid updating of a specific representation. Learning occurs as the result of a dopamine-based reinforcement learning rule. In their present article, Hazy et al. (2007) propose two new extensions to this architecture. First, that basal ganglia loops involving the deeper layers of the PFC allow gating of the effects of working memory representations on processing elsewhere in the brain, and, second, that phasic noradrenaline release plays a role in regulating the balance between exploitation of learnt strategies and exploration of new alternatives. To validate their model, Hazy et al. (2007) have adopted the strategy of attempting to replicate, in the same basic model, empirical findings from a wide range of working memory tasks. Data pertaining to each new phenomenon to be captured provide additional constraints and thus progressively more stringent tests of the model and its underlying functional hypotheses. This validation approach can be seen as a general strategy for evaluating and extending computational models in cognitive science (Newell 1990; Gurney et al. 2004).

3. The decomposition of control for behavioural sequencing

The problem of action selection is often not a matter of making the best one-off decision, but of generating appropriate behavioural sequences whose net consequence is to achieve some desirable, but longer-term outcome. In ethology (Seth 2007), neuroscience and psychology (Botvinick 2007), robotics (Bryson 2000; Crabbe 2007) and machine learning (e.g. Barto & Mahadevan 2003), it has often been argued that behavioural sequences have a natural hierarchical decomposition. In other words, action selection is best organized to take place at multiple levels of abstraction from choosing among high-level objectives (e.g. whether to eat, drink or rest) through to selecting among alternative movements that could serve the same specific, immediate goal (e.g. which grasp to use in picking up a cup).

Botvinick (2007) presents an alternative perspective on this topic by arguing that behaviour that might appear to be hierarchically organized might be best implemented by a processing framework that does not have a hierarchical decomposition. To motivate this proposal, Botvinick (2007) describes a connectionist model of sequence learning based on Elman's (1990) recurrent neural network architecture. When required to learn the task of ‘making a cup of coffee’—which can be decomposed as a series of sub-tasks each containing a number of elementary actions—the network achieves this goal without representing the hierarchical nature of the problem in an explicit way. Botvinick (2007) argues that this non-hierarchical solution shows a context sensitivity and ability to take advantage of structural overlap between tasks that is also characteristic of natural sequential behaviour.

As an existence proof of sequential learning without hierarchical decomposition, Botvinick's model tells us that the brain could solve such problems in a similar way, but is there any evidence that it actually does so? On first examination, the neuroanatomical data, as summarized by Fuster's (1997) description of functional relationships between cortical regions, seem to point towards an explicitly hierarchical solution. Botvinick (2007) warns, however, against assuming that an anatomical hierarchy implies a functional one. In his model, adding intermediate layers of processing leads to a functional architecture in which higher levels preferentially encode broader temporal contexts, but in which some aspects of context are represented at all levels. This account thus may offer a reconciliation of hierarchical and non-hierarchical views, in which multiple layers of processing facilitate learning of structured tasks, while distributed representations at each level capture relevant context in a manner that allows more flexible expression of behaviour.

4. Sub-cortical substrates for action selection

The cortico-basal ganglia loops that are the focus of much of the research described above are known to exist alongside loops connecting the basal ganglia to a wide range of sub-cortical structures involved in sensorimotor coordination (McHaffie et al. 2005). The evolution of mammals saw a substantial increase in the role of the forebrain in action specification and control (Butler & Hodos 1996), largely supplementing, rather than replacing, the sensorimotor functionality of these systems lower down the neuraxis. This complex layered architecture provides multiple levels of competence (Prescott et al. 1999) including the option to choose between sub-cortical systems that provide a rapid response to immediate contingencies and cortical systems that provide sophisticated adjudication between alternatives, taking greater account of context, past experiences and the probable future opportunities (McHaffie et al. 2005). Key structures such as the subthalamic nucleus, which appears to provide a global, inhibitory ‘hold your horses’ signal (Frank et al. 2007), may have evolved, at least in part, in order to regulate this trade-off between speed and sophistication of action selection.

Evidence from decerebrate and altricial animals, and from surgical interventions in patients with Parkinson's disease, suggests the presence of a further substrate for action selection. Humphries et al. (2007) argue that a centralized brainstem structure, the medial reticular formation (mRF), fits the Redgrave et al. (1999) criteria for an action-selection mechanism. The identification of the mRF with a possible action-selection role has a long history that includes one of the first, if not the first, computational neuroscience models (Kilmer et al. 1969). Based upon the most recent anatomical data, Humphries et al. (2006) have described a structural model of the mRF intrinsic architecture which suggests that it is configured at the neuronal level as a ‘small-world network’ (Watts & Strogatz 1998)—implying properties such as rapid cross-network synchronization, consistent stabilization and persistent activity that could be useful for the representation and resolution of action-selection competitions. In their present article, Humphries et al. (2007) further show that the intrinsic organization of the mRF connectivity may have evolved to minimize connection length for a characteristic pattern of network configuration that can be described as a series of stacked ‘clusters’. These authors then set out a computational model that explores alternative hypotheses concerning the possible functional organization of the mRF, concluding that an architecture in which the output of each cluster represents a sub-component of a complete behaviour appears most consistent with available evidence. In other words, this model suggests that the co-activation of a set of mRF clusters would correspond with the expression of a coordinated behavioural response by the animal. The relationship between selection systems in the mRF and basal ganglia remains to be worked out. One possibility is that the mRF coordinates extended patterns of behaviour that can be selected in toto at the level of the basal ganglia. Another is that the mRF forms a parallel substrate for action selection, in keeping with the notion of layered control, such that damage to forebrain systems for action selection leaves intact a brain-stem network that is capable of preventing the most damaging forms of behavioural disintegration.

5. Disorders of action selection—clinical implications of action selection modelling

As the above discussion suggests, modelling approaches to the understanding of natural action selection can give useful insights in circumstances where the normal flow of integrated behaviour becomes disrupted as a consequence of damage or disease. A cluster of neurological conditions that includes Parkinson's disease (PD), schizophrenia, Huntington's disease, Tourette's syndrome, attention deficit/hyperactivity disorder (ADHD) and obsessive–compulsive disorder, can be linked to the same cortico-basal ganglia circuits that have been identified above as key substrates for the selection of action. Computational models of these substrates therefore have the potential to provide improved explanations for how these disorders arise and why particular patterns of symptoms are observed, and to serve as vehicles by which to investigate possible avenues for treatment.

Building on a broad platform of earlier modelling work investigating motor control and decision making in intact basal ganglia loops, Frank et al. (2007) explore the functional consequences of model manipulations that simulate specific types of brain damage or abnormality thought to underlie PD, decision making deficits in patients with frontal cortex damage, and ADHD. For each condition, their model is able to explain much of the observed symptomatology and also to suggest novel predictions that can be tested at the neural or behavioural level. For instance, with respect to PD, Frank et al. (2007) present a unifying computational hypothesis—reduced dynamic range in dopaminergic modulation of the basal ganglia—that explains a range of motor and cognitive deficits seen in this disease. On the basis of their model, Frank et al. (2007) were also able to make predictions about the effects of dopamine medications on learning that were later verified experimentally, and to make suggestions about the probable impact of deep brain stimulation (a surgical treatment for PD) on cognitive processes. These authors further propose that by providing an improved characterization of the function of different circuit components (e.g. the various intrinsic basal ganglia pathways), computational modelling could facilitate the development of targeted treatments for the motor deficits seen in PD that have fewer negative side effects on patients' cognitive and emotional capabilities. The aetiology of ADHD is poorly characterized when compared with that of PD; nevertheless, for such conditions, modelling can serve as a powerful aid to theory development. For instance, by investigating the consequences, in terms of model behaviour, of candidate causal factors, Frank et al. (2007) show that a cluster of symptoms associated with ADHD could result from a combination of reduced dopaminergic modulation of the basal ganglia and abnormal patterns of noradrenaline release in cortex.

The relevance of computational neuroscience models for understanding brain disorders involving action selection is also considered by Houk et al. (2007), focusing particularly on schizophrenia. Like Frank et al. (2007), these authors emphasize that success in understanding these complex neurological disorders will most likely require models at multiple levels of analysis from biophysically detailed models of cellular processes through to system-level models, of the type we have focused on in this issue, that can link neurophysiological systems to observable behaviour.

6. Perceptual selection in decision-making

Action selection can be mediated not only by motor control systems but also by perceptual systems. For example, mechanisms of selective attention (Posner 1980) can guide action selection by linking a specific motor output to one stimulus among a range of stimuli. The issue of perceptual selection is raised by a number of articles in this issue, including Bogacz et al. (2007) and Stafford & Gurney (2007). Consider the problem of detecting coherent motion in a cloud of otherwise randomly moving dots. A popular solution to this problem is provided by the leaky competing accumulator (LCA) model (Usher & McClelland 2001), which proposes that during decision making, noisy evidence supporting each of a range of alternatives is accumulated. A significant feature of this solution is that it can be shown to be optimal, i.e. under certain conditions the LCA model performs as well as is theoretically possible (Usher & McClelland 2001). Bogacz et al. (2007) review various neural implementations of the LCA model from the perspective of optimality, and describe extensions to this work which show that non-linear neuronal mechanisms can perform better than linear mechanisms in terms of speed of decision-making between multiple alternatives. This result raises the interesting hypothesis that non-linearities in neuronal response functions may have evolved, at least in part, as a result of selective pressures favouring rapid decision-making.

Bogacz et al. (2007) next extend the LCA framework beyond perceptual selection to account for the so-called ‘value-based’ decisions in which alternatives are compared on the basis of their match to a set of internal motivations as well as to sensory signals. They discuss two examples of apparent irrationality (see above and Houston et al. 2007): risk-aversion, where humans and animals prefer the less risky of the two alternatives that are equated for expected value, and preference reversal, where the preference order between the two alternatives can be reversed by the introduction of a third, irrelevant, choice option. The LCA model can account for these phenomena given a non-linear utility function which is applied to the difference in value between each alternative and a ‘referent’, which may correspond to the present (or expected) state of the decision maker.

Stafford & Gurney (2007) focus on the Stroop task, in which subjects have to name the ink colour of letter strings that spell out the name of a (congruent or incongruent) colour. Previously, Cohen et al. (1990) showed that a model which used a response mechanism similar to the LCA model just described could account for the basic Stroop phenomenon. Stafford & Gurney (2007) show that by introducing a response-selection method based on a previous model of the basal ganglia (Gurney et al. 2001), additional empirical data can be accounted for, making certain cognitive interpretations of the Stroop result (e.g. speed of processing) less probable. This study highlights the notion that the neural mechanisms of selection can apply equally to both perceptual and motor domains. It also shows how models incorporating relevant neuroanatomy can usefully extend those based primarily on computational principles.

7. The units of selection

The impact of perception on action selection highlights the problem of specifying precisely what a ‘selection’ mechanism selects. As noted above, for Cisek (2007), visual processing helps to specify potential actions and these actions compete against each other for further processing in a competition that is continually biased by the ongoing sensory input: action selection and action specification occur concurrently. Cisek (2007) argues that the functional architecture for behaviour might then be best understood as a set of competing, nested, sensorimotor loops, drawing parallels to some classic notions from ethology and to more recent insights from synthetic approaches such as behaviour-based robotics (Brooks 1991) and active perception (e.g. Ballard et al. 1995). Along similar lines, Seth (2007) argues that there has been significant confusion in the ‘artificial life’ literature as to the units of selection. Although there is considerable consensus that perception is integral to the ‘actions’ that are selected, the behaviour categories that seem apparent to an external observer may have no correlates within the control mechanism itself. What the observer terms action-selection behaviour arises from the interaction of agent-side mechanisms with the environment. When this interaction is subsequently categorized by an external observer, what appear to be multiple different actions may actually emerge from a single internal mechanism. This can lead to an overly complex assessment of the agent's own capacities (Simon 1969). To illustrate this argument, Seth (2007) describes a simple model in which action selection arises from the joint activity of parallel, loosely coupled sensorimotor processes.

8. Action selection in social contexts

In nature, action selection is rarely purely an individual matter; rather, adaptive action selection usually involves a social context. As mentioned above, Seth (2007) shows that the apparently irrational behaviour described by Herrnstein's (1961) ‘matching law’ can, in fact, be optimal in a semi-social context. This result illustrates that action-selection behaviour that is adaptive in an environment that includes competing conspecifics may be ‘irrational’ in an isolated individual.

It is important to note that in Seth's model the individuals are not explicitly societal—there are no direct costs or benefits associated with social interactions. Agents in the model only interact indirectly via their effects on resource distribution in the environment. Bryson et al. (2007) investigate a similarly minimalist model of factors that influence the troop structure of primate species. The organization of primate societies has been characterized as ranging between egalitarian and despotic extremes (van Schaik 1989; Thierry et al. 2004). Using an agent-based model, encompassing both individual motion and social conflict, Hemelrijk (2000, 2002, 2004) has sought to account for this observed variation in social organization by manipulating a single variable—the level of violence in aggressive encounters. In their present article, Bryson et al. (2007) replicate this model, and attempt to validate it against data from a well-studied set of closely related primate species, the macaques (genus Macaca). In doing so, they find several discrepancies between the model and their target systems, that lead them to, at least partially, reject the Hemelrijk model as a hypothesis of macaque behaviour. On the other hand, they also show that when the model is altered, in a way that takes into account their critique, some of Hemelrijk's original findings still hold. Bryson et al. (2007) therefore make both a contribution to the theory of macaque social behaviour and a methodological point by demonstrating the accessibility and robustness of simulation-based theory building.

Simulations have also been used to account for more precise quantitative descriptions of social behaviour. Pratt et al. (2005) have recently demonstrated a model of a hypothesis of how one species of ant, Temnothorax albipennis, determines when and where to move a colony to a new nest. Here, a highly distributed algorithm with no central arbiter reliably results in a single nest site being chosen which is likely to be optimal for a weighted consideration of a set of criteria (e.g. capacity of nest and defendability of entrance). The action selection of the individuals affects the action selection of the colony. In this issue, Sellers et al. (2007) look to account for group action selection for navigation and resource consumption on a much larger scale. They produce a simulation mapping the set of habitats in the precise range of a particular troop of baboons (Papio hamadryas ursinus) and then try to replicate their annual patterns of foraging. The primary goal of this model is to determine the mechanism by which the troop as a whole chooses a direction to go in, given that this must reflect the needs of individuals in the troop (which cannot, of course, be fully negotiated or understood by the other animals). The assumption is that there is essentially a ‘voting mechanism’ where, if a critical number of agents move in one direction, the entire troop will follow. The model is evaluated by comparing the amount of time the real and virtual troops spend in various foraging environments at various times of the year. As might be expected from such an ambitious model, the outcome does not perfectly match the real data, but there is sufficient commonality that where differences exist, they motivate topics of research for field biologists to attempt to explain. For example, is there an undiscovered disadvantage to foraging near a vlei (African wetland)? Or would another model of collective action arbitration better capture the data?

Finally, Laver & Schilperoord (2007) examine an instance of social action selection for which there is a fairly well-understood centralized selection mechanism—voting in a democratic society. Yet, although in this case we know how the final social arbitration is determined, we neither know how individual voters choose their parties nor how parties choose their platforms. These authors start from a standard political science assumption that a multidimensional space can be defined by both voter and party positions along issue vectors, and that voters will vote for the party nearest them in that space. They then examine the way parties develop new positions, modelling and hypothesizing four ‘species’ of party: parties that seek ‘open’ positions, parties that take the average position of their voters, parties that ‘stick’ to their platforms and parties that chase the positions of more successful parties. In a previous work, Laver (2005) successfully modelled the trajectories of voting and position histories of Irish political parties. In their present article, Laver & Schilperoord (2007) extend this earlier model to encompass party birth and death and to model citizen discontentment. Their results include an explanation for situations in which parties tend to migrate towards the centre of the issue space, and how this increases the average level of citizen discontentment with their representation.

9. Modelling strategies

As is clear from the above discussion, the contributions within this theme issue make use of a wide range of modelling strategies. At one end of the spectrum, Houston et al. (2007) present analytical mathematical models in support of their arguments. While Bogacz et al. (2007) also use mathematical formalisms, they do so in order to analyse a series of simple neuronal network models from a normative perspective. A similar strategy is pursued by Crabbe (2007), with the distinction that normative analysis is applied to a formalized agent-based model, in which sensory data is transformed into motor output, and not to idealized neural network models. Instead of using mathematical formalisms to establish optimality, Seth (2007) applies numerical optimization techniques (e.g. genetic algorithms; Mitchell 1997) to agent-based models. Although genetic algorithms cannot guarantee optimality, they nevertheless allow model behaviour to be interpreted from a normative perspective. A practical advantage of this approach is that models can incorporate subtle but significant agent–environment interactions; conceptually, such models help to bridge the divide between agent-based modelling and optimality modelling.

Particularly well represented in the present issue are mechanistic computational neuroscience models of the neural substrates of action selection. A primary advantage of mechanistic modelling is that neurobiological detail can be incorporated to an arbitrary degree, allowing specific predictions to be made with regard to empirical data (Dayan & Abbott 2001; Gurney et al. 2004). The models included in this issue are mostly at a relatively high level of abstraction (e.g. Humphries et al. 2006, 2007; Cisek 2007; Frank et al. 2007; Hazy et al. 2007; Stafford & Gurney 2007) in that they incorporate constraints such as the gross functional anatomical organization between and within brain regions, and are typically composed of relatively simple ‘leaky-integrator’ style units whose operation often stands for that of relatively large neuronal ensembles. It is important to note the limitations of such models in that many, possibly critical, characteristics of real neural processing are passed over at this level. The contribution of Houk et al. (2007) serves to illustrate this problem by showing that a relatively low-level detail—the location of synaptic contacts on basal ganglia input neurons—can have significant implications for network function. However, neurobiological specificity is not necessarily required for mechanistic models to make useful contributions. The connectionist model described by Botvinick (2007) demonstrates that significant insights, in this case into the utility of hierarchical organization, can be obtained by exploring the functional capabilities of neural-like processing architectures that are not intended to directly mimic specific structural features of the brain.

Agent-based modelling, which is used by both Crabbe (2007) and Seth (2007) to investigate normative models of action selection, is put to a different but equally productive use in the articles concerned with action selection in social groups. Bryson et al. (2007) and Sellers et al. (2007) focus on incorporating biological details into agent-based models in order to generate specific predictions. These authors view their models as well-specified scientific hypotheses that can be tested against empirical data using standard statistical procedures. Laver & Schilperoord (2007) demonstrate both the generality and the broadening acceptance of agent-based modelling by showing its current standard of use in the social sciences. Note that, here, though the spatial aspects of agent-based modelling have become metaphorical, the outcome is still a substantive quantitative result.

The methodological practice of sharing research resources can benefit not only the wider scientific community but also the contributors of the resource (King 1995; Gleditsch et al. 2003). For this theme issue most of the models of natural action selection are provided, in source code form or similar, as electronic supplements to their respective articles. We are grateful to the authors for their enthusiastic support of this methodological research standard and urge interested readers to use this material to replicate, evaluate and extend the work presented here.

10. Conclusion

The scientific study of action selection integrates a broad range of topics including, but not limited to, neuroscience, psychology, ecology, ethology and even political science. These domains have in common a complexity that invites the development and deployment of advanced modelling techniques drawn from computer science, artificial intelligence and artificial life. Our goal for this special issue has been to collect and showcase some of the most exciting research in the biological sciences currently being performed using action-selection modelling techniques. As we have seen, this research promises to illuminate many important questions such as: why animals, including humans, sometimes act irrationally; how damage to neural selection substrates can lead to debilitating neurological disorders; and how action selection at the level of the individual impacts on the organization of societies. In addition, this theme issue has sought to foster a cross-disciplinary fertilization of concepts, tools and modelling techniques. We hope that its lasting impact will be reflected not only in an improved scientific understanding of behaviour but also in an enhanced methodological tool suite for the biological sciences.

Acknowledgements

This theme issue arose as the result of a workshop on ‘Modelling Natural Action Selection’ held in Edinburgh, in July 2005, part-sponsored by UK Biotechnology and Biological Sciences Research Council. We are grateful to all the participants in that workshop, and to the reviewers, of both the workshop papers and the theme issue articles, whose multidisciplinary input has helped to refine the contributions presented here. We also gratefully acknowledge the financial support of the UK Engineering and Physical Sciences Research Councils (grants GR/R95722/01 and GR/S79299/01) and of the US Neurosciences Research Foundation.

Footnotes

  • One contribution of 15 to a Theme Issue ‘Modelling natural action selection’.

    References

    View Abstract