The principles of goal-directed decision-making: from neural mechanisms to computation and robotics

1. Introduction

The brain has evolved to act in a complex and unpredictable world, and it must continuously select among many goals and action options. In the past 20 years, research in neuroscience and robotics has principally focused on the neural underpinnings of habitual choice, linked to stimulus–response mechanisms in restricted contexts. However, a recent trend of research is to focus on goal-directed choice mechanisms that select among anticipated action outcomes in a context-dependent way. These goal-oriented processes provide the foundations of many everyday human and animal decisions.

Goal-directed decision-making relies on multiple sources of information beyond immediately available sensory stimuli, such as representations of (proximal and distal) goal states and their utility, memory and predicted future states. Furthermore, it has to take into consideration the specific behavioural context in which a decision is to be made rather than just activate habits. Considering these elements—all of which are hallmarks of the flexible goal-directed mechanisms of choice—requires expanding the scope of current research on decision-making in living and artificial organisms such as robots. We need to understand the specific contributions of goals, episodic memory, working memory and cognitive control to decisions, and to analyse the neural substrates of these brain processes and how they link to flexible forms of computation (e.g. model-based methods of reinforcement learning, which consider specific action outcomes and not only stimulus–response pairs). We also need to link these neural substrates to psychological processes, and understand if and how currently leading models in decision-making based on the accumulation of (sensory or reward) evidence scale up to forms of choice that are more flexible and far-sighted than those afforded by habitual mechanisms, and are also closer to those deployed under real-world conditions (e.g. everyday human decisions, choices concerning distal outcomes). Finally, we need to understand how the brain orchestrates multiple processes and sources of information coherently (e.g. considering the costs and benefits of distal outcomes, selecting the right contextual rules, overriding prepotent responses) in its pursuit of goals. This search may be facilitated by using an interdisciplinary approach, with a combination of experimental, computational and robotic research to explore how goal-directed decisions and actions are deployed in ecological contexts.

A key hypothesis motivating this Special Issue is that the study of the neuro-computational architecture of goal-directed choice requires a systems-level perspective on brain and behaviour. This is necessary because, we argue, goals have a truly integrative function in decision-making and cognition. Goals are multifarious constructs that include at minimum affective and motivational components along with predictive state representations that guide action specification and selection [1–7]. Because different aspects of goals (e.g. motivational, sensory and motor) are probably represented in distributed brain networks they can only be studied at the level of systems—asking for example how more primitive drive systems connect to abstract cognitive goals used in our everyday decisions—as opposed to focusing on a single component (say, a value or reward signal). Furthermore, in the proposed goal-centred perspective, cognitive processing does not consist of a series of reactions to external stimuli but is first and foremost organized around internally formulated goals, with cascading effects on cognition and behaviour at large. During choice, cognitive processing is orchestrated to serve the identification and selection of goals, including perception and attention (to goal- and task-relevant aspects of the sensorium), memory (to retrieve goal-related information), prediction and valuation (of action outcomes), selection and monitoring (of actions that achieve goals while also caring about effort).

The ambition of this Special Issue is to offer a fresh perspective on the vibrating research domain of goal-directed choice, and to stimulate future interdisciplinary research towards the formulation of an integrative, systems-level proposal on goal-directed cognition and its neuronal substrate. Most current research in neuroscience, psychology and artificial systems does not sufficiently recognize the importance of goals in how the brain mediates perception, cognition and action. This Special Issue is a representative of a new breed of studies that not only ask how the brain learns to link sensory stimuli to stereotyped actions (habitual system), but also ask how it integrates multiple information streams (e.g. internal needs and external opportunities) and coordinates them in time to formulate, maintain and achieve goals. Multiple brain areas contribute to goal-directed choice—and this Special Issue will highlight the functioning of prefrontal, hippocampal, amygdaloid and basal ganglia networks—and it is important to fully understand how their contributions are orchestrated in ecologically valid scenarios.

Progress in the field can only emanate from the realization that multiple disciplines can contribute to the understanding of the multifaceted phenomenon of goal-directed choice. The Special Issue thus brings together neuroscientists, psychologists, computational modellers, behavioural ecologists and roboticists to present recent results and theoretical insights in the most relevant research directions in the study of goal-directed choice—including its neuronal underpinnings in the brain of humans, monkeys and rodents—its functioning and malfunctioning in health and disease, its underlying computational principles and its validity in real-world robotic and ecologic settings. The Special Issue is strongly multidisciplinary reflecting the importance of interaction between empirical, computational and robotic methods in the study of goal-directed choice and the integrative functions of the nervous system in general. Up to now the required synergies between the fields were not as strong as they could and should be. Indeed, in the study of goal-directed choice, the scientific community has proceeded in two diverging directions. Experimental neuroscientists proceeded from the bottom-up, that is, from the analysis of the neuronal circuits that permit humans to make decisions to the linkage of neuronal dynamics with possible optimization principles. At the same time, theoretical neuroscientists and modellers have often proceeded from the top-down, deriving principles from normative theories (e.g. model-based reinforcement learning) and applying them to the study of neuronal mechanisms. Now these two paths are converging and consensus is growing on how neuronal circuits for goal-directed choice link to specific computations and learning mechanisms. The tight linkage between theoretical/computational and experimental approaches is already bringing major advancements in the field. The logical next step is to accomplish a better integration of experimental and computational methods with real-world robotic systems, which will permit a validation of the proposed decision-making approaches in ecologically valid scenarios [8–12].

The dual goal of this Special Issue is to foster an interdisciplinary debate and to offer initial proposals for the realization of a much-needed unitary goal-centred framework for cognition. It reports the most recent advancements in the field that come from a variety of perspectives, including multiple empirical methods (from single cell recordings to neuroimaging) and formal approaches (biologically motivated machine learning, computational modelling and robotics). For the sake of clarity, we have grouped the contributions in three main topics: ‘Topic A: Brain circuits for goal-directed decision-making in humans, monkeys, and rodents; Topic B: Formal and computational approaches to goal-directed decision-making; and Topic C: Decisions in ecological and robotic contexts’.

2. Brain circuits for goal-directed decision-making in humans, monkeys and rodents

Recent research in neuroscience has revealed that several brain areas contribute to goal-directed decision-making, but a unified approach of how the brain supports it is not yet available. Some outstanding questions are: how are actions, goals and values (i.e. computational results predictive of future outcomes) represented in the brain, and how are these representations recruited to produce goal-directed choice? Given that the brain appears to represent value in multiple ways (e.g. in the amygdala, prefrontal cortex, ventral and dorsal striatum) how do these different systems cooperate to inform choices? What are the most important brain circuits for goal-directed choice and how are they linked? How are different kinds of information (e.g. sensory, memory, prediction and value) coming from multiple sources integrated to produce goal-directed choice? What memory mechanisms (e.g. episodic) are involved, and what are their neuronal underpinnings? Realistic goal-directed choices require cognitive control and the ability to evaluate uncertainty, balance exploration against exploitation and monitoring the ongoing performance; how are these functions (typically attributed to prefrontal cortex, but not exclusively) integrated in a coherent neuronal architecture? and how are these (and other) ‘executive functions’ orchestrated? Can we find integrated principles of functioning of goal-directed decision and action?

Six contributors to this Special Issue address these and related questions. The contributions target key ‘players’ of goal-directed choice in the brain of mammals and rodents—most prominently prefrontal cortex, striatal systems and hippocampus. Not only do they present or review data but they also have a strong theoretical flavour that resonates with the general aim of the Special Issue to seek the general (brain and computational) principles of goal-directedness.

A first issue here is how goal-directed behaviour and cognitive control are neurally implemented and organized. Buschman & Miller [13] present a theory of top-down control that uses two complementary (and interacting) systems: one quick-and-concrete that is primarily based on basal ganglia circuits, and another gradual-but-abstract that is primarily based on the prefrontal cortex. The paper presents neurobiological evidence supporting the coexistence and integration of the two systems and explains how their balance explains key aspects of goal-directed learning and habit formation. Koechlin [14] tackles the same problem of the balance of different learning and control systems from a slightly different angle. He presents empirical and computational arguments supporting the idea that the prefrontal cortex of rodents and mammals gradually adds new inferential (Bayesian) capabilities—and in particular hypothesis-testing abilities—to more primitive reinforcement learning systems, thus realizing an integrated ‘strategy creation’ system. Genovesio & Ferraina [15] explore in more detail the ways the prefrontal cortex maintains and monitors goals—not only those currently pursued but also those arising from previous choices that are actively maintained. It emerges from this framework that the frontal lobe actively tracks goal constructs—past, present and future—to support proactive and reactive control, and for learning purposes. Lisman [16] investigates the selection-and-learning problems faced by the striatum and presents a two-phase model in which it firstly biases multiple actions for their selection in cortex and successively uses an efference copy for the reinforcement of the selected action only. The model predicts that the motor system operates discontinuously, and the empirical validity of this prediction is assessed here. Taken together, these four contributions provide a broad view of how different learning and control systems coexist in the brain that support habitual versus goal-directed systems (a recurrent theme that we will also see below among formal models) and how goal constructs in prefrontal and striatal systems support cognitive control, executive function and action selection.

A second issue concerns the valuation side of goal-directed choice. Stott & Redish [17] examine the specific role of two brain areas, orbitofrontal cortex (OFC) and ventral striatum (vStr), both of which have been implied in coding value information. The data reported here from a rodent delay-discounting task support a differential engagement of vStr in action selection and OFC in post-decision processing, including outcome monitoring and learning. Larsen & O'Doherty [18] use a combined electroencephalogram-functional magnetic resonance imaging approach to study the temporal aspects of valuation and choice in humans. The results speak to the time course of engagement of different brain areas, with an initial recruitment of posterior cortical areas, such as intraparietal sulcus, and a successive shift during choice processes to more anterior areas: ventromedial and lateral prefrontal cortex. The fact that additional signals emerged later in time in dorsomedial prefrontal cortex suggests that this area might support post-decision action-selection rather than decision per se. These two studies, using different techniques, are illuminating on the temporal dynamics of decision-making in the brain, suggesting a distributed architecture for valuation that supports selection, monitoring and learning purposes in a highly coordinated way.

Taken together, the six contributions to this first topic give a comprehensive view of the systems-level brain architecture supporting goal-directed choice, analysing its multifarious aspects that link to cognitive control, selection, valuation, monitoring and learning among others. The emerging picture is that of a neuronal architecture that is widely distributed and includes some prominent ‘players’ that continuously interact in making choices. All the contributions motivate their findings within an increasingly detailed framework for goal-directed cognition that begin to incorporate detailed functional and anatomical hypotheses. The next contributions to the Special Issue take a complementary perspective and start instead from well-established formal frameworks to shed light on the neural architecture of goal-directed choice from a top-down perspective.

3. Formal and computational approaches to goal-directed decision-making

There is nowadays a strong tradition—especially in neuroeconomics and perceptual decision-making—of coupling neuroimaging and neurophysiological studies with formal models that explain the neural computations required for goal-directed choice and the associated behavioural patterns. Some prominent examples are model-based reinforcement learning, Bayesian decision theory and hierarchical reinforcement learning [19–24]. Nevertheless, the study of the computational aspects of goal-directedness is less mature than the study of stimulus–response learning and habitual mechanisms choice [25]. Furthermore, we still lack integrated approaches that span several levels (i.e. from neural computation to behaviour and decision-making in laboratories as well as in realistic and ecological settings) and illuminate the similarities and differences between simpler and more complex choices (i.e. from simple perceptual decisions to abstract decisions). Some outstanding research questions are: what constitutes goals? What formal or computational schemes are more adequate to explain goal-directed choice and the planning of action sequences? How do they map to neuronal circuits? What are the relations between habitual and goal-directed mechanisms of choice? Can we find ‘optimality’ principles in goal-directed choice and map them to neuronal computations?

Six contributions to this Special Issue address these and related questions. The contributions present a variety of computational-level ideas on goal-directed choices and establish links with the empirical literature to propose how they might be neurally realized.

A first issue here concerns the computational principles guiding goal-directed choice, and their neuronal implementation. Daw & Dayan [26] present a comprehensive and accessible view of goal-directed choice in terms of model-based reasoning (as opposed to model-free computations associated instead with habits). It explores in detail the challenges of model-based computations (e.g. their computational complexity) and advances several proposals on how the brain might solve them, for example by combining model-based and model-free methods or using one or more approximations, also pointing to existing evidence on the neuronal implementation of the proposed computations. Friston et al. [27] explain goal-directed decision-making within the ‘active inference’ framework. They discuss how this normative framework can assimilate several notions in neuroeconomics and computational theory such as expected utility and the exploration–exploitation dilemma. Moreover, the paper presents a novel perspective on dopaminergic responses in terms of changes in precision (inverse variance of the probability distributions implied in active inference) discussing how it relates to other widespread ideas in the field (e.g. the idea that dopamine encodes reward prediction errors).

A second issue concerns the architecture of hierarchical and sequential behaviour that affords multistep goal-directed actions. Botvinick & Weinstein [28] discuss abstraction in goal-directed choice in terms of hierarchical model-based computations. They discuss how these methods permit the prospective evaluation of potential action outcomes to proceed in a ‘saltatory’ way; that is, how they afford a form of planning at a quite abstract level (e.g. in terms of macro-actions or skills such as ‘log into email account’) thus disregarding the exact details of action at least initially, potentially extending the scope and complexity of human reasoning. Kachergis et al. [29] propose a model for sequential behaviour. The model uses principles from a biology-based computational model (Leabra, [30]) to implement the constructs of a psychological theory of ideomotor action (TEC, [31]), demonstrating how it can work in hierarchically structured tasks such as coffee making. Dezfouli et al. [32] offer an accessible discussion of the differences between goal-directed and habitual action from both a theoretical and an empirical perspective. They thus propose a hierarchical view of control that explains the interactions between these two (goal-directed and habitual) modes of control; for example, how outcome devaluation, one marker of habits, can occur as a by-product of hierarchical decision-making.

At still another side of the decision-making spectrum, Misyak & Chater [33] offer a game-theoretic model of social choice: a ‘virtual bargaining’ perspective that formally describes how humans might solve fundamental social dilemmas and coordination problems, and which also might be the basis for more complex social interactions such as joint actions and communication.

Taken together, these contributions illustrate the illuminating role of formal frameworks in the study of goal-directed choice, especially when bidirectional interactions and cross-fertilizations are established between these frameworks and empirical research. Much current empirical research is conducted in restricted laboratory settings and current formal models tend to focus on a restricted and well-controlled number of elements and parameters. The next contributions we review take a different perspective and focus instead on goal-directed choice in ecologically valid settings from empirical and robotic perspectives.

4. Decisions in ecologic and robotic contexts

Most models of goal-directed choice have been developed for simple situations (e.g. with only one source of evidence, and with clearly specified behavioural alternatives). However, animals (including humans) make complex decisions outside the laboratory; and robots are increasingly facing similar problems. A key challenge is extending the scope of current models of goal-directed choice to more complex and ecologically valid situations, for example when outcomes are distal, and understanding how models developed for simple choices fail in complex contexts. Important insights can come from both experimental research on humans and non-human animals, and biologically grounded robot studies in ecologically valid settings that allow principles of neuronal computation to be evaluated in the context of real-world behaviour. Some outstanding research questions are: how are complex and abstract goals represented and selected? What are the planning mechanisms that permit achieving distal goals? How do animals choose their actions in ecologically valid contexts, in social and in potentially risky environments? What are the state-of-the-art methods in robot planning and choice and how can they help us understanding natural cognition? Can robotic models bridge the knowledge gap between neuronal dynamics and goal-directed choice in ecologically realistic scenarios (e.g. in foraging)?

Five contributions to this Special Issue address these and related questions. The contributions come from a variety of perspectives and disciplines—neuroscience, robotics, psychology and ethology—reflecting the shared interest for ecological scenarios but also the importance of multidisciplinary research in the field.

A first issue here concerns the characteristics of embodied and situated choices (e.g. choosing between fighting or fleeing for a hungry predator) as contrasted to the more abstract choices used in laboratory studies (e.g. pressing one of two buttons in response to stimuli shown on a computer screen), and whether the constraints from the former continue to be used in the latter. Cisek & Pastor-Bernier [34] discuss the challenges of embodied decision-making faced by animals interacting with their environment in real time. The evidence they review points towards an embodied architecture in which multiple processes of action specification, selection and execution all run in parallel and sensorimotor processes are part and parcel of the decision-making process rather than ways to report choices made elsewhere. This paper presents a challenge to neuroeconomic theories that completely detach decision from sensorimotor systems. Verschure et al. [35] consider the challenges of ecological choice from the perspective of a situated agent (animal or robot) and present an integrated view of goal-directed choice that solves the ‘H4W problem’: ‘What do I need and Why, Where and When can this be obtained and How do I get it?’ The H4W problem is explained by mapping it to the structure and functioning of the Distributed Adaptive Control embodied cognitive architecture [36] and through that to the rodent neuronal architecture of goal-directed choice.

A second issue that is shared with the latter contribution is the importance of deploying fully embodied, robotic models of the neuronal mechanisms of perception, cognition and action to ecologically valid problems such as navigation and foraging. Milford & Schultz [37] review the state of the art in (probabilistic) goal-directed robotic navigation and discuss how they currently perform in real-world scenarios. They also discuss the possible relevance of these models for other disciplines by comparing the similarities and differences between algorithmic versus biologically motivated computational models and their behavioural validity.

Gillan & Robbins [38] explore yet another corner of choice behaviour: its malfunctioning in obsessive-compulsive disorder (OCD). They propose that OCD results from a lack of control over one's own actions to realize goals, which in turn depends on an imbalance between (goal-directed and habitual) associative learning systems. Beside which, the paper illustrates the importance of studying goal-directed systems to understand dysfunctions in psychiatry.

Osvath & Martin-Ordas [39] discuss ecological choices and especially future-oriented cognition from the primatologist's perspective. They extensively review evidence indicating that non-human animal cognition can be future-oriented (focusing on well-studied cases in the great apes) but also advances theoretical arguments against current conceptualizations of future-thinking and mental time travel.

5. Pressing scientific questions

Taken together, the contributions to this Special Issue provide a comprehensive panorama of current empirical evidence, theories and perspectives on goal-directed mechanisms that, we suggest, are central to human (and animal) cognition and behaviour. Clearly, the contributions to this Special Issue advance our current understanding but also at the same time raise new and important questions for the future. Although we have grouped the contributions thematically, several pressing scientific questions emerged from this Special Issue that cut across the topics, methods and disciplines.

One such pressing question is: ‘is there one brain area doing the actual choice or not? If not, how are the contributions of different areas (e.g. cortex, striatum and hippocampus) orchestrated for goal-directed choice?’ All the contributions to the first topic [13–18] but also other contributions [35] present pieces of evidence to solve this complex puzzle. Research in the field is often compartmentalized, with scholars that tend to develop theories that are prefrontal-centric or striatum-centric depending on the empirical methods and target areas they choose. From this Special Issue, the need emerges for a systems-level view in which multiple brain processes can be linked to specific mechanisms (that should be, when possible, computationally specified) and that, taken together, constitute goal-directed choice. If, for example, we consider that multiple brain substrates have been linked to utility representation and computations (e.g. OFC, amygdala and vStr), it is necessary to disentangle their specific roles [17] but also to connect all of them within a general framework [35]. Furthermore, this Special Issue indicates how ecological and evolutionary considerations can guide research in the field; for example, by considering how complex cognitive control abilities might derive from the sophistication of initially simpler neuronal systems [14,34,39].

Another pressing question is: ‘what are the key—neural, behavioural and computational—fingerprints of goal-directed choice and behaviour? and what are the differences with other (e.g. habitual) systems?’ The Special Issue includes three contributions that provide accessible introductions to the key neural and behavioural [32] and computational [26,28] signatures of goal-directed systems and compare them with alternatives (e.g. habitual) systems. The Special Issue also revolves around distinct computational schemes that are proposed to define goal-directed choice in a normative framework, which include model-based reinforcement learning [26], hierarchical reinforcement learning [28], ideomotor action [29], active inference [27] and game-theoretic approaches in the social domain [33]. Clearly, these and other proposals remain to be assessed empirically, and an important open question is whether they apply to the wide research field in which goal-directed choice is at play: from simple laboratory studies to complex ecological scenarios [39] and malfunctioning [38].

Several other important questions recur in at least several contributions of the Special Issue that we have no place to discuss here in detail. Indeed, the overall goal of the Special Issue was to bring together major contributors to the field, to report the current state of the art but also, and more importantly, to foster an interdisciplinary debate on goal-directed choice—a topic that, we predict, will increasingly gain prominence within and across multiple disciplinary domains as we refine our theoretical, empirical and computational methodologies. Within a new goal-centric framework for cognition that we have sketched here, we are confident that some of these scientific questions will be answered and new ones will emerge.

Besides being an important scientific objective per se, understanding (human) goal-directed choice and its pathologies is expected to bring a major societal impact. Most ‘economic’ decisions in our everyday lives (e.g. where to go for the weekend, what career to choose) require using goal-directed mechanisms of choice and thus understanding how the brain forms and selects among goals is central to discover the mechanistic bases of complex human behaviours. The malfunctioning of goal-directed choice mechanisms entails several pathologies and psychiatric disorders, including for example addictions, OCDs, depression and anxiety. Moreover, mechanisms of choice have susceptibilities that could be systematically exploited to influence people; consider for example how marketing strategies influence our choice of what to buy or whom to vote for. This implies that understanding the mechanisms that regulate goal-directed choice and its abnormalities might help shed light on fundamental problems of our modern societies.

Insights in the mechanics of goal-directed choice will also permit researchers to design a novel generation of artefacts including robots that are more autonomous (as they may become able to act on purpose well beyond what they are pre-programmed to do) and can better interact with humans (as they predict and evaluate the consequences of their actions, and their associated risks). Ultimately, society will benefit when robots become more flexible, versatile, intelligent and sentient as they incorporate principles from goal-directed decision-making.

Funding statement

The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. FP7-ICT-270108—Goal-Leaders (www.goal-leaders.eu).