Based on accumulating evidence, simulation appears to be a basic computational mechanism in the brain that supports a broad spectrum of processes from perception to social cognition. Further evidence suggests that simulation is typically situated, with the situated character of experience in the environment being reflected in the situated character of the representations that underlie simulation. A basic architecture is sketched of how the brain implements situated simulation. Within this framework, simulators implement the concepts that underlie knowledge, and situated conceptualizations capture patterns of multi-modal simulation associated with frequently experienced situations. A pattern completion inference mechanism uses current perception to activate situated conceptualizations that produce predictions via simulations on relevant modalities. Empirical findings from perception, action, working memory, conceptual processing, language and social cognition illustrate how this framework produces the extensive prediction that characterizes natural intelligence.
Accumulating evidence suggests that simulation constitutes a central form of computation throughout diverse forms of cognition, where simulation is the re-enactment of perceptual, motor and introspective states acquired during experience with the world, body and mind (Barsalou 2008). As described next, the re-enactment process has two phases: (i) storage in long-term memory of multi-modal states that arise across the brain's systems for perception, action and introspection (where ‘introspection’ refers to internal states that include affect, motivation, intentions, meta-cognition, etc.), and (ii) partial re-enactment of these multi-modal states for later representational use, including prediction. Each phase is addressed in turn.
(a) Storage of modal states
When an entity or event is experienced, it activates feature detectors in the relevant neural systems. During visual processing of a bicycle, for example, neurons fire for edges and surfaces, whereas others fire for colour, configural properties and motion. The overall pattern of activation across this hierarchically organized distributed system represents the entity in vision. Analogous patterns of activation in other sensory modalities represent how the bicycle might sound and feel. Activations in the motor system represent actions on the bicycle. Activations in the amygdale and orbitofrontal areas represent affective reactions.
As a feature system becomes active to represent an entity or event, conjunctive neurons in association areas capture the activation pattern for later representational use. Populations of conjunctive neurons code the pattern, with each individual neuron participating in the coding of many different patterns (e.g. Damasio 1989; Simmons & Barsalou 2003). Locally, association areas near a modality capture activation patterns within it (e.g. visual association areas capture patterns of visual features). In turn, higher association areas in the temporal, parietal and frontal lobes integrate activations across modalities.
(b) Re-enactments of modal states
The architecture just described has the functional ability to produce modal re-enactments (simulations). Once associative neurons capture a feature pattern, they can later reactivate it in the absence of bottom-up stimulation. When retrieving a memory of a bicycle, associative neurons partially reactivate the visual state active during its earlier perception. Similarly, when retrieving an action performed on the bicycle, associative neurons partially reactivate the motor state that produced it. A re-enactment never constitutes a complete reinstatement of an original modal state, and various sources of bias may often distort it. Thus, re-enactments are always partial and potentially inaccurate. Regardless, some semblance of the original modal state is re-enacted for representational use during cognitive activity.
Re-enactment is not necessarily conscious but may also be unconscious, probably being unconscious even more often than conscious. Unconscious re-enactments may occur frequently during perception, memory, conceptualization, comprehension and reasoning, along with conscious re-enactments. When re-enactments reach awareness, they can be viewed as constituting mental imagery, given that imagery is typically assumed to be conscious.
2. Simulators and simulations
Barsalou (1999, 2003a) developed a theory of cognition based on the neural re-enactment of modal states. Simulators and simulations constitute the theory's two central constructs. Simulators integrate information across a category's instances, whereas simulations are specific conceptualizations of the category. Each is addressed in turn.
Because the instances of a category typically have statistically correlated properties, encountering these instances should tend to activate similar neural patterns in feature systems (e.g. Farah & McClelland 1991; Cree & McRae 2003). Additionally, similar populations of conjunctive neurons in association areas—tuned to these feature conjunctions—should typically capture these patterns (Damasio 1989; Simmons & Barsalou 2003). After experiencing a category's instances over time, a distributed multi-modal system develops to represent the category as a whole. Barsalou (1999) referred to these distributed systems as simulators. Theoretically, a simulator functions as a concept or type in more traditional theories by integrating the multi-modal content of a category across instances, and by providing the ability to interpret individuals as tokens of the type (Barsalou 2003a).
Consider a simulator that represents the concept of bicycle. Across encounters with different instances, visual information about how bicycles look becomes integrated in the simulator, along with auditory information about how they sound, somatosensory information about how they feel, motor sequences for interacting with them, affective responses to experiencing them and so forth. The result is a distributed system throughout the brain's feature and association areas that accumulates and integrates modal content processed for the category. As Barsalou (2003a) describes, many additional simulators develop to represent properties, relations, events and mental states relevant to bicycles (e.g. spokes, mesh, pedal, effort).
Once a simulator exists to represent a category, it can re-enact small subsets of its content as specific simulations. As mentioned earlier, these simulations may never reach consciousness, although some aspects may become conscious, as in mental imagery. All content in a simulator never becomes active simultaneously—only small subsets become active to represent the category on particular occasions. Diverse factors such as frequency, recency and context determine the simulations that become active during a specific simulation of the category (cf. Barsalou 1987, 1989, 1993). The bicycle simulator, for example, might simulate a touring bike on one occasion and mountain bike or a cross bike on others. Because all the experienced content for bicycles resides implicitly in the bicycle simulator, diverse subsets can be re-enacted on different occasions, and tailored to specific contexts. Often, simulations may represent specific instances of a category (e.g. individual bicycles), but they may also represent groups of individuals in a more generic manner (cf. Medin & Ross 1989).
Simulations are almost certainly never exact re-enactments of previously experienced category instances. Instead, simulations may typically only re-enact instances partially, with various factors producing distortion (see examples later for simulations in perception and action). Furthermore, the level of detail in a simulation may vary widely, from being relatively vague and skeletal to be being vivid and detailed, depending on the amount of information available or required. Finally, when simulations of events occur over time, they may often only include a small subset of the originally perceived points within the temporal sequence, rather than being a complete re-enactment of all points.
Barsalou (1999, 2003a) proposed that simulations, combined with simulators, perform a wide variety of functions. Simulations represent a category's instances in their absence during memory, language and thought. Simulations produce inferences and predictions about a category's perceived instances using the pattern completion inference mechanism described later. Simulations combine productively to produce infinite conceptual combinations. Simulations represent the propositions that underlie type-token predication and complex propositional structures. Simulations represent abstract concepts. In general, simulations implement the functionality of classic symbolic systems, with this functionality emerging implicitly in modal and association systems rather than being represented explicitly in symbolic structures, as in predicate calculus and frames.
(c) Sources of simulators
An infinite number of simulators can develop in long-term memory for objects, settings, events, actions, introspections, properties, relations and so forth. Specifically, a simulator develops for any component of experience that attention selects repeatedly (Barsalou 1999, 2003a). When attention focuses repeatedly on a type of object in experience, such as bicycle, a simulator develops for it. Analogously, if attention focuses repeatedly on an action (pedalling), introspection (happiness) or relation (mesh) simulators develop to represent them as well. Such flexibility is consistent with Schyns et al.'s (1998) proposal that the cognitive system acquires new concepts as they become relevant. Because selective attention is flexible and open ended, a simulator develops for any component of experience selected repeatedly.
An important issue concerns how simulators for abstract concepts develop. Barsalou (1999) proposed that simulators for abstract concepts generally capture complex multi-modal simulations of temporally extended events, with simulation of introspections being central. According to this account, simulators develop to represent categories of internal experience just as they develop to represent categories of external experience. In support of this account, Barsalou & Wiemer-Hastings (2005) found that concrete and abstract concepts both contain extensive amounts of situation information, but that abstract concepts tend to contain more information about introspections and events (also see Wiemer-Hastings et al. 2001).
(d) Empirical evidence
Accumulating evidence implicates simulation as a basic computational mechanism in the brain. Reviews of relevant evidence can be found in Pulvermüller (1999, 2005), Martin (2001, 2007), Barsalou (2003b, 2008), Thompson-Schill (2003), Pecher & Zwaan (2005), Smith (2005), Smith & Gasser (2005), Gibbs (2006), de Vega et al. (2008) and Semin & Smith (2008).
3. Situated conceptualizations
Barsalou (2003b) proposed that concepts are not typically processed in isolation but are typically situated in background settings, events and introspections. When representing bicycle, for example, people do not represent a bicycle in isolation but represent it in relevant situations. According to Yeh & Barsalou (2006), people situate concepts for the following reason: if the brain attempts to simulate a perceptual experience when representing a concept, it should typically simulate a situation, because situations are intrinsic in perception. At any given moment in perception, people perceive the immediate space around them, including agents, objects and events present. Even when people focus attention on a particular entity or event in perception, they continue to perceive the background situation—the situation does not disappear. If perceptual experience takes the form of a situation, and if a conceptual representation simulates perceptual experience, then the form of a conceptual representation should take the form of a perceived situation. When people construct a simulation to represent a category, they should simulate the category in a relevant perceptual situation, not in isolation.
Barsalou (2003b) referred to situated representations of categories as situated conceptualizations. Across different occasions, diverse situated conceptualizations represent a category. Because a single general conceptualization would be too vague to support relevant inferences in specific situations, representations that are more specialized are constructed instead.
Consider the representation of bicycle. According to traditional views, bicycle is represented as a generic set of amodal propositions that become active as a whole every time the category is processed. According to the view proposed here, however, the cognitive system produces many different situated conceptualizations of bicycle, each tailored to help an agent interact with bicycles in different situations. For example, one situated conceptualization for bicycle might support riding a bicycle, whereas others might support locking a bicycle, repairing a bicycle and so forth. On this view, the concept for bicycle is not a single generic representation of the category. Instead, the concept is the skill or ability to produce a wide variety of situated conceptualizations that support goal achievement in specific contexts.
(a) Multi-modal simulations implement situated conceptualizations
Barsalou (2003b) proposed that a complex multi-modal simulation becomes active to represent a situated conceptualization. Consider a situated conceptualization for riding a bicycle. Such a conceptualization might simulate how riding a bicycle appears perceptually from the perspective of riding one. When riding a bicycle, one views the bicycle from above, sees wheels rolling below, hears the sprockets and chain meshing, feels the wind blowing by and so forth. All these perceptual aspects of the situation can be represented as modal simulations in a situated conceptualization that represents the experience in its absence. Rather than amodal propositions representing these perceptions, simulations represent them in the relevant modal systems.
A situated conceptualization of riding a bicycle is likely to simulate actions that the agent takes, such as peddling, changing gears and watching for traffic. Modal simulations could also represent these aspects of a situated conceptualization via simulations of the actions.
A situated conceptualization of riding a bicycle is likely to include simulations of introspections, such as effort, happiness, the goal to reach a destination, planning a route and motivation to push ahead at maximum speed. Again, simulations of these internal states could represent them in the situated conceptualization. Just as external experience can be simulated, so can internal experience.
Finally, a situated conceptualization for riding a bicycle simulates a setting where the event could occur—the event is not simulated in a vacuum. Riding a bicycle, for example, might be simulated on a route to work or on a mountain trail. Again, such knowledge can be represented as simulations of particular settings.
In summary, a situated conceptualization typically simulates four basic types of information from a particular perspective: (i) perceptions of relevant people and objects, (ii) actions, (iii) introspections, and (iv) settings. Putting all these together, a situated conceptualization is a multi-modal simulation of a multi-component situation, with each modal component simulated in the respective neural system.
A further assumption is that a situated conceptualization consists of simulations from many different simulators (Yeh & Barsalou 2006). A situated conceptualization for riding a bicycle is likely to include simulations from simulators for people, objects, actions, introspections and settings. Thus, a single simulator alone does not produce a situated conceptualization. Instead, many simulators produce the components that a situated conceptualization contains.
Finally, a situated conceptualization places the conceptualizer in the respective situation, creating the experience of ‘being there’ (Barsalou 2002, 2003b). By re-enacting actions and introspections from a particular perspective, a situated conceptualization creates the experience of the conceptualizer being in the situation—the situation is not represented as detached and separate from the conceptualizer.
(b) Entrenched situated conceptualizations
Across the lifespan, people experience many situations repeatedly while interacting with people, artefacts, social institutions and so forth. Consequently, knowledge about these familiar situations becomes entrenched in memory, supporting skilled performance in them. Situated conceptualizations represent this entrenched knowledge. When a situation is experienced repeatedly, multi-modal knowledge accrues in the respective simulators for the people, objects, actions, introspections and settings. Components of the conceptualization become entrenched as simulations in the respective simulators, as do associations between simulations and simulators. Over time, the situated conceptualization becomes so well established that it becomes active automatically and immediately when the situation arises. After riding a bicycle on many occasions, a situated conceptualization becomes entrenched in memory such that minimal cuing activates it when relevant. Different but similar situations may also activate entrenched conceptualizations by analogy on later occasion. Indeed most experienced situations, regardless of how novel they are, may typically activate whatever situated conceptualization happens to be the most similar. As described next, activating situated conceptualizations plays central roles in processing a current situation.
(c) Inference via pattern completion
Once situated conceptualizations become entrenched in memory, they support a pattern completion inference process (Barsalou 2003b; Barsalou et al. 2003). On encountering a familiar situation, an entrenched situated conceptualization for the situation becomes active. Typically, though only part of the situation is perceived initially. A relevant person, setting, event or introspection may be perceived, which then predicts that a particular situation—represented by a situated conceptualization—is about to unfold. By running the situated conceptualization as a simulation, the perceiver anticipates what will happen next, thereby performing effectively in the situation. The agent draws inferences from the simulation that go beyond the information given (e.g. Bruner 1957).
When a situated conceptualization becomes active, it constitutes a rich source of prediction via this pattern completion inference mechanism. A situated conceptualization is essentially a pattern, namely, a complex configuration of multi-modal components that represent a familiar situation. When a component of this pattern matches something experienced in the environment, the larger pattern becomes active in memory. The remaining pattern components—not yet experienced—constitute inferences, namely, educated guesses about what might occur next. Because the remaining components co-occurred frequently with the perceived components in previous situations, inferring the remaining components is plausible. When a partially viewed situation activates a situated conceptualization, the conceptualization completes the pattern that the situation suggests.
Imagine seeing a bicycle with a flat tyre in someone's garage. Furthermore, imagine that the perceived bicycle matches components of one or more situated conceptualizations entrenched in memory for bicycles. Once one conceptualization wins the activation process—perhaps one containing bicycles with flat tyres—it provides inferences via pattern completion, such as introspections likely to be present (being unhappy about a flat tyre), actions likely to result (taking the wheel off the bicycle), tools that might be useful (a wrench) and so forth. The unfolding of such inferences—realized as simulations—provides a powerful source of prediction.
(d) Empirical evidence
Accumulating evidence implicates situated conceptualization and pattern completion inference on situated conceptualizations as basic computational mechanisms in the brain. Reviews of relevant evidence can be found in Barsalou (2003b, 2005), Barsalou et al. (2003), Yeh & Barsalou (2006) and Robbins & Aydede (2008).
4. Simulations and situated conceptualizations as sources of prediction
Simulations and situated conceptualizations—coupled with pattern completion inference—produce continual predictions across the spectrum of cognitive activities. The perception of something familiar in the environment, body or introspection activates a simulation or situated conceptualization that contains it. Components of the simulation or situated conceptualization not yet experienced constitute predictions about events likely to occur, actions likely to be effective and introspections likely to result. Because simulations represent predictions, predictions are simulated in the same modalities in which they would be experienced. Consequently, predictions can be readily matched to corresponding components, should they actually occur.
Imagine seeing a coffee bean container that activates a situated conceptualization for making espresso. Although only this component of the situated conceptualization is perceived, it activates simulations of other components as predictions about what could happen next: the container could be opened to reveal coffee beans inside; further actions could be taken to grind the beans and make espresso; psychological states such as pleasure, feeling stimulated, and being more awake could result. As this situated conceptualization becomes active, its simulated components can be used to monitor perceptions, actions and introspections as they actually occur, assessing whether the situated conceptualization's predictions are satisfied. Because simulated predictions reside in the same systems that perceive the environment, carry out actions, and introspect on internal states, they can be matched to actual experience as it occurs, thereby assessing whether events have unfolded as predicted. Assessing the accuracy of predictions determines whether the situational conceptualization currently active is likely to provide effective guidance for interacting with the current situation. If initial predictions match, then the active conceptualization can be trusted to provide further predictions. If not, a new conceptualization must be retrieved or constructed.
The remaining sections illustrate how simulations and situated conceptualizations generate predictions in this manner across cognitive processes from perception to social cognition.
Goldstone (1995) illustrated how simulations produce perceptual predictions in a simple perceptual learning paradigm. Participants learned associations between a visual stimulus (e.g. the letter E) and a colour (e.g. dark red). Later, when a coloured stimulus was flashed as a standard (e.g. a red E), participants used a slider to reproduce its colour on an adjustable token of the same stimulus simultaneously present in black (e.g. a black E). Interestingly, participants distorted the reproduced colour of the second stimulus towards the prototypical colour of the perceived standard across trials (i.e. the actual colour of the standard was distorted). Perceiving the standard's shape, activated a simulation of its prototypical colour across trials, which then distorted the standard's perceived colour, perhaps fusing with it. Although, these simulations of the prototypical colour distorted perception of the standard's colour, they produced predictions about colour likely to be true, statistically speaking, across instances of the stimulus. If the perceiver was searching for the stimulus in the environment, these colour predictions would be maximally informative in finding it (for a similar finding, see Hansen et al. 2006).
Simulations that originate in situated conceptualizations for objects produce predictions about motion. During motion continuation, viewers view the visual trajectory of a moving object. As they view the actual trajectory, they simultaneously simulate the object's future trajectory, predicting where it is likely to move next (e.g. Freyd 1987). Knowledge about whether an object moves quickly or slowly affects these simulated trajectories accordingly (e.g. Reed & Vinson 1996). Similarly, during apparent motion, simulations of possible human actions predict interpolated motion likely to be present between two alternating pictures of a human body in different positions (e.g. Shiffrar & Freyd 1990, 1993). Stevens et al. (2000) showed that simulations in the motor system underlie these predictions.
During auditory perception, lexical knowledge produces predictions via simulation that contribute to speech perception. In the phoneme restoration effect, listeners use auditory knowledge about a word's phonemes to simulate and predict a missing phoneme (e.g. Warren 1970). When a phoneme is missing, information present for surrounding phonemes is sufficient to activate a simulation of the word that includes the missing phoneme. Simulations appear to be present because participants hear speech sounds that are not present physically. Samuel (1997) showed that these simulations originate in early auditory processing. In the McGurk effect (McGurk & MacDonald 1976), perceiving a mouth producing a particular phoneme (e.g. ba) produces a corresponding auditory simulation that conflicts with hearing a different phoneme actually uttered (e.g. ga). Once the simulated phoneme is fused with the perceived phoneme, the result is the auditory perception of a phoneme that is the average of the simulated and perceived phonemes (e.g. da). Thus, an auditory simulation fuses with a perceived auditory phoneme to create the perception of a phoneme neither spoken visually nor heard auditorally. Because multiple modalities underlie the McGurk effect—visual information triggering auditory simulations—multi-modal simulations such as those in situated conceptualizations generate predictions about the phonemes spoken.
As people perceive visual objects, situated conceptualizations produce predictions about actions likely to be effective while interacting with them. When people perceive a cup, for example, a situated conceptualization for interacting with cups produces a simulation of grasping it. Tucker & Ellis (2001) showed that when a cup handle is on the left, people simulate grasping it with the left hand, but when the cup handle is on the right, people simulate grasping it with the right hand. In each case, a simulation predicts which hand is likely to be the most effective in grasping the cup. Bub et al. (2008) further showed that a perceived object (or its name) automatically triggers simulations of both grasping and functional actions predicted to be effective during interactions with the object.
Using fMRI, Chao & Martin (2000) showed that perceived objects activate situated conceptualizations for interacting with the objects. On perceiving a hammer, for example, the grasping circuit in the brain becomes active, predicting—via a motor simulation—appropriate actions to take on the object. As in Tucker & Ellis (2001), these predictions are hand specific, with the grasping circuit for the right hand becoming active in right-handed participants. Lewis (2006) reviews similar and related findings.
Researchers report a wide variety of other findings also consistent with the conclusion that situated conceptualizations produce ubiquitous predictions about action via simulation. In Bosbach et al. (2005), accurately predicting the weight of an object lifted by another agent required simulating the lifting action in one's own motor and somatosensory systems. In Repp & Knoblich (2004), a pianist's ability to identify auditory recordings of his or her own playing depended on simulating the motor actions underlying it. In Proffitt (2006), simulations of perceived effort appeared to affect the perceived steepness of a hill and the perceived length of a path.
Motor simulations of predicted limb position are central for motor control. As an action is performed, the motor system constructs a feed-forward simulation of the action that monitors and guides it (e.g. Wolpert et al. 1999; Grush 2004). These motor simulations also play roles in predicting the anticipated actions of agents perceived visually (Wilson & Knoblich 2005) and in speeding visual object recognition (e.g. Helbig et al. 2006).
(c) Implicit memory
Implicit long-term memory is closely related to perceptual prediction. In both cases, perceptual memories—implemented as simulations and situated conceptualizations—become active and affect perception. As we just saw for perception, simulations and situated conceptualizations represent perceptions and actions that go beyond stimulus information. In implicit memory, simulations increase perceptual fluency by facilitating the processing of perceptual information and speeding the formation of perceptions (repetition priming). According to the account proposed here, when a perceptual stimulus activates a similar perceptual memory, the perceptual memory runs as a simulation of the stimulus and speeds its processing by activating relevant processing areas, with the simulation perhaps fusing with the stimulus information. To the extent that the perceptual memory matches the stimulus, the memory predicts that the stimulus is another instance of itself, thereby increasing the fluency of perceiving it via top-down activation. When, for example, a perceived face activates an implicit memory, the face memory predicts that the perceived face is another instance of itself and speeds its processing.
Numerous findings support the proposal that the prediction underlying implicit memory results from the simulation of perceptual memories (Roediger & McDermott 1993; Schacter et al. 2004). First, perceptual processing is important for establishing robust repetition priming effects, consistent with the proposal that perceptual memories play central roles in priming (e.g. Jacoby 1983). Second, repetition priming is the strongest when the modalities of the memory and the perceived stimulus match (e.g. when an auditory memory exists to help process an auditory stimulus; Kirsner et al. 1989). Third, repetition priming is the strongest when the perceptual details of a memory and a perceived stimulus match, such as orientation, size, font, etc. (e.g. Jolicoeur 1985; Jacoby & Hayman 1987). Fourth, imagining a stimulus produces repetition priming similar to actually perceiving it, suggesting that perceptual representations underlie both (e.g. Roediger & Blaxton 1987; Schacter & Graf 1989). For these reasons, simulations of perceptual states appear central to the predictions that implicit memories generate about corresponding perceptions.
(d) Working memory
Mental imagery can be viewed as the conscious and explicit manipulation of simulations in working memory to predict future events. When imagining a rotating object, for example, a person attempts to predict the object's future orientation, analogous to the less intentional and less conscious inferences in simulating perceptual trajectories discussed earlier. Considerable evidence indicates that simulations in working memory underlie visual imagery (e.g. Kosslyn 1980, 1994; Shepard & Cooper 1982; Finke 1989), motor imagery (e.g. Jeannerod 1995; Grèzes & Decety 2001) and auditory imagery (e.g. Halpern et al. 2004). In all cases, imagery attempts to predict what is likely to occur next if an imagined event on the respective modality actually occurred.
When action is relevant to visual imagery, the motor system becomes engaged, implying the presence of a situated conceptualization. When visual rotation of a body part is imagined, for example, bodily constraints shape the rotational trajectory (e.g. Parsons 1987a,b). Similarly, mental rotation of visual objects is often accompanied by motor simulations of making them turn (e.g. Richter et al. 2000). In both cases, predictions on multiple modalities are conveyed via multi-modal simulations.
(e) Conceptual processing
When entities and events are categorized, conceptual knowledge about the respective categories becomes active to predict what is likely to happen next. As we saw earlier, when people categorize visual objects, such as hammers, conceptual knowledge associated with the respective categories becomes active to predict relevant actions (e.g. Chao & Martin 2000; Lewis 2006). Because multi-modal simulations represent these predictions, situated conceptualizations appear responsible. Similarly, when people categorize instances of food categories, situated conceptualizations—again implemented via multi-modal simulations—produce predictions about how foods will taste and how rewarding they will be to consume (Simmons et al. 2005). Additionally, perceiving an object predicts settings in which it is likely to occur. Bar (2004), for example, found that perceiving isolated objects activates neural systems that represent their environmental contexts. Indeed, recognizing the presence of certain objects is a powerful means of predicting the scene or situation likely to be present.
Much behavioural research supports these conclusions (for a review, see Yeh & Barsalou 2006). In object perception, for example, perceiving an object activates a background scene that can speed object categorization (e.g. Biederman 1972, 1981; Palmer 1975; Bar & Ullman 1996). Perceiving an object activates a situated conceptualization associated with its category, which—via interactive activation—speeds processing of the object, along with related objects and events in the situation.
Situated conceptualizations also play central roles when people attempt to predict information associated with categories in the absence of category exemplars being present. When people are asked to describe the properties of an absent object category (e.g. watermelons), they do not simply generate properties of the object itself. Instead, they also generate properties for likely settings (porch), events (eating) and introspections (pleasure). Rather than simply describing the object per se as instructed, people further predict other aspects of situations likely to be present (Barsalou & Wiemer-Hastings 2005; Chagneau et al. 2009; Santos et al. submitted; Wu & Barsalou submitted). As people describe an object, they situate it from their own perspective, simulating themselves in the situation as well. In other words, people construct the experience of being there with the object to generate its properties (Barsalou 2002, 2003b; Yeh & Barsalou 2006).
Prediction lies at the heart of language comprehension. When processing language, a comprehender's task is to predict what the language means. Accumulating evidence indicates that simulations and situated conceptualizations convey these predictions. Consider research reviewed by Zwaan & Madden (2005) on the role of visual simulation in comprehension. In these experiments, participants read sentences that implied something perceptual not stated literally. For example, participants read sentences such as ‘Mary hammered the nail into the wall’, which implied that the nail was oriented horizontally. If readers constructed simulations to predict what the sentence meant, the simulation should have contained implicit perceptual information not mentioned in the sentence, such as object orientation. After reading the sentence, participants viewed a nail in either a horizontal or a vertical orientation and named it. As predicted, participants were faster to name the nail in a horizontal position than in a vertical position, suggesting that they had simulated its orientation while comprehending the sentence (other participants showed the opposite pattern after reading sentences such as ‘Mary pounded the nail into the floor’). Many similar experiments demonstrate diverse perceptual matching effects, consistent with the view that comprehenders simulate situations to predict what sentences mean.
Much research has also shown that comprehenders use motor simulations to generate predictions about meaning. When participants comprehend the word for an action, the motor system represents its meaning (Pulvermüller 2005). Specifically, verbs for head, arm and leg actions produce head, arm and leg simulations in the respective cortical areas. These simulations play causal roles in language processing, given that transcranial magnetic stimulation over the relevant motor areas affects the processing of predicting meaning (e.g. Buccino et al. 2005; Pulvermüller et al. 2005). Much behavioural research corroborates these results. For example, comprehension is facilitated when the action described in a sentence is consistent with the action to make a response (Glenberg & Kaschak 2003; Zwaan & Taylor 2006). Similarly, when reading about a sport, such as hockey, experts produce motor simulations absent in novices (Holt & Beilock 2006). In all these experiments, participants use motor simulations to predict the meanings of words and sentences that describe action.
Other research shows that participants predict motion through space, as they comprehend language. Richardson et al. (2003) found that readers simulate horizontal and vertical paths to predict the meanings of concrete and abstract verbs (e.g. push versus lift, argue versus respect). Matlock (2004) found that implied motion in a sentence (e.g. The road runs through the valley) produces corresponding simulations of motion through space. Richardson & Matlock (2007) found that these simulations produce predictive eye movements.
(g) Social cognition
For decades, social psychologists have shown that bodily states predict relevant social situations, and conversely, that social situations predict relevant bodily states (Niedenthal et al. 2005). For example, slumping posture is associated with negative affect and failure, whereas upright posture is associated with positive affect and success. Barsalou et al. (2003) proposed that situated conceptualizations represent these patterns of association, and that pattern completion inference produces extensive predictions from these patterns during social interaction. For example, knowing that a student failed in an exam activates a situated conceptualization for this type of event, which in turn generates predictions that the individual will be slumped and unhappy. Conversely, seeing a slumped and unhappy student generates the prediction that the student might have failed an exam. In general, situated conceptualizations provide a powerful form of prediction in social cognition, with these predictions being conveyed via simulations during pattern completion inference.
Accumulating evidence suggests that simulators, simulations and situated conceptualizations play central roles throughout cognition. Besides representing knowledge, these systems provide a powerful and ubiquitous source of prediction. Indeed, the brain can be viewed as a coordinated system that generates a continuous stream of multi-modal predictions during situated action and social interaction (Barsalou et al. 2007).
Perhaps the most pressing issue surrounding this area of work is the lack of well-specified computational accounts. Our understanding of simulators, simulations, situated conceptualizations and pattern completion inference would be much deeper if computational accounts specified the underlying mechanisms. Increasingly, grounding such accounts in neural mechanisms is obviously important as well, as is designing increasingly sophisticated experiments to assess and develop these accounts.
I am grateful to Moshe Bar for helpful comments. This work was supported by National Science Foundation grants SBR-9421326, SBR-9796200 and BCS-0212134 to L.W.B.
One contribution of 18 to a Theme Issue ‘Predictions in the brain: using our past to prepare for the future’.
- © 2009 The Royal Society