Can emotion recognition be taught to children with autism spectrum conditions?

Simon Baron-Cohen, Ofer Golan, Emma Ashwin

Abstract

Children with autism spectrum conditions (ASC) have major difficulties in recognizing and responding to emotional and mental states in others' facial expressions. Such difficulties in empathy underlie their social-communication difficulties that form a core of the diagnosis. In this paper we ask whether aspects of empathy can be taught to young children with ASC. We review a study that evaluated The Transporters, an animated series designed to enhance emotion comprehension in children with ASC. Children with ASC (4–7 years old) watched The Transporters every day for four weeks. Participants were tested before and after intervention on emotional vocabulary and emotion recognition at three levels of generalization. The intervention group improved significantly more than a clinical control group on all task levels, performing comparably to typical controls at time 2. The discussion centres on how vehicles as mechanical systems may be one key reason why The Transporters caused the improved understanding and recognition of emotions in children with ASC. The implications for the design of autism-friendly interventions are also explored.

1. Introduction

Can empathy be taught? This turns out to be both a philosophical and a practical problem, because it is relevant both to our definition of empathy and to how we approach those people in the population who—for whatever reason—do not develop empathy in the usual way. In this paper, we begin with a definition of empathy and argue that at least one of the two major ‘fractions’ of empathy is amenable to teaching. We then summarize an experiment that evaluates a new method designed to teach this element of empathy to young children with autism who—for largely genetic reasons—have specific difficulties in developing empathy.

2. What is empathy?

We define empathy as the ability to attribute mental states to others, and to respond with an appropriate emotion to the other person's mental states (Baron-Cohen & Wheelwright 2004). This definition of empathy suggests that the two main ‘fractions’ of empathy are a ‘cognitive’ component (the recognition of another person's mental state) and an ‘affective’ component (the emotional reaction to another person's mental state). The cognitive component is sometimes also called a ‘theory of mind’ (Dennett 1989) (see figure 1).

Mental states include thoughts and emotions, thoughts being traditionally fractionated into beliefs, desires, intentions, goals and perceptions (Dennett 1989; Baron-Cohen 1995). Emotions are traditionally fractionated into six ‘basic’ emotions (happy, sad, angry, afraid, disgusted and surprised) (Ekman 1999) and numerous ‘complex’ emotions. Complex emotions involve attributing a cognitive state as well as an emotion and are more context and culture dependent (Griffiths 1997). The basic emotions are held to be so because they are universally recognized and expressed in the same way. This distinction, however, is not without its critics, since it may be that more emotions are universally recognized and expressed than these six but have been overlooked because cross-cultural research is very expensive, time consuming and difficult (Baron-Cohen et al. 2006). Indeed, research into complex emotions (usually towards developing taxonomies) has been mostly language and culture specific (Ortony et al. 1987; Storm & Storm 1987). Our own work suggests that in the English language, there are at least 412 distinct emotions and related mental states (each with their own descriptor that is not just a synonym for another emotion) that are recognizable by independent judges within the UK (Baron-Cohen et al. 2004; Golan & Baron-Cohen 2006).

Having defined empathy into at least two major fractions, we turn now to the question of the ‘teachability’ of empathy. Some individuals in the population may be delayed in the development of empathy, for different reasons. These include people with autism spectrum conditions (ASC) who for neurological—and ultimately genetic—reasons have difficulties in putting themselves into someone else's shoes and knowing how to respond to another's feelings, in real time. Since such deficits may have a significant impact on their social functioning, this raises the challenge of whether aspects of empathy can be facilitated or taught to individuals with ASC. We will summarize some evidence that the first component of empathy—cognitive empathy—can indeed be taught. This task is made easier through the design of educational resources (including computer-based methods) that tap into systematic areas of interest, characteristic of autism, that are therefore intrinsically motivating. While we do not rule out that the second component of empathy—affective empathy—can be taught, it remains the case that all efforts have so far been focused on the cognitive fraction, so that it is unknown if the second could also be taught.

3. Empathy and systemizing in autism spectrum conditions

People with ASC have social-communication difficulties alongside circumscribed interests (‘obsessions’) and a strong preference for sameness and repetition (APA 1994). Underlying these characteristics are difficulties understanding the emotional and mental states of others (Baron-Cohen 1995). Individuals with ASC have difficulties recognizing emotions from facial expressions, vocal intonation, body language, separately (Hobson 1986a,b; Yirmiya et al. 1992; Baron-Cohen et al. 2001a,b) and in context (Klin et al. 2002; Golan et al. 2008). Although some individuals with ASC recognize basic emotional expressions (Baron-Cohen et al. 1993; Grossman et al. 2000), difficulties in identifying more complex emotions persist into adulthood (Baron-Cohen et al. 1997, 2001a,b; Golan et al. 2006).

The emotion recognition difficulties are in part the result of altered face processing (Klin et al. 2002; Dawson et al. 2004), which itself may be due to a failure to interpret the mentalistic information conveyed by the eyes (Baron-Cohen et al. 1997). Others' facial expressions may also be less intrinsically rewarding. Children with ASC show reduced attention to faces and to eyes in particular (Swettenham et al. 1998). The result of this reduced experience with faces is that children with ASC thus do not become ‘face experts’ (Dawson et al. 2005). For example, while the typically developing brain shows an electrophysiological response to upright faces called the N170 waveform, the autistic brain shows a reduced N170 (Grice et al. 2005).

In contrast to their difficulties in emotion recognition, individuals with ASC have intact or even enhanced abilities in ‘systemizing’ (Baron-Cohen 2002, 2006). Systemizing is the drive to analyse or build systems, allowing one to predict the behaviour of the system and control it. Systems may be mechanical (e.g. vehicles), abstract (e.g. number patterns), natural (e.g. the tide) or collectible (e.g. a library classification index). The ‘obsessions’ or narrow interests of children with ASC cluster in the domain of systems (Baron-Cohen & Wheelwright 1999). These include vehicles, spinning objects and computers, all of which are attractive to individuals with ASC. At the heart of systemizing is the ability to detect patterns or rules of the form ‘if a, then b’. The systemizing theory of autism relates this affinity to their systematic and predictable nature. In the study summarized later in this paper, we illustrate how these special interests can be harnessed when teaching children with ASC, using computer-based or multimedia formats, to keep them intrinsically motivated.

The systemizing theory of autism has been supported by different studies: children with ASC have been found to out perform matched controls on tests of ‘intuitive physics’ (Baron-Cohen et al. 2001a,b), and adults with ASC were at least intact on such tests (Lawson et al. 2004), as well as on other tests that involve excellent attention to detail (Mottron et al. 2006), a prerequisite for good systemizing (Shah & Frith 1983; Jolliffe & Baron-Cohen 1997; O'Riordan et al. 2001; Baron-Cohen 2008). In addition, individuals with ASC score above average on the systemizing quotient (SQ), a self-report (or parent-report) measure of how strong one's interests are in systems (Baron-Cohen et al. 2003; Wheelwright et al. 2006; Wakabayashi et al. 2007; Auyeung et al. in press).

If children with ASC possess intact or enhanced systemizing skills, it may be possible for them to use such skills to facilitate their empathy, particularly in the cognitive component of emotion recognition. Lego therapy (Owens et al. 2008) is an example that encourages young children with ASC to build Lego models in groups of three, thereby gaining opportunities for social interaction. Children participating in Lego therapy are intrinsically motivated by Lego because it involves constructional systems. While Lego is not directly relevant to the focus of this special issue on ‘affective computing’, it is indirectly relevant because (like computers) Lego can be assembled in predictable and repeating sequences.

A method that is far more directly relevant to affective computing as applied to autism is the Mindreading DVD. This comprises educational software that was designed to be an interactive, systematic guide to emotions (Baron-Cohen et al. 2004; www.jkp.com/mindreading). It was developed to help people with ASC learn to recognize both basic and complex emotions and mental states from video clips of facial expressions and audio recordings of vocal expressions. It covers 412 distinct emotions and mental states, which are organized developmentally and classified taxonomically to be attractive to a mind that learns through systemizing. The principle behind this was that individuals with autism may not learn to recognize emotional expressions in real time during live social situations because emotions are fleeting and do not repeat in an exact fashion, which may reduce the number of opportunities to systematically learn from repetition. Putting emotions into a computer-based learning environment enables emotions to be played and replayed over and over again in an identical fashion, such that the learner can have control over their speed and the number of exposures they need in order to analyse and memorize the features of each emotion.

Furthermore, since emotions vary depending on who is expressing them, in the real world it can be difficult to see what defines each specific emotion. Mindreading helps its users overcome this problem by having each of the 412 emotions portrayed by six different actors (male and female, old and young, different ethnicities), to facilitate learning to recognize emotions independently of the identity of the person expressing that emotion. In addition, in the real world, emotions can appear unlawful (some people smile when they are happy other people smile when they are pretending to be happy, and yet others are happy when they are not smiling at all), so Mindreading imposes some lawfulness onto emotions by assigning a clear label to each emotional expression, including masked, or insincere emotional expressions (e.g. emotions in the ‘sneaky’ category). Finally, emotions in the real world can be hard to classify, so Mindreading offers the user a pre-designed classification system to assist in finding patterns among inherently unpatterned emotional information.

Using Mindreading over a 10 week intervention (2 h usage per week), individuals with ASC improved in their ability to recognize a range of complex emotions and mental states (Golan & Baron-Cohen 2006). In a follow-up conducted one year after the completion of the intervention period, individuals with ASC who used Mindreading reported an improved ability to form friendships and relationships and increased awareness of the importance of emotions and emotional expressions in everyday life, improving their understanding of emotions and their corresponding expressions, and affecting their ability to function socially (Golan & Baron-Cohen 2007). This is an encouraging result because it suggests that at least one cognitive component of empathy can be taught and that it may have a long-term effect and affect social functioning. It is not known if such improvement would be seen if the intervention was shorter in duration, or if the users had not just autism but additional learning difficulties (below average IQ). Finally, it could reasonably be objected that learning to recognize emotions in the simplified context of a computer screen, devoid of the ‘noise’ of a real social situation like a school playground or a birthday party or an argument, is likely to be both simplified and therefore easier to achieve. This objection is important since it raises the question of whether such learning from ‘artificial’ contexts generalizes to more natural settings. Guarding against the risk of artificiality, Mindreading used real faces rather than cartoon or schematic faces. However, future work using the Mindreading DVD could assess the benefits of a longer intervention than just 10 weeks. The DVD could also be used with more interactive teaching methods such as discussion of each emotion in small groups, or as part of dramatic role-play.

Difficulties with generalization have been found both in computer-based intervention programmes (Silver & Oakes 2001; Bölte et al. 2002) and in social skills training courses (Bauminger, 2002; Barry et al. 2003). The limited effectiveness of these interventions could be related to a lack of intrinsic motivation, since they use explicit rather than implicit teaching methods. The study that will be reviewed next and that is reported in detail elsewhere (Golan et al. in press) evaluates the effectiveness of another DVD, this time an animation series created to motivate young children with ASC to learn about emotions and facial expressions by embedding them in a world of mechanical vehicles. Although this animation series is not computer based (it can be watched on a regular TV or DVD player), it is again relevant to the notion of ‘affective computing’ in that emotional expressions are grafted onto vehicles that move systematically. Once again, it harnesses the predictability of systems.

4. The transporters DVD

This DVD was again based on the premise that the reason children with ASC love to watch films about vehicles (according to parental report) may be because they are strong ‘systemizers’ (Baron-Cohen 2006, 2008). That is, they are drawn to predictable, rule-based systems, whether these are repeating mathematical patterns, repeating electrical patterns (e.g. light switches) or repeating patterns in films. Kanner's first descriptions of autism drew attention to their ‘need for sameness’ and their ‘resistance to change’ (Kanner 1943). At the core of autism may be an ability to deal effortlessly with systems because they do not change and produce the same outcome every time; and by the same token, a disabling difficulty to deal with the social world because it is always changing unpredictably and because the outcome is different every time.

According to the hyper-systemizing theory (Baron-Cohen 2006), vehicles whose motion is determined only by physical rules (such as vehicles that can only go back and forth along linear tracks) would be much preferred by children with autism over vehicles like planes or cars whose motion could be highly variable, moving at the whim of the human driver operating them. In vision neuroscience, this relates to the distinction between physical-causal/mechanical motion (Michotte 1963) versus animate/biological motion (Premack 1990; Castelli et al. 2000). The former requires intuitive physics (Wellman & Inagaki 1997; Saxe et al. 2004), while the latter requires intuitive psychology, in particular the ability to detect others' goals, desires and intentions (Baron-Cohen 1995).

We therefore created a children's animation series, The Transporters (www.thetransporters.com), based around eight characters who are all vehicles that move according to rule-based motion. Onto these vehicles we grafted real-life faces of actors showing emotions. We tested whether creating an autism-friendly context of predictable mechanical motion could render facial expressions of emotion more learnable and increase the motivation to learn them. The different toy vehicles (two trams, two cable cars, a chain ferry, a coach, a funicular railway and a tractor) had motion that was constrained in a linear manner (all the vehicles moved on tracks or cables).

The Transporters is a high-quality three-dimensional children's animation series and consists of 15 five-minute episodes, each of which focuses on a key emotion or mental state. This is relevant to this special issue's focus on ‘affective computing’ in exploiting a cutting-edge graphic animation technique called ‘live action’. This allowed real human faces to be superimposed on animated vehicles. Using real faces was a design feature again chosen to maximize generalization to the real social world. The 15 key emotions depicted on the vehicles are happy, sad, angry, afraid, disgusted, surprised, excited, tired, unfriendly, kind, sorry, proud, jealous, joking and ashamed. The emotions selected include the six ‘basic’ emotions (Ekman 1999), emotions that are more ‘complex’ but still developmentally appropriate (e.g. jealous, proud, ashamed) and emotions and mental states that are important for everyday social functioning (e.g. kind, unfriendly, tired, joking). These emotions were chosen because typically developing children recognize and understand these between 2 and 7 years of age (Bretherton & Beeghly 1982; Ridgeway et al. 1985).

In the study by Golan et al. (in press), three groups were assessed twice: at time 1 and then after four weeks at time 2. In each assessment participants were tested at four levels of generalization, one testing participants' emotional vocabulary and the other three testing their ability to match a socio-emotional situation to the appropriate facial expression. Level 1: emotional vocabulary: participants were asked to define 16 emotion words and give examples of situations that evoked them. These were the 15 key emotions from the series (see above), in addition to worried. Level 2: situation–facial expression matching: this was tested using three tasks, each consisting of 16 items (one for each emotion). Each item included a photograph depicting a scene with a short description.

The three tasks represented three levels of generalization: (i) Familiar close generalization: participants had to match familiar situations taken from the intervention series to facial expressions of familiar characters from the series. (ii) Unfamiliar close generalization: participants had to match novel situations with novel expressions from The Transporters characters. These expressions were not shown by these characters in the intervention series. (iii) Distant generalization: to test generalization to facial expressions that are not attached to vehicles, participants had to match novel situations with novel expressions using a selection of human non-Transporters faces taken from Mindreading software (Baron-Cohen et al. 2004). Examples of items from Levels 1 to 3 are shown in figure 2.

Figure 2.

Examples of questions from two of the three emotion recognition task levels. (a) Level 1 task: match familiar scenes from the series with familiar faces. (b) Level 3 task: match novel scenes and faces using real human faces.

Three groups took part in the study: an ASC intervention group, an ASC control group and a typically developing control group. Participants in the two clinical groups were randomly assigned and took part according to the following test conditions: (i) ASC intervention group: the parents of 20 participants were given the intervention series and DVD guide to use with their child at home. Children were asked to watch at least three episodes per day over a period of four weeks. (ii) ASC control group: nineteen participants did not participate in any intervention during the four-week interval, except for their standard school curriculum. One participant dropped out of the study after the first session. (iii) Typical control group: eighteen participants were recruited for this group. The three groups were matched for sex, age and verbal ability (using the British Picture Vocabulary Scale (BPVS); Dunn et al. 1997).

At time 1, there were significant differences between groups on the emotional vocabulary task and on the three situation–expression matching tasks. These differences were due to the significantly higher scores of the typical controls on all tasks compared to the two clinical groups, which did not differ from each other. Analysis of results after time 2 testing revealed significant time by group interactions, with the ASC intervention group showing significant improvement across all task levels between time 1 and time 2. Furthermore, this improvement was comparable to levels of performance found in the typical control group. In contrast, the ASC and typical control groups showed no significant improvement on any of the tasks between test sessions. These effects are illustrated in figure 3.

Figure 3.

Graphs to show mean scores (with standard error bars) for each group on the four tasks. (a) Situation–expression matching task—level 1. (b) Situation–expression matching task—level 2. (c) Situation–expression matching task—level 3. (d) Emotional vocabulary task. *p < 0.001. Blue, ASC intervention group; Magenta, ASC control group; yellow, typical control group.

5. Implications

The study we have reviewed (reported in detail in Golan et al. in press) investigated the effectiveness of individual use of The Transporters animated series (with parental support) over a four-week period. The results show that use of the DVD led children with ASC to improve significantly in their emotion comprehension and recognition skills on tasks including the emotions presented by The Transporters: from the same level of ability seen with the ASC control group at time 1 to a level that was indistinguishable from the typically developing group at time 2.

The improvement of the intervention group was not limited to tasks that required close generalization; these participants were also able to generalize their knowledge to perform at the level of typical controls on the distant generalization task, which required emotion recognition from naturalistic clips of human characters that were not attached to vehicles. The Transporters may have facilitated generalization because the series was designed using intrinsically motivating media, such that the children enjoyed watching the vehicles while learning about emotions from real faces grafted onto them (incidental rather than explicit learning). The Transporters used characters and an environment that appealed to a preference for order, systems and predictability that is characteristic of ASC. Anecdotal evidence from the parents of the intervention group suggests that their children became more willing to discuss emotions, and became more interested in facial expressions. Parents also noticed a change in their children's behaviour and in their ability to interact with others. Such anecdotal changes need formal evaluation.

We expect that the integration of The Transporters with other educational or therapeutic methods for children with ASC will improve its effect even further. We conclude that the use of systemizing as an intrinsically motivating method for learning about empathy allows affective information, which would otherwise be confusing, to become more intelligible and appealing to the autistic mind.

If The Transporters is having such a positive effect on the learnability of emotional expressions by children with ASC, might there be other ways to harness the same preference for systemizing in the teaching of emotions to these children? Clearly, vehicles are not the only kind of systems that children with autism enjoy, and others might include robots (Dautenhahn & Werry 2004) or rules (Hadwin et al. 1996). We believe these sorts of interventions as part of an adaptation of the mainstream environment to be more suited to people with autism, and such environmental adaptations need not be restricted to the teaching of emotions. An example outside the emotional domain might involve sensory perception, where people with ASC may experience hypersensitivity (Ashwin et al. 2009; Baron-Cohen et al. 2009). In this case, as with the use of specifically oriented media for the teaching of emotions, classrooms may need to be specially designed to ensure that information, which may otherwise be easily processed by the neurotypical brain, is not overstimulating and therefore aversive to the autistic brain. We cannot expect learning to proceed smoothly or even to occur at all if the information is in a form that causes distress or is even painful. We conclude that a little empathy on the part of designers of educational resources may therefore facilitate the development of empathy in children with autism.

Acknowledgements

All the authors were affiliated with the Autism Research Centre at University of Cambridge during the study. This work was conducted in association with the NIHR CLAHRC for Cambridgeshire and Peterborough NHS Foundation Trust. Parts of this paper appeared in the Journal of Autism and Developmental Disorders (Golan et al. in press). We are grateful to Yael Granader, Suzy McClintock, Kate Day and Victoria Leggett for help with data collection and to Gina Owen and Kimberly Peabody and Ben Weiner for useful discussions. Ofer Golan now holds a position at Bar-Ilan University, Faculty of Social Sciences, Department of Psychology in Israel. We are grateful to Culture Online and the Department for Culture, Media and Sport (DCMS) for funding and to Catalyst Ltd and Culture Online (particularly Claire Harcup, Paul Bason, Khairoun Abji and Jon Drori) for their production of the film. We are also indebted to the families who participated in the intervention. The Transporters series was nominated for a BAFTA in the Children's Awards Category, November 2007.

Footnotes

References

View Abstract