Speaker gaze effects on language comprehension

Acronym: 
SGELC
Term: 
2009-05 till 2012-10
Research Areas: 
C
B
Abstract: 

Seeing where a speaker is looking provides a valid cue about what s/he will refer to next. This project uses eye tracking to examine the time course with which such information is integrated in the unfolding comprehension process, and the sub-tasks which it affects. Ultimately, a better understanding of speaker gaze effects across different speaker-listener settings and various sentence types will help to extend existing processing accounts of situated comprehension and can inform the construction of human-computer interfaces.

Methods and Research Questions: 

In language comprehension, a large body of research has shown that listeners can rapidly integrate the unfolding speech content with information in visual context. The shifting focus of another person’s gaze is one cue that could be highly informative for listeners, because speakers robustly gaze at objects before mentioning them.

Speaker gaze can allow listeners to anticipate what the speaker will talk about next, thus benefitting listeners’ comprehension. Our studies identify the specific factors influencing how speaker gaze is used, and determine the extent to which its effects generalize across different settings. Also, while we know that speaker gaze can facilitate language comprehension, we don’t know yet whether its usefulness is independent of sentence difficulty and comprehension tasks. If speaker gaze is beneficial across the board, we should see similar effects for different sentence structures (e.g., German subject-verb-object and non-canonical object-verb-subject order), and also for different comprehension tasks.

In addition, in all other studies to date the speaker has been facing the listener fully frontally. In many everyday situations however, a fully frontal view of the interlocutor’s eyes is not available. Since the accuracy of gaze detection is known to decrease at an angle between viewer and gazer, it is important to establish whether the benefit of seeing the speaker actually generalizes to situations where interlocutors are positioned at an angle to each other.

Participants see videos in which SecondLife characters that are displayed on a screen are referred to in a spoken sentence. In half the trials, they also see the speaker of the sentence looking at these characters. Each sentence describes an action involving the central character (e.g., the waiter in the example image) and one of the two outer characters on the screen (e.g., the millionaire). The sentences differ with regard to comprehension ease, depending on whether they use a subject- or an object-initial structure. In the videos, the speaker fixates the first-mentioned central character just before producing the sentence. Then during the sentence, she shifts her gaze to the referent of the second noun phrase (e.g., the millionaire).

We measure participants’ reaction times across a range of comprehension tasks: judging sentence difficulty (Experiment 1), identifying referents (Experiment 2) or determining sentence role relations (Experiments 3 and 4). Our current Experiment 5 directly compares two comprehension tasks: reference assignment vs. verification of role relations. Critically, we record participants’ eye movements throughout the entire trial (EyeLink 1000). The analyses focus on fixations to the second-mentioned character following the speaker’s gaze shift, comparing both the relative number of fixations and mean fixation onset times per condition.

Outcomes: 

Using this paradigm, we have shown for the first time that

  1. even viewed from an angle, speaker gaze can rapidly influence visual attention in a listener: listeners looked at the second-mentioned character earlier when the speaker was (versus wasn’t) visible;
  2. effects of speaker gaze on visual attention are not independent of incremental syntactic structure building and thematic role assignment: for “easy” subject-initial sentences, speaker gaze had a stronger effect on listeners’ anticipation of the second-mentioned character than for object-initial sentences;
  3. effects of speaker gaze on visual attention are sensitive to small differences between comprehension tasks (e.g., identifying the characters vs. locating the experiencer of the action).

However, despite this substantial facilitation through speaker gaze on a listener's visual attention during comprehension, the effects seem to be short-lived and do not affect post-comprehension response times (at least in the tasks we have employed to date).

Publications: