Effects of Perceptual Structures on Incremental Language Comprehension

Research Areas: 

The present research project aims to study the extent to which non-referential aspects of visual context can affect real-time comprehension of abstract sentences. It is an attempt to bridge between evidence coming from psycholinguistic and embodied language research on the role of perceptual processes in language comprehension. Using eye-tracking methodologies, we assessed the effects of different perceptual structures (e.g., distance between objects) on reading times of abstract sentences.

Methods and Research Questions: 

Psycholinguistic findings have shown that non-linguistic visual context can rapidly inform comprehension, when language references it. Embodied language research has shown that perceptual processes affect comprehension of concrete sentences. We ask whether (a) non-referential visual contexts modulate real-time language comprehension, and (b) to which extent comprehending abstract sentences involves perceptual processes.

When language refers directly to objects or events in a visual context, information from those object and events can rapidly modulate language comprehension (‘referential visual context effect’). In addition, a large body of evidence suggests that perceptual information (e.g., the shape or position of objects) is involved in language processing, at least for concrete language. For abstract language, in contrast, the relationship between concepts and perceptual information such as object shape or distance is less well examined. A potentially useful framework for informing hypotheses about visual context effects on the comprehension of abstract sentences is conceptual metaphor theory. It proposes that abstract concepts are understood in terms of more concrete ones, through metaphors.

Our first two studies examined how spatial distance between objects affects comprehension of sentences that describe similarity (vs. dissimilarity) between abstract nouns. Previous studies have shown how distance between two abstract words affects participants’ similarity judgments of these abstract words (words were judged as more similar when they were close to one another in space than far apart). Yet, it is unclear whether distance can also affect real-time comprehension of abstract sentences.

Two eye-tracking reading studies examined whether distance between objects that are not directly referred to in a sentence, can modulate incremental semantic interpretation of sentences like those in (1) and (2), which imply similarity or dissimilarity respectively

  1. Kampf(NP1) und(coord.) Krieg(NP2) sind(VP1) freilich(ADV) entsprechend(ADJ), das verriet(VP2) der Anthropologe(NP3).
    ‘Battle and war are surely analogical, so suggested the anthropologist‘.
  2. Frieden(NP1) und(coord.) Krieg(NP2) sind(VP1) bestimmt(ADV) verschieden(ADJ)...
    ‘Peace and war are certainly different…'

Experiment 1 used a visual context linked to a sentence (two playing cards displayed the first two sentential nouns) whereas Experiment 2 used unrelated, empty, cards. By tracking eye gaze we can measure which words people inspect and when; whether they inspect a word for the first time or re-read it. Thus, gaze measures can distinguish between "early" (e.g., first-pass) and "late" (e.g., re-reading) effects as a function of visual context. Inspection duration of a word is taken to reflect ease or difficulty of processing.

In our first set of experiments, we analyzed gaze duration on the noun phrases (NP1 and NP2) and the adjective phrase (ADJ), where similarity / dissimilarity is expressed. Analyses of additional regions (VP2, NP3) can reveal whether context effects extend to other sentence regions.


Results of the first two experiments suggest that distance between objects can rapidly affect reading times of sentences that imply similarity, although gaze patterns differed when the cards displayed words (Experiment 1) versus were blank (Experiment 2). In Experiment 1, we observed a reliable facilitation effect. Reading times (first pass and total times) were shorter for sentences implying similarity after seeing cards close together (vs. far apart), and vice versa for sentences implying dissimilarity. However, we also observe an interference effect with longer reading times for sentences implying similarity after seeing cards close together (vs. far apart), and vice versa for sentences implying dissimilarity. In contrast, we observed only interference effects in Experiment 2.

The rapid and extended time course with which word (Experiment 1) and card (Experiment 2) distance differentially affected semantic interpretation implicates more than just a referential mechanism, and suggests that relating spatial distance to abstract content is instantaneous and part and parcel of ongoing semantic interpretation.