Picture-sentence verification in ERPs

Acronym: 
PSVERP
Term: 
ongoing
Research Areas: 
C
Abstract: 

To re-establish picture-sentence verification -discredited possibly for its over-reliance on post-sentence response times (RTs)– as a task for situated comprehension, we collected event-related brain potentials (ERPs) as participants read a subject-verb-object sentence, and RTs indicating whether the sentence matched a previous scene. Analyses of ERPs show (1) verification isn’t all post-sentence; (2) simple priming cannot accommodate the results; (3) verification tasks can inform comprehension; and (4) different visual context effects on comprehension (e.g., actions vs. who-does-what-to whom) are temporally dissociable.

 

Methods and Research Questions: 

Situated comprehension research has emphasized how visual context “facilitates” disambiguation of temporary language ambiguity. It matters, however, to examine visual context and language comprehension interactions more generally, even when these information sources don’t completely match. A potentially useful paradigm– sentence-picture verification – has been discredited, perhaps for using post-sentence response times.

In the area of situated language comprehension existing studies to date have focused on the ‘‘facilitative’’ effects of visual contexts (scenes) on language processing, e.g., how visual context might incrementally disambiguate temporary linguistic and/or referential ambiguities. It is important, however, to examine the nature and timing of visual context and language comprehension interactions more generally, even when the two information sources are not completely in accord, which, upon careful consideration, is often the case. A potentially useful paradigm for this enterprise – sentence-picture verification – has been discredited, perhaps for using post-sentence measures (response latencies, RTs) rather than continually tracking the time course of comprehension across the sentence.

To re-establish picture-sentence verification - discredited possibly for its over-reliance on post-sentence RT measures - as a task for situated comprehension, we collected event-related brain potentials (ERPs) as participants read a subject-verb-object sentence, and RTs indicating whether or not the verb matched a previously presented picture. An example image pair for one item shows a gymnast punching a journalist (Figure 1a) and a gymnast applauding a journalist (Figure 1b). These were paired with one of the following sentences:

(1a) The gymnast punches the journalist.
(1b) The gymnast applauds the journalist.

The experiment was a 1-factor (congruence) within-subjects design with two levels (incongruent versus congruent verb-action relation): Sentence (1a) “The gymnast punches the journalist” is congruent when following Figure 1a and incongruent following Figure 1b.

 

Outcomes: 

Analyses of the event-related brain potentials show that for mismatches (vs. matches), speeded RTs were longer, verb N400s over centro-parietal scalp larger, and ERPs to the object noun more negative. RTs (congruence effect) correlated inversely with the centro-parietal verb N400s, and positively with the object ERP congruence effects. Verb N400s, object ERPs, and verbal working memory scores predicted more variance in RT effects (50%) than N400s alone.

Thus, (1) verification processing is not all post-sentence; (2) simple priming cannot account for these results; and (3) verification tasks can inform studies of situated comprehension.

Publications: