Co-reference and coherence in visual narratives

This research delves into questions of co-reference: knowing that two things are the same object/figure. Co-reference is vital for creating a cohesive, logical narrative.

See the two images on this page: To understand these as part of one story, readers must relate the figures and events together logically.

How do we recognize the environment is the same or different? How do we know that this is the same person, despite the differences in appearance? And how do we unite these two actions into a flowing narrative, occurring across time & space? These questions may seem easy, but research has shown that people not familiar with visual stories struggle to make these connections. It seems you have to learn certain ‘rules’ to understand comic narratives.

Publications

We published a theory that explains what readers have to learn to make co-reference in visual narratives. This co-reference across locations, figures, and actions provides the basis for a coherent story.

We investigated patterns of co-reference in published comics, investigating a corpus of over 300 comics from Europe, Asia, and the United States. We found these patterns to differ across regions, which may show that readers from different places would have different expectations for processing comics. 

 Finally, we tested what happened if a story was no longer coherent, and readers had to infer a major event by themselves. We replaced this major event with panels showing something related but not really informative, such as an action star, a sound effect, or metaphoric event. We found that the type of event matters for how well readers can infer events.