Research at Discourse & Multimodality Lab
When humans use language, their utterances are typically embedded in a larger context. That is, we rarely use words or sentences in isolation; instead, our utterances are always part of a connected stretch of discourse (e.g., a story we are telling, a task we are explaining, or simply a conversation we are having). Furthermore, communication with language is always multimodal. Spoken language is combined with visible bodily action (hands, body and face), and written language is combined with other visual design elements (such as punctuation, typographic choices, layout, and images). At the Discourse and Multimodality Lab, we examine how different modalities interact in language production processes at the intersection of the semantic, pragmatic, cognitive levels. Our particular aim is to uncover how non-lexical elements contribute to our understanding of the human communication system.
To this end, we conduct empirical studies by using different data gathering techniques, which range from the recording of spontaneous speech events (e.g., talk-in-interaction), elicited speech events (e.g., narrative retellings based on carefully chosen stimuli) to more experimental set ups using both offline and online methods (e.g., preference judgment tasks). We also work with written discourse, which we elicit through written production tasks or extract from text corpora across different genres (e.g., picture books, detective stories), as well as with scripted language (e.g., as used in movies). When necessary, we integrate computational tools and annotation technologies to systematically capture the temporal coordination of the various modalities being integrated in multimodal packages to support large-scale, machine-assisted analysis. Special attention is given to macro- and micro-structural properties of long(er) stretches of sustained discourse and how they influence multimodal language behaviour.
Areas of research
- Anaphoric expressions
- Cognitive compression
- Cognitive sociolinguistics
- Conceptual Metaphor Theory
- Deixis
- Dialogic Syntax
- Discourse acts and moves
- Discourse markers/particles
- Discourse strategies in oral epic traditions and in storytelling
- Documentary papyrology
- Figurative language
- Iconicity
- Insubordination
- Morphosyntax
- Multimodality in songs
- Punctuation
- Pragmatics
- Subordination
- Segmentation in language
PhD dissertations
- (finished) “A Basket of Mysteries” A Cognitive Linguistic Analysis of Rita Dove’s reworking of Greek myths
The project focuses on the reuse of ancient myth in the works of the African-American poet Rita Doves, from a cognitive perspective. The research and the linguistic analysis strongly revolve around the cognitive mappings emerging from Rita Dove’s narrative constructs, names, and words, with respect to the ancient Greek texts. Central concepts of cognitive-linguistic studies, such as the Conceptual Metaphor Theory and Conceptual Integration Theory are applied in order to explore cognitive models and structures at the base of Rita Dove’s creative process of re-elaboration, re-conceptualization, and re-adapation of mythical episodes and stories.
- (finished) Analyzing Tag Questions in Jordanian Arabic Talk-in-Interactions from a Multimodal Perspective
Based on transcribed audio and video recordings of casual face-to-face Jordanian Arabic conversation of 24 university students in Irbid City, this study explores the role of linguistic structures, prosodic and visual features as well as their joint effect on the construction of the different forms and interactional functions of tag questions in three Jordanian Arabic dialects. This study adopts the methodologies of conversation analysis and interactional linguistics and aims to overcome the lack of research on multimodality and tag questions in Jordanian Arabic interaction.
The project revolves around multimodal communication and in particular on multimodal and cross-modal references to objects in forensic science movies. The focus is how words mentioning and referring to objects are linked with diegetic images depicting the objects as well as diegetic sounds produced by the objects to constitute anaphoric expressions in co-referential chains with respect to the viewers’ perception. By implementing semantic, pragmatic, and cognitive frameworks, in combination with concepts related to telecinematic discourse, this study explores the complex processes the viewers undergo that lead them to construe the meaning of multimodal cues and connect the modal elements anaphorically to the given referents.
The project focuses on figurative and non-figurative conceptions of loss, grief and coping in picturebooks in the German and English language. Instead of investigating just the conceptualisation of death, the study embraces the discursive and dynamic dimensions of the cognitive-emotional processes around someone's death as stories unfold: (i) the sense of loss, (ii) the process of grieving, and (iii) the process of coping. As multimodally designed objects, picturebooks are particularly useful for an analysis: they convey meaning through each mode and across modes.
Preliminary studies
- Manual annotations of more than 3.500 anaphoric expressions with people as referents in 6 short crime stories (annotations by Pinelopi Ioannidou and Zala Salarzai; coordination by Anna Bonifazi)
Cooperations
- Cologne Myth network
After its inaugurating meeting in February 2019, the international and interdisciplinary research group “Cologne Mythological Network” was established with the aim of bringing together scholars with different academic backgrounds who share a common interest in myth and mythology. The first outcome of the group’s work has been the research seminar series Ancient Myths in Modern Art: Comparative Perspectives on Micro- and Macro-Structures, organized by (Prof. Anna Bonifazi) and Comparative Literature (Prof. Joachim Harst). The research seminar series has not only brought together speakers from different departments of the University of Cologne, but also from other academic institutions in Cologne, as well as from other parts of Germany and Europe. The group is currently planning further events dealing with myth and mythology, maintaining its highly interdisciplinary perspective and international character; anyone interested in past and future activities and events may contact Riccardo Ginevra (riccginevra[at]gmail.com) and Penelope Kolovou (penelope.kolovou[at]uni-bonn.de), coordinators of the group.
