Integrating Visual and Linguistic Discourse Information

Part of my Ph.D. focusing on designing a discourse framework that integrated the visual information, from the situation of the context, with the linguistic information extracted from a user's previous utterances. The main components in this framework are:

• the data structure or context model that stores relevant contextual information used to identify the referents of referring expressions,
• the set of interpretive functions that update and access the context model when resolving a reference.

By integrating visual salience with linguistic information the framework is able to resolve deictic and anaphoric references, including: definite descriptions, indefinites, pronominals, singular demonstratives accompanied by a deictic gesture (input using the mouse), other anphora, one anaphora, locative expressions, coordinating expressions.

Also, by integrating visual salience information with the representation of referents entering the dscourse model from the visual domain the system was able to resolve some references that were linguistically underspecified in the context.