Publications
2007
Kelleher, J.D. DIT - Frequency Based Incremental Attribute Selection for GRE. In Proceedings of the MT Summit XI Workshop Using Corpora for Natural Language Generation: Language Generation and Machine Translation (UCNLG+MT), Pages 90-92. Blez, A. and Varges, S. (eds).
[pdf]
[bib]
[shared task website]
[shared task evaluation (pdf)]
@InProceedings{kelleher:07b,
author = {Kelleher, J.D.},
title = {DIT - Frequency Based Incremental Attribute Selection for GRE.},
booktitle = {In Proceedings of the MT Summit XI Workshop Using Corpora for Natural Language Generation: Language Generation and Machine Translation (UCNLG+MT)},
year = {2007},
pages = {90-92},
}
[hide]
Kelleher, J.D. Attention driven reference resolution in multimodal contexts. Artificial Intelligence Review 25:21-35.
[pdf]
[abstract]
[bib]
In recent years a a number of psycholinguistic experiments have pointed to the interaction between language and vision. In particular, the interaction between visual attention and linguistic reference. In parallel with this, several theories of discourse have attempted to provide an account of the relationship between types of referential expressions on the one hand and the degree of mental activation on the other. Building on both of these traditions, this paper describes an attention based approach to visually situated reference resolution. The framework uses the relationship between referential form and preferred mode of interpretation as a basis for a weighted integration of linguistic and visual attention scores for each entity in the multimodal context. The resulting integrated attention scores are then used to rank the candidate referents during the resolution process, with the candidate scoring the highest selected
as the referent. One advantage of this approach is that the resolution process occurs within the full multimodal context, in so far as the referent is selected from a full list of the objects in the multimodal context. As a result situations where the intended target of the reference is erroneously excluded, due to an individual assumption within the resolution process, are avoided. Moreover, the system can recognise situations where attention cues from different modalities make a reference potentially ambiguous.
[hide]
@article{kelleher:07,
author = {Kelleher, J.D.},
title = {Attention driven reference resolution in multimodal contexts},
journal = {Artificial Intelligence Review},
volume = {25}
pages = {21-35}
year = {2007},
}
[hide]
Costello, F., Kelleher, J.D. and Volk, M. (eds.) Proceedings of the 4th ACL-SIGSEM Workshop on Prepositions at ACL-2007. June 28, Prague, Czech Republic.
[workshop url]
[bib]
@Proceedings{costelloetal:07,
editorr = {Costello, F., Kelleher, J.D. and Volk, M.},
title = {Proceedings of the 4th ACL-SIGSEM Workshop on Prepositions at ACL-2007},
address = {Prague, Czech Republic},
month = {June 28},
year = {2007},
publisher = {The Association of Computational Linguistics},
}
[hide]
Brenner, M., Hawes, N., Kelleher, J.D. and Wyatt, J. (2007) Mediating between qualitative and quantitative representations for task-oriented human robot interaction. In Proceedings of IJCAI-07, Hyderabad, India.
[pdf]
[abstract]
[bib]
In human-robot interaction (HRI) it is essential that the robot interprets and reacts to a human's utterances in a manner that reflects their intended meaning. In this paper we present a collection of novel techniques that allow a robot to interpret and execute spoken commands describing manipulation goals involving qualitative spatial constraints (e.g. "put the red ball near the blue cube"). The resulting implemented system integrates computer vision, potential field models of spatial relationships, and action planning to mediate between the continuous real world, and discrete, qualitative representations used for symbolic reasoning.
[hide]
@InProceedings{brenneretal:07,
author = {Brenner, M., Hawes, N., Kelleher, J.D. and Wyatt, J},
title = {Mediating between qualitative and quantitative representations for task-oriented human robot interaction},
booktitle = {In Proceedings of IJCAI-07},
year = {2007},
location = {Hyderabad, India},
}
[hide]
2006
Kelleher, J., van Genabith, J. (2006) A Computational Model of the Referential Semantics of Projective Prepositions. Syntax and Semantics of Prepositions, (ed.) P. Saint-Dizier. Kluwer Publishing.
[pdf]
[abstract]
[bib]
In this paper we present a framework for interpreting locative expressions containing the prepositions in front of and behind. These prepositions have different semantics in the viewer-centred and intrinsic frames of reference (Vandeloise, 1991). We define a model of their semantics in each frame of reference. The basis of these models is a novel parameterized continuum function that creates a 3-D spatial template. In the intrinsic frame of reference the origin used by the continuum function is assumed to be known a priori and object occlusion does not impact on the applicability rating of a point in the spatial template. In the viewer-centred frame the location of the spatial template's origin is dependent on the user's perception of the landmark at the time of the utterance and object occlusion is integrated into the model. Where there is an ambiguity with respect to the intended frame of reference, we define an algorithm for merging the spatial templates from the competing frames of reference, based on psycholinguistic observations in (Carlson-Radvansky, 1997).
[hide]
@InBook{kelleher/vangenabith06,
author = {John Kelleher and Josef van Genabith},
editor = {Patrick Saint-Dizier},
title = {Syntax and Semantics of Prepositions},
chapter = {A Computational Model of the Referential Semantics of Projective Prepositions},
year = {2006},
}
[hide]
Kelleher, J. and Kruijff, G.J. (2006) Incremental Generation of Spatial Referring Expressions in Situated Dialogue. In Proceedings of COLING-ACL'06. Sydney, Australia. Association of Computational Linguistics.
[pdf]
[abstract]
[bib]
This paper presents an approach to incrementally generating locative expressions. It addresses the issue of combinatorial explosion inherent in the construction of relational context models by: (a) contextually defining the set of objects in the context that may function as a landmark, and (b) sequencing the order in which spatial relations are considered using a cognitively motivated hierarchy of relations, and visual and discourse salience.
[hide]
@InProceedings{kelleher/kruijff:06,
author = {John Kelleher and Geert-Jan Kruijff},
title = {Incremental Generation of Spatial Referring Expressions in Situated Dialogue},
booktitle = {In Proceedings of Coling-ACL '06, Sydney Australia},
year = {2006},
organisation = {Association of Computational Linguistics},
}
[hide]
Kelleher, J. and Kruijff, G.J. and Costello, F. (2006) Proximity in Context: an empirically grounded computational model of proximity for processing topological spatial expression. In Proceedings of COLING-ACL'06. Sydney, Australia. Association of Computational Linguistics.
[pdf]
[abstract]
[bib]
The paper presents a new model for context-dependent interpretation of linguistic expressions about spatial proximity between objects in a natural scene. The paper discusses novel psycholinguistic experimental data that tests and verifies the model. The model has been implemented, and enables a conversational robot to identify objects in a scene through topological spatial relations (e.g. ''X near Y''). The model can help motivate the choice between topological and projective prepositions.
[hide]
@InProceedings{kelleher/etal:06,
author = {John Kelleher and Geert-Jan Kruijff and Fintan Costello},
title = {Proximity in Context: an empirically grounded computational model of proximity for processing topological spatial expression},
booktitle = {In Proceedings of Coling-ACL '06, Sydney Australia},
year = {2006},
organisation = {Association of Computational Linguistics},
}
[hide]
Kruijff, G.J. and Kelleher, J. and Hawes, N. (2006) Information Fusion For Visual Refer ence Resolution In Dynamic Situated Dialogue. In Proceedings of Perception and Interactive Technologies (PIT06). Kloster Irsee, Germany. LNCS/LNAI/LNBI Series by Springer Verlag. Eds. Andre, E., Baratoff, G., Dybkjaer, L., Hennecke,
M., Minker, W., Neumann, H. and Weber, M.
[pdf]
[abstract]
[bib]
Human-Robot Interaction (HRI) invariably involves dialogue about objects in the environment in which the agents are situated. The paper focuses on the issue of resolving discourse references to such visual objects. The paper addresses the problem using strategies for intra-modal fusion (identifying that different occurrences concern the same object), and inter-modal fusion, (relating object references across different modalities). Core to these strategies are sensori-motoric coordination, and ontology-based mediation between content in different modalities. The approach has been fully implemented, and is illustrated with several working examples
[hide]
@InProceedings{kruijff/etal:06,
author = {Geert-Jan Kruijff and John Kelleher and Nick Hawes},
title = {Information Fusion For Visual Refer ence Resolution In Dynamic Situated Dialogue},
booktitle = {In Proceedings of Perception and Interactive Technologies (PIT’06)},
year = {2006},
editor = {Elisabeth Andre and Laila Dybkjaer and Wolfgang Minker and Heiko Neumann and Michael Weber},
publisher = {Springer Berlin / Heidelberg},
month = {June},
ee = {http://www.springerlink.com/content/e25jn35776r02327/},
series = {Lecture Notes in Computer Science},
volume = {4021},
address = {Kloster Irsee, Germany},
pages = {117 -- 128}
}
[hide]
Costello, F. and Kelleher, J. (2006) Spatial Prepositions in Context: The Semantics of near in the Presence of Distractor Objects. In Proceedings of the 3rd ACL-Sigsem Workshop on The Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications, Association for Computational Linguistics. Eds. Arsenijevic, B., Baldwin, T. and Trawinski, B.
[pdf]
[bib]
@InProceedings{costello/kelleher:06,
author = {Fintan Costello and John Kelleher},
title = {Spatial Prepositions in Context: The Semantics of near in the Presence of Distractor Objects},
booktitle = {In Proceedings of the 3rd ACL-Sigsem Workshop on The Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications},
year = {2006},
organisation = {Association of Computational Linguistics},
editor = {Boban Arsenijevic and Tim Baldwin and Beata Trawinski}
}
[hide]
Kruijff, G.J. and Kelleher, J. and Berginc, G. and Leonardis, A. (2006) Structural descriptions in human-assisted robot visual learning. In Proceedings of the 1st Annual Conference on Human-Robot Interaction (HRI06). Salt Lake City UT, USA.
[pdf]
[abstract]
[bib]
The paper presents an approach to using structural descriptions, obtained through a human-robot tutoring dialogue, as labels for the visual ob ject models a robot learns. The paper shows how structural descriptions can relate models for different aspects of one and the same ob ject, and how relating descriptions for visual models and discourse referents enables incremental updating of model descriptions through dialogue (either robot- or human-initiated). The approach has been implemented in an integrated architecture for human-assisted robot visual learning.
[hide]
@InProceedings{kruijff/etal:06b,
author = {Geert-Jan Kruijff and John Kelleher and Gregor Berginc and Ales Leonardis},
title = {Structural descriptions in human-assisted robot visual learning},
booktitle = {in Proceedings of the 1st Annual Conference on Human-Robot Interaction (HRI’06). Salt Lake City, UT USA.},
year = {2006},
}
[hide]
2005
Kelleher, J.D. and Costello, F. and van Genabith, J. Dynamically Updating and Interrelating Representations of Visual and Linguistic Discourse. Artificial Intelligence 167 62-102.
[pdf]
[abstract]
[bib]
The fundamental claim of this paper is that salience - both visual and linguistic - is an important overarching semantic category structuring visually situated discourse. Based on this we argue that computer systems attempting to model the evolving context of a visually situated discourse should integrate models of visual and linguistic salience within their natural language processing (NLP) framework. The paper highlights the importance of dynamically updating and interrelating visual and linguistic discourse context representations. To support our approach, we have developed a real-time, natural language virtual reality (NLVR) system (called LIVE, for Linguistic Interaction with Virtual Environments) that implements an NLP framework based on both visual and linguistic salience. Within this framework saliency information underpins two of the core subtasks of NLP: reference resolution and the generation of referring expressions. We describe the theoretical basis and architecture of the LIVE NLP framework and present extensive evaluation results comparing the system's performance with that of human participants in a number of experiments.
[hide]
@Article{kelleher/etal:05,
author = {John Kelleher and Fintan Costello and Josef van Genabith},
title = {Dynamically Structuring Updating and Interrelating Representations of Visual and Linguistic Discourse},
journal = {Artificial Intelligence},
year = {2005},
volume = {167},
pages = {62-102},
}
[hide]
Kelleher, J. (2005) Integrating Visual and Linguistic Salience for Reference Resolution, In Proceedings of the 16th Irish conference on Artificial Intelligence and Cognitive Science (AICS-05). Ed. Creaney, E. Portstewart, Northern Ireland.
Kelleher, J. and Kruijff, G.J. (2005) A context-dependent algorithm for generating locative expressions in physically situated environments., In Processdings of the 10th European Workshop on Natural Language Generation. Eds. Mellish, C. and Reiter, E. and Jokinen, K. and Wilcock, G. SIGGEN. Aberdeen, Scotland.
Kelleher, J., (2005) Visual Salience and the Other One, In Proceedings of the 6th International Workshop on Multidisciplinary Approaches to Discourse: Salience in Discourse. Eds. Stede, M. and Chiarcos, C. and Grabski, M. and Lagerwerf, L. Pages. 79-88. Publisher: Stichting Neerlandistiek Amsterdam & Nodus Publikationen Munster. Chorin/Berlin, Germany.
Kelleher, J. and Kruijff, G.J. (2005) A context-dependent model of proximity in physically situated environments. In Proceedings of the 2nd ACL-Sigsem Workshop on The Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications. Eds. Kordoni, V. and Villavicencio,
A. ACL-Sigsem, Colchester, U.K.
Kelleher, J. and Costello, F. (2005) Cognitive Representations of Projective Prepositions. In Proceedings of the 2nd ACL-Sigsem Workshop on The Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications. Eds. Kordoni, V. and Villavicencio, A. ACL-Sigsem, Colchester. U.K.
[pdf]
[abstract]
[bib]
This paper describes a psycholinguistic experiment that investigates the cognitive representations of the projective prepositions in front of and behind. In particular: (1) what is the constituency of
the spatial template (i.e., the prototypical regions of acceptability) associated with these prepositions; (2) what effect does frame of reference ambiguity have a these regions; (3) do these preposi-
tions exhibit a bias towards a particular frame of reference. Our results indicate that: (1) the spatial templates associated with these prepositions correlate with projective spatial templates for above, below, left and right, reported in (Logan and Sadler, 1996); (2) when frame of reference ambiguity occurs these spatial template are modified in a similar manner to that reported in (Carlson-Radvansky and Logan, 1997) for the preposition above; (3) in contrast with the results of previous work (Carlson-Radvansky and Irwin, 1993; Carlson-Radvansky and Logan, 1997) which indicate a preference towards the absolute and viewer-centred frames of reference for the preposition above, the prepositions in front of and behind exhibit a bias towards the landmark's intrinsic frame of reference.
[hide]
@InProceedings{kelleher/costello:05,
author = {John Kelleher and Fintan Costello},
title = {Cognitive Representations of Projective Prepositions},
booktitle = {In Proceedings of the 2nd ACL-Sigsem Workshop on The Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications.},
year = {2005},
organisation = {Association of Computational Linguistics},
editor = {Valia Kordoni and Aline Valencia}
}
[hide]
2004
Kelleher, J., van Genabith, J. (2004) Visual Salience and Reference Resolution in Simulated 3-D Environments. Artificial Intelligence Review, Kluwer Publishing. Vol. 21(3).
[pdf]
[bib]
Hawes, N., Kelleher, J. (2004) Context Sensitive Word Selection For Single-Tap Text Entry. Accepted for the 2nd European Starting Artificial Intelligence Researchers Symposium at the 16th European Conference on Artificial Intelligence (ECAI'04). Valencia, Spain. IOS Press.
[pdf]
[bib]
Hawes, N., Kelleher, J. (2004) Analogy by Alignment: On Structure Mapping and Similarity. Accepted for the 2nd European Starting Artificial Intelligence Researchers Symposium at the 16th European Conference on Artificial Intelligence (ECAI'04). Valencia, Spain. IOS Press.
[pdf]
[bib]
Kelleher, J., van Genabith, J. (2004) Exploiting Visual Salience for the Generation of Referring Expressions. In Proceedings of at the 17th International FLAIRS conference. Miami, Florida. AAAI Press.
[pdf]
[bib]
Inverso, S.A., Hawes, N., Kelleher, J., Allen, R. and Haase, K. (2004) Think And Spell: Context-Sensitive Predictive Text for an Ambiguous Keyboard Brain-Computer Interface Speller. Biomedizinische Technik, 49(1), pages 53-54.
[pdf]
[bib]
2003
Kelleher, J. (2003). A Perceptually Based Computational Framework for the Interpretation of Spatial Language in 3D Simulated Environments, Unpublished Ph.D., Dublin City University, Dublin.
[pdf]
[bib]
Kelleher, J., van Genabith, J. (2003) A False Colouring Real Time Visual Saliency Algorithm for Reference Resolution in Simulated 3-D Environments. Proceedings of the 14th Irish Conference on Artifical Intelligence and Cognitive Science (AICS'04). Trinity College Dublin, Ireland.
Kelleher, J., van Genabith, J. (2003) A Computational Model of the Referential Semantics of Projective Prepositions. Proceedings of ACL-SIGSEM Workshop: The Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications, Toulouse, France.
2002...
Kelleher, J., O Nuallain, S., (2001) The SONAS Algorithm for the Interpretation of Spatial Prepositions. Proceedings of Seventh International Colloquium on Cognitive Science (ICCS01). Donostia San Sebastian, Spain.
Kelleher, J., Doris, T., et al. (2000). SONAS: Multi-modal, Multi-user Interaction with a Modelled Environment. Spatial Cognition - Foundation and Applications. (ed.) S. Ó Nualláin. Amsterdam/Philadelphia, John Benjamins Publishing.
ONuallain, S., Kelleher, J., (1998) Spoken Image meets VRML and Java. Proceeding of MIND 2: Spatial Cognition, Dublin City University, Ireland.
Invited Talks
Handling Spatial References in Visually Situated Discourse. Presented at the special session on Visual Salience and Reference to the Visual Scene at the 10th Workshop on the Semantics and Pragmatics of Dialogue (Brandial06). University of Potsdam, Germany. 12th September 2006.
[pdf]
Media Lab Europe and the Atelier Model: A Strategy for Innovation and Education. Presented at the Association of Northern Ireland Colleges Annual Conference (ANIC'04). The Manor House, Co. Fermanagh, Ireland, 6th November 2003.
A Saliency Based Semantics for Natural Language Virtual Reality Systems. Presented at the Computer Science Seminar Series, Trinity College Dublin, Dublin, Ireland, 21st November 2003.
Dynamically Updating and Interrelating Representations of Visual and Linguistic Discourse. Presented at the Second Working Meeting of the SIGSEM Working Group on the Representation of Multimodal Semantic Information, LORIA Research Centre, Nancy, France, 22-23 September 2003.
The Situated Language Interpreter: Natural Language Processing in 3D Simulated Environments. Presented at the Dublin Computational Linguistic Research Seminar Series (DCLRS), University College Dublin, Dublin, Ireland, 7th March, 2002.
