A Computational Model of Concept Generalization in Cross-Modal Reference

Cross-modal interactions between visual understanding and linguistic processing substantially contribute to the remarkable robustness of human language processing. We argue that the formation of cross-modal referential links is a prerequisite for the occurrence of cross-modal interactions between vision and language. In this paper we examine a computational model for a cross-modal reference formation with respect to its robustness against conceptual underspecification in the visual modality. This investigation is motivated by the fact that natural systems are well capable of establishing a cross-modal reference between modalities with different degrees of conceptual specification. In the investigated model, conceptually under-specified context information continues to drive the syntactic disambiguation of verb-centered syntactic ambiguities as long as the visual context contains the situation arity information of the visual scene.