Modeling the Visual and Linguistic Importance of Objects

Abstract

Previous work measuring the visual importance of objects has shown that only spatial information, such as object position and size, is predictive of importance, whilst low-level visual information, such as saliency, is not (Spain and Perona 2010, IJCV 91 , 59–76). Objects are not important solely on the basis of their appearance. Rather, they are important because of their contextual information (eg, a pen in an office versus in a bathroom), which is needed in tasks requiring cognitive control (eg, visual search; Henderson 2007, PsySci 16 219–222). Given that most visual objects have a linguistic counterpart, their importance depends also on linguistic information, especially in tasks where language is actively involved—eg, naming. In an eye-tracking naming study, where participants are asked to name 5 objects in a scene, we investigated how visual saliency, contextual features, and linguistic information of the mentioned objects predicted their importance. We measured object importance based on the urn model of Spain and Perona (2010) and estimated the predictive role of visual and linguistic features using different regression frameworks: LARS (Efron et al 2004, Annals of Statistics 32 407–499) and LME (Baayen et al 2008, JML 59 , 390–412). Our results confirmed the role of spatial information in predicting object importance, and in addition, we found effects of saliency. Crucially to our hypothesis, we demonstrated that the lexical frequency of objects and their contextual fit in the scene significantly contributed to object importance.