Towards Understanding the Role of Information Graphics in Multimodal Documents

Our work focuses on the understanding of the role of information graphics such as bar charts, line graphs and pie charts in multimodal texts (newspaper or magazine articles, for example). We posit that such nonpictorial visual devices are a form of language according to Clark’s view of language as any deliberate signal that is intended to convey a message. To support this claim, in two empirical studies of separate corpora of bar charts, human coders (using consensus-based annotation) were asked to identify the intended message of the graphic using a provided list of possible message schemas, and the instantiation of the parameters in the message (the begin and end points of a trend, for example). In both experiments, coders agreed on both the primary message being conveyed by the bar chart and its parameters over 90% of the time. Given the support that these studies provide for the ability of humans to consistently infer the intended message of an information graphic, it then becomes important to understand how viewers are able to do so. This should be done for at least two practical purposes: 1) to assist individuals who may not be able to accurately access the information in its original form for physical or cognitive reasons and 2) to support the design of effective information graphics. To this end, we have identified a variety of communicative signals that information graphic designers utilize, such as salience, annotations, and perceptual task effort, to enable the typical user to infer the intended message of a graph and we have performed an additional corpus study to examine the frequency and utility of these signals in supporting the conveyance of the message of an information graphic.