Data representations are critical to making analyses of multimodal interaction understandable and to demonstrate their validity. To optimally support both aspects, we argue that representations should be primarily visual so as to be as simple and as objective as possible for the goal of each analysis.
While spelling conventions facilitate orthographic representation of spoken language, no such system exists for embodied data. Gestures are complex communicative indices that are designed for specific interactional and physical contexts. Various approaches have been developed to integrate language and body into single representations (e.g. Mondada, 2014; Luff & Heath, 2015). While these systems facilitate analysis into how language and the body are symbiotically mobilized to accomplish social actions, they are difficult to read and interpret, and are labour intensive to produce. The main goal of representing data should be to optimally support the presented analysis. For multimodal representation, there is no one-size-fits-all solution. In this paper we draw on our research on video consultations in healthcare to show three ways of representing multimodal data: (i) multimodal transcriptions that include detailed representation of linguistic and embodied behaviour in line with established conventions, supported by screengrabs; (ii) transcriptions of linguistic behaviour only, supported by screengrabs; and (iii) screengrabs only without transcriptions of either linguistic or embodied behaviour. In terms of the choices behind when to use different forms of representation, our overarching principle is to use the simplest form of representation to facilitate understanding of the analysis. This means we take screengrabs as the primary means of data representation and that unless integration of language and the body is the focal point of the analysis, we refrain from using detailed transcriptions of language and the body. By focusing on a visual representation, we present the data in the most objective way available. Although selection of specific frames is inherently guided by our analytical focus, it prevents unnecessary and unforeseen interpretation of the data that are part and parcel to transcription.
