View source for Gestural Semiotics and App Fiction

In his theory of gestural manipulations, Bouchardon (2014; see Ensslin 2014: 82-83) starts from the assumption that “all clicking is interpretive” (160). He proposes a refined view of how readers interact with digital media through a “repertoire of gestures” (161-162), the use of which depends on the physical device used to interact with a particular text. In his view, analyses of gestural manipulations happen at five distinct levels: (1) the gesteme, which is an individual move, such as a key stroke or a mouse click, linking haptic move and interactive interface item; (2) the acteme, which refers to a sequence of individual gestemes combining to form a larger unit of gestural meaning, such as drag-and-drop; (3) the semiotic unit of manipulation (“SUM”), which is the sum of identical or similar actemes and their semiotic function; (4) media coupling, which denotes specific functions and meanings of SUMs in their medial contexts; and (5) interactive discourse, which happens at the superordinate level of digital text in context, and relates to the meanings of gestural interactions against this larger backdrop. 

This paper aims to complement and build upon Bouchardon’s (2014) theory in that it offers an examination of the ways in which innovative, experimental gestural manipulations give rise to new vocabularies of haptic interaction. I will report on a small study performed with a group of students, in which they induced a list of verbal expressions for specific actemes and their semiotic transpositions from interacting with Cannizzaro & Gorman’s Pry (2015), Erik Loyer’s Strange Rain (2011), and Steve Jackson’s Sorcery (2013). 

Against the backdrop of Bouchardon’s platform-independent theory, I will propose a tentative and fluid “gestionary” for mobile app fictions that reflects both the complexities of “spatialized relations” (Drucker 2011: 10), as well as the conceptually widening scope of gestural manipulations that is at reader-players’ disposal, as well as the multiple iconic, symbolic, and indexical meanings reader-players can perform with them in the storyworld, via touchscreen interfaces. The paper will end on an inevitably critical note, pertaining to the limitations of black-box media for expanding gestural vocabularies into algorithmic modes and into truly collaborative maker spaces (Emerson 2014).