Screenshot: jdr.tudelft.nl
Marco C. Rozendaal and David V. Keyson published a paper about designing user interfaces with gestures and sound at “The Journal of Design Research”:
abstract:
“This paper explores the use of sound and gestures within a tangible user interface, towards influencing the performance and appeal of browsing through a voice mail collection. Three browsers were developed, in which speech and non-speech sounds were projected in a sound-scape above a tangible interface. For each of the browsers, users were given the task of locating voice mail messages. Search behavior in terms of gestures, task time, and errors, were recorded and categorized. Users’ experience was measured in terms of appeal, consisting of hedonic and ergonomic qualities. Results showed that users’ explorative behavior changed, depending on the specific type of browser. Subjects were observed to utilize non-speech sound cues effectively in searching for target voice mail messages. Subjective results showed that ergonomic and hedonic qualities of appeal were found to interact in an averaging way, such that overall appeal was similar across conditions. Implications towards designing multimodal interfaces are discussed. ”
conclusions:
“The benefits of the multimodal feedback combined with non-speech representations of content via sound, appeared to positively support browsing behavior. Further design work is needed towards improving human performance. For example, the perceived complexity of non-speech sounds to communicate message properties may be reduced if the number of factors communicated in the sounds is reduced. Reference sounds may also further help differentiate sound properties. For example, urgency could be a two-tone sound, such that the first part of the tone is the baseline; an increase in pitch in the second tone could indicate higher urgency. In designing non-speech sounds, voice mail samples themselves could also be used. In this case, the translation from voice mail information into compressed abstractions could be automated. Important aspects of the voice mail, or of other application’s content, must be identified and manipulated as to create appropriate abstractions. Slot region cues could also be optimized by creating rod-like regions above the slots with a fixed diameter. The cone-like regions caused the slot position cues to spread out horizontally further from the slots. This led to difficulties in localization. With rods-like regions, the position cues would not change in terms of horizontal latitude, giving the user clearer information.
In terms of understanding how non-speech sounds are mapped in a sound-scape emitting from a tangible product, the physical product form may play an important role. For example, in the current prototype the round form conveyed the affordance of circular browsing movements, a cone-like physical form may help communicate up-down browsing gestures in addition to circular movements. The problem of scaling virtual information to a physical spatial layout has not been solved in this study. The sixteen slots used in the current study, facilitate a maximum of 16 voice mails. Storing multiple messages within a slot or changing the physical slots into touch sensitive displays, may increase the maximum number of voice mails that may be stored in the system. However, this could reduce the intuitiveness of the tangible user interface since the close connection between the physical structure and the virtual content is lost. To address the problem of scalability, actuator technology could be used to change physical structures in relation to virtual content.
As discussed earlier, results of the experiment showed a ceiling effect on the hedonic scores, such that the differences between the additional browsing styles were obscured. The ceiling effect could be attributed to the innovative character of the tangible user interface itself. When the user interface itself is perceived as innovative, subjects may have difficulty in judging the innovativeness of the additional browsing styles. In future studies, a tangible user interface may be designed having lower hedonic properties, for example by creating a tangible user interface with a more familiar product form. In this way, additional browsing styles can be measured more effectively in terms of their hedonic qualities.
Given that the hedonic features of a user interface may decrease after using the system for some time, one could consider adding novel features. In this way, the appeal of the user interface may be maintained. For the interface to add novelty without disturbing the user, behavioral data is needed that can indicate if the addition of novelty is valued by the user. For example, a user that is bored with the interface may welcome novelty, while a user that is satisfied with the system, may not be pleased. Regulating the hedonic features of the user interface by means of behavioural data would be an interesting issue within the field of multimodality, and we will investigate this further.(…)”
