Multimodal Interaction for Remote Physical Artefacts

forthcoming

Digitally replicating the tactile, spatial, and multisensory qualities critical for interacting with physical objects remains an unsolved complex problem. This is challenge is felt acutely in cases such as remote collaboration, which is central to modern work practices, offering flexibility and sustainability. Significant challenges arise when physical objects are integral to collaboration. MIRA answers the question: How can we create a multimodal proxy for remote physical objects, effectively replicating and going beyond the experience of in-person interaction with the actual object? MIRA proposes a new Theory of Interaction with Remote Physical Objects by combining research in Human-Computer Interaction, psychological theories, and innovative applications integrating materials science and Multimodal Interaction. Central to this is the creation of a new class of interactive tools, termed multimodal proxies. The effectiveness of these proxies will be tested by designing effective and engaging remote collaboration systems. MIRA has three objectives: 1. Develop Foundational Principles: Establish a novel set of principles for designing systems supporting remote interaction with physical objects. This has never been done before and will draw on psychological theories and empirical data from large-scale interview and observation studies. 2. Create Multimodal Proxies: Develop a new class of interactive tools that simulate the physical properties of remote objects, integrating sensory feedback—visual, auditory, and haptic—for synchronized interaction, addressing technological asymmetries. 3. Extend Interaction Paradigms: Explore unique affordances of multimodal proxies to unlock interaction possibilities beyond those in traditional co-located settings, for richer, more transformative collaboration. MIRA’s contributions include a novel theoretical framework, cutting-edge multimodal technologies, and real-world evaluations.