Pad-based Interaction in Mixed Reality environments

Jury

• Maud MARCHAL, Professeure, INSA Université de Rennes – Reviewer
• Thomas PIETRZAK, Professeur, Université de Lille – Reviewer
• Ferran ARGELAGUET, Chargé de recherche, Inria Université de Rennes – Examiner
• Céline COUTRIX, Directrice de recherche, CNRS Université Grenoble Alpes – Examiner
• Jean-Daniel FEKETE, Directeur de recherche, Inria Saclay – LISN – Examiner

Abstract

Mixed Reality (MR) environments integrate virtual elements into the user’s physical space, creating immersive experiences where real objects and digital content coexist. This spatial distribution of virtual content is valuable, as it allows information to be displayed where it is most meaningful, but it also complicates interaction with virtual elements scattered throughout the environment.

This thesis investigates the use of a pad, a small touch-sensitive surface, as a means of indirect interaction in MR. It explores the design of opportunistic pads, created from existing surfaces and relying solely on sensors embedded in MR devices, particularly hand-tracking. The goal is to facilitate and enrich interaction in MR, with a particular focus on industrial maintenance scenarios.

The first contribution studies Spatial Window Switching (SWS) techniques, which allow users to locate a specific window among those distributed in the environment. We designed and evaluated techniques based on an overview of all available windows represented as miniatures, visually linked to their actual positions. Results show that these techniques significantly reduce search time and physical movement, helping users locate virtual content of interest more efficiently.

The second contribution explores the use of the non-dominant hand as a support surface for an always-available pad, relying solely on the hand-tracking capabilities of MR devices. Three anchoring strategies were evaluated: a pad floating near the hand, a pad aligned with a flat object held in the hand, and a pad directly placed on the palm. Results show similar performance across all approaches, indicating that hand tracking has reached a level of maturity sufficient to allow the palm to serve as a reliable touch surface. Although these pads perform worse than capacitive screens, they let users keep both hands free and provide proprioceptive and haptic cues. 

The third contribution, TriPad, introduces a technique that enables users to transform any passive physical surface into an interactive pad using only hand tracking. The user places their hand flat on the surface, then performs a “tap” by briefly lifting and lowering it. The system recognizes this characteristic movement and infers the plane by analyzing the positions of three fingertips when the hand was stationary, anchoring the virtual pad accordingly. We validated TriPad’s robustness across various surface types (wood, glass, drywall) and compared it to existing direct interaction techniques available in MR devices. Results show that TriPad provides a better speed–accuracy trade-off, particularly on horizontal surfaces where it reduces fatigue and improves comfort.

Taken together, Spatial Window Switching techniques, hand-bound pads, and surface-anchored pads open new possibilities for indirect interaction in Mixed Reality. They address accessibility, comfort, and precision issues inherent to direct interaction.

The thesis concludes with a discussion of future perspectives for improving this paradigm of indirect interaction and its applicability to industrial contexts, especially MR assisted maintenance, where technicians can benefit from more efficient, and safe interaction in complex environments.