Designer-Led Interaction with Generative A

Jury

Björn HARTMANN, Associate Professor, University of California, Berkeley (Rapporteur)

• Nick BRYAN-KINNS, Professor, University of the Arts London (Rapporteur)
• Elena L. GLASSMAN, Assistant Professor, Harvard University (Examinatrice)
• Steven DOW, Professor, University of California, San Diego (Examinateur)
• Nicolas SABOURET, Professor, Université Paris-Saclay (Examinateur)

Abstract

Creative practitioners create, collect, and manipulate images and sketches, composing and recomposing them as they develop ideas. Generative AIArtificial Intelligence (GenAI) offers practitioners a novel creation medium, but the dominant interaction paradigm relies on text-based prompting, a departure from their traditional workflows. This thesis investigates how GenAI can support and augment practitioners’ visual, expressive, and reflective practices by exploring interaction designs that bridge existing creative workflows with the new possibilities introduced by GenAI. This understanding guides my approach to the interaction design of creative tools with GenAI. 

My research follows a theory-informed, empirically grounded methodology that treats system design as a synthesis of HCI theory, empirical study with professional designers, existing artifact design, and engineering constraints. I designed, implemented, and evaluated three GenAI systems through this approach. DesignPrompt introduces multimodal GenAI input moodboarding that lets designers compose prompts from images, colors, and semantic labels, externalizing creative intent in forms that connect with existing ideation practice. FusAIn embeds visual properties such as color and texture into tangible pen interaction that designers manipulate through familiar sketching gestures, enabling compositional and embodied control over content generation. DesignTrace captures design rationale through a Palette, Node, and Template architecture that helps designers trace and organize their evolving work with GenAI across semantic and visual forms, to support non-linear creative exploration.

The empirical findings identify key considerations for designing GenAI systems for creative professionals:

(1) Practitioners’ perceived control is not solely a technical property; it emerges from how the interaction is decomposed into manipulable units that users can act on directly; how those units couple meaningfully; and how their temporal rhythms match users’ workflows. This suggests designing interactions around meaningful units that enable users to understand, manipulate, and steer different aspects of the generative process.

(2) Practitioners benefit most from modalities that align with their existing practices. 

However, richer modalities alone do not close the gap between practitioners’ intentions and what generative models can operationalize. This suggests that interfaces should support transitions across complementary interaction modalities while keeping these translations legible for users. 

(3) Practitioners’ creative processes span multiple temporal scales, and different interaction units remain relevant for different durations. 

This suggests that interfaces should support interaction units with appropriate temporal rhythms and persistence, enabling practitioners to move fluidly between immediate actions, revisitable artifacts, and evolving workflows.

To conclude, this thesis contributes three demonstrations of how GenAI can enhance designers’ existing work practices, empirical findings from five studies of creative practice and human–GenAI interaction, and a theoretical approach for designing GenAI tools that align with the situated practices of creative professionals.