Vistoria: A Multimodal System to Support Fictional Story Writing through Instrumental Text-Image Co-Editing

Humans think visually-we remember in images, dream in pictures, and use visual metaphors to communicate. Yet, most creative writing tools remain text-centric, limiting how authors plan and translate ideas. We present Vistoria, a system for synchronized text-image co-editing in fictional story writing that treats visuals and text as coequal narrative materials. A formative Wizard-of-Oz co-design study with 10 story writers revealed how sketches, images, and annotations serve as essential instruments for ideation and organization. Drawing on theories of Instrumental Interaction and Structural Mapping, Vistoria introduces multimodal operations-lasso, collage, filters, and perspective shifts that enable seamless narrative exploration across modalities. A controlled study with 12 participants shows that co-editing enhances expressiveness, immersion, and collaboration, enabling writers to explore divergent directions, embrace serendipitous randomness, and trace evolving storylines. While multimodality increased cognitive demand, participants reported stronger senses of authorship and agency. These findings demonstrate how multimodal co-editing expands creative potential by balancing abstraction and concreteness in narrative development.

💡 Research Summary

The paper introduces Vistoria, a multimodal co‑editing environment that treats images and text as co‑equal narrative materials for fictional story writing. Grounded in Dual Coding Theory, the authors argue that creative writing naturally involves simultaneous visual and verbal processing, yet most existing tools remain text‑centric, forcing writers to translate visual ideas into words and incurring extra cognitive load. To address this gap, the authors adopt the Instrumental Interaction framework and design a set of “instrumental operations” – Lasso, Collage, Perspective Shift, and Filter – that can be applied uniformly to both text blocks and image objects. These operations embody the principles of reification (turning abstract commands into manipulable objects), polymorphism (the same tool works across modalities), and reuse (operations can be replayed or adapted).

A formative Wizard‑of‑Oz co‑design study with ten experienced writers revealed that sketches, reference images, and annotations are essential for ideation, world‑building, and narrative organization. Writers expressed a strong desire for tools that let them manipulate visual and textual elements together, rather than switching back and forth between separate interfaces. Building on these insights, Vistoria implements a canvas‑based UI where text snippets and images coexist on a shared 2‑D space. The system integrates a large language model (LLM) backend that automatically generates and synchronizes text‑image mappings: when a user selects a region with Lasso, the LLM produces a corresponding textual description; when a Perspective Shift changes an image’s viewpoint, the LLM adjusts the narrative perspective accordingly; Filter modifies visual style while simultaneously altering textual tone.

A controlled lab study with twelve participants compared Vistoria to a conventional text‑only editor. Quantitative results showed significant gains in expressive richness, immersion, and idea generation. Participants used the instrumental operations to explore divergent plot directions, re‑compose character relationships, and experiment with scene framing, leading to more detailed and vivid narratives. Collaboration was also smoother because image‑text alignment was maintained automatically, reducing the need for verbal coordination. The main drawback was a modest increase in mental and physical workload due to the added multimodal interactions. However, participants reported a heightened sense of authorship and agency, indicating that the benefits outweighed the costs.

The paper contributes (1) a formative study that maps how fiction writers employ multimodal artifacts during planning and translation, (2) the design and implementation of Vistoria—a system that unifies text and visual editing through instrumental operations and LLM‑driven synchronization, and (3) empirical evidence that multimodal co‑editing can enhance creative output while preserving narrative coherence. Limitations include the need for better workload management (e.g., gesture shortcuts, automatic layout), support for multi‑user version control, and validation across other narrative domains such as game design or advertising copy. The authors suggest future work on scaling the approach, refining the interaction model, and exploring broader genre applications.