A new research paper introduces MIST (Multimodal Interactive Speech-based Tool-calling Dataset), a synthetic dataset designed to advance the development of voice-based interfaces for smart home devices. The dataset aims to address the challenges of creating effective voice assistants that can handle complex user interactions within the context of the Internet of Things (IoT) environment 1.
The rise of IoT devices necessitates voice-based interfaces capable of handling complex user experiences 1.
The MIST dataset focuses on multi-turn, voice-driven code generation tasks that operate over IoT devices 1.
The research highlights the difficulties in modeling real-world IoT devices, which require the integration of spatiotemporal constraints, speech inputs, dynamic state tracking, and mixed-initiative interaction patterns 1.
The study found a significant performance gap between open- and closed-weight multimodal Large Language Models (LLMs) on the MIST dataset 1.
The researchers also observed that even the most advanced closed-weight LLMs have "substantial headroom" for improvement 1.
The MIST dataset and its associated data generation framework are released to facilitate research on mixed-initiative voice assistants that can reason about physical world constraints 1.
The paper's authors are Maximillian Chen, Xuanming Zhang, Michael Peng, Zhou Yu, Alexandros Papangelis, and Yohan Jo 1.
The research is categorized under Computer Science, specifically Computation and Language (cs.CL), Artificial Intelligence (cs.AI), Human-Computer Interaction (cs.HC), Multimedia (cs.MM), Sound (cs.SD), and Audio and Speech Processing (eess.AS) 1.
The project page for MIST is available online 1.
The paper was submitted on May 7, 2026 1.
ops.llm_calls. Every fact traces to a citation. If a fact looks wrong, write to corrections.