Mudra: a unified multimodal interaction framework

Lode Hoste, Bruno Dumas, Beat Signer

Research output: Contribution in Book/Catalog/Report/Conference proceedingConference contribution

40 Downloads (Pure)


In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic inference-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.
Original languageEnglish
Title of host publicationProceedings of the 13th international conference on multimodal interfaces (ICMI '11)
PublisherACM Press
Number of pages8
ISBN (Print)9781450306416
Publication statusPublished - 2011
Externally publishedYes


  • declarative programming
  • rule language
  • multimodal interaction
  • multimodal fusion
  • Mudra


Dive into the research topics of 'Mudra: a unified multimodal interaction framework'. Together they form a unique fingerprint.

Cite this