Multimodal interfaces: A survey of principles, models and frameworks

Bruno Dumas, Denis Lalanne, Sharon Oviatt

Research output: Contribution in Book/Catalog/Report/Conference proceedingChapter

705 Downloads (Pure)


Abstract. The grand challenge of multimodal interface creation is to build reliable processing systems able to analyze and understand multiple communication means in real-time. This opens a number of associated issues covered by this chapter, such as heterogeneous data types fusion, architectures for real-time processing, dialog management, machine learning for multimodal interaction, modeling languages, frameworks, etc. This chapter does not intend to cover exhaustively all the issues related to multimodal interfaces creation and some hot topics, such as error handling, have been left aside. The chapter starts with the features and advantages associated with multimodal interaction, with a focus on particular findings and guidelines, as well as cognitive foundations underlying multimodal interaction. The chapter then focuses on the driving theoretical principles, time-sensitive software architectures and multimodal fusion and fission issues. Modeling of multimodal interaction as well as tools allowing rapid creation of multimodal interfaces are then presented. The article concludes with an outline of the current state of multimodal interaction research in Switzerland, and also summarizes the major future challenges in the field.
Original languageEnglish
Title of host publicationHuman Machine Interaction
Subtitle of host publicationResearch Results of the MMI Program
EditorsDenis Lalanne, Jürg Kohlas
PublisherLNCS - Springer-Verlag
Number of pages24
ISBN (Print)3642004369
Publication statusPublished - 2009
Externally publishedYes

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume5440 LNCS


  • Multimodal interaction
  • multimodal interfaces


Dive into the research topics of 'Multimodal interfaces: A survey of principles, models and frameworks'. Together they form a unique fingerprint.

Cite this