Hybrid Procedural Semantics for Visual Dialogue: An Interactive Web Demonstration

Lara Verheyen, Jérôme Botoko Ekila, Jens Nevens, Paul Van Eecke, Katrien Beuls

Research output: Contribution in Book/Catalog/Report/Conference proceedingConference contribution

15 Downloads (Pure)


Visual dialogue refers to the task in which a conversational agent needs to hold a meaningful and coherent conversation with a human interlocutor about a scene they observe. To tackle this task, we introduce a novel methodology that makes use of (i) a novel data structure, called the conversation memory, which holds information that is incrementally conveyed in the conversation and (ii) a hybrid procedural semantic representation that is grounded in both the visual input and the conversation memory. In this paper, we present a demonstration that showcases this novel methodology. In this demonstration, a user can interact with a visual dialogue agent and discuss an image of their choice. While the agent is answering questions, the user can follow the agent’s reasoning process. Due to its explainable and interpretable nature, the novel methodology can be used in a wide range of application domains, especially when it is important that the system is human-interpretable. We believe that this novel methodology of hybrid procedural semantics combined with a conversation memory paves the way for building truly intelligent and explainable systems that are able to hold human-like conversations.

Original languageEnglish
Title of host publicationProceedings of the Workshop on Semantic Techniques for Narrative-Based Understanding
Subtitle of host publicationco-located with 31st International Joint Conference on Artificial Intelligence and the 25th European Conference on Artificial Intelligence (IJCAI-ECAI 2022)
EditorsLise Stork, Katrien Beuls, Luc Steels
Number of pages5
Publication statusPublished - 2022
EventWorkshop on semantic techniques for narrative-based understanding: Workshop at IJCAI-ECAI 2022 - Wien Messe, Vienne, Austria
Duration: 23 Jul 2022 → …
Conference number: 31

Publication series

NameCEUR Workshop Proceedings
ISSN (Print)1613-0073


WorkshopWorkshop on semantic techniques for narrative-based understanding
Period23/07/22 → …
Internet address


  • procedural semantics
  • construction grammar
  • Fluid Construction Grammar
  • conversational agents


Dive into the research topics of 'Hybrid Procedural Semantics for Visual Dialogue: An Interactive Web Demonstration'. Together they form a unique fingerprint.

Cite this