Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks

Lara Verheyen, Jérôme Botoko Ekila, Jens Nevens, Paul Van Eecke, Katrien Beuls

Research output: Contribution in Book/Catalog/Report/Conference proceedingConference contribution

5 Downloads (Pure)

Abstract

This paper introduces a novel approach to visual dialogue that is based on neuro-symbolic procedural semantics. The approach builds further on earlier work on procedural semantics for visual question answering and expands it on the one hand with neuro-symbolic reasoning operations, and on the other hand with mechanisms that handle the challenges that are inherent to dialogue, in particular the incremental nature of the information that is conveyed. Concretely, we introduce (i) the use of a conversation memory as a data structure that explicitly and incrementally represents the information that is expressed during the subsequent turns of a dialogue, and (ii) the design of a neuro-symbolic procedural semantic representation that is grounded in both visual input and the conversation memory. We validate the methodology using the reasoning- intensive MNIST Dialog and CLEVR-Dialog benchmark challenges and achieve a question-level accuracy of 99.8% and 99.2% respectively. The methodology presented in this paper responds to the growing interest in the field of artificial intelligence in solving tasks that involve both low-level perception and high-level reasoning using a combination of neural and symbolic techniques.
Original languageEnglish
Title of host publicationECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings
Subtitle of host publicationECAI 2023
EditorsKobi Gal, Ann Nowé, Grzegorz J. Nalepa, Roy Fairstein, Roxana Rădulescu
Place of PublicationAmsterdam
Pages2419 - 2426
Number of pages8
ISBN (Electronic)9781643684369
DOIs
Publication statusPublished - 28 Sept 2023
EventTwenty-sixth European Conference on Artificial Intelligence (ECAI 2023) - ICE, Krakow, Poland
Duration: 30 Sept 20234 Oct 2023
https://ecai2023.eu

Publication series

NameFrontiers in Artificial Intelligence and Applications
Volume372
ISSN (Print)0922-6389
ISSN (Electronic)1879-8314

Conference

ConferenceTwenty-sixth European Conference on Artificial Intelligence (ECAI 2023)
Abbreviated titleECAI 2023
Country/TerritoryPoland
CityKrakow
Period30/09/234/10/23
Internet address

Keywords

  • neuro-symbolic AI
  • procedural semantics
  • visual dialogue
  • personal dynamic memory
  • computational construction grammar

Fingerprint

Dive into the research topics of 'Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks'. Together they form a unique fingerprint.

Cite this