Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks

Lara Verheyen; Jérôme Botoko Ekila; Jens Nevens; Paul Van Eecke; Katrien Beuls

doi:10.3233/FAIA230544

Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks

Lara Verheyen, Jérôme Botoko Ekila, Jens Nevens, Paul Van Eecke, Katrien Beuls

Research output: Contribution in Book/Catalog/Report/Conference proceeding › Conference contribution

19 Downloads (Pure)

Abstract

This paper introduces a novel approach to visual dialogue that is based on neuro-symbolic procedural semantics. The approach builds further on earlier work on procedural semantics for visual question answering and expands it on the one hand with neuro-symbolic reasoning operations, and on the other hand with mechanisms that handle the challenges that are inherent to dialogue, in particular the incremental nature of the information that is conveyed. Concretely, we introduce (i) the use of a conversation memory as a data structure that explicitly and incrementally represents the information that is expressed during the subsequent turns of a dialogue, and (ii) the design of a neuro-symbolic procedural semantic representation that is grounded in both visual input and the conversation memory. We validate the methodology using the reasoning- intensive MNIST Dialog and CLEVR-Dialog benchmark challenges and achieve a question-level accuracy of 99.8% and 99.2% respectively. The methodology presented in this paper responds to the growing interest in the field of artificial intelligence in solving tasks that involve both low-level perception and high-level reasoning using a combination of neural and symbolic techniques.

Original language	English
Title of host publication	ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings
Subtitle of host publication	ECAI 2023
Editors	Kobi Gal, Ann Nowé, Grzegorz J. Nalepa, Roy Fairstein, Roxana Rădulescu
Place of Publication	Amsterdam
Pages	2419 - 2426
Number of pages	8
ISBN (Electronic)	9781643684369
DOIs	https://doi.org/10.3233/FAIA230544
Publication status	Published - 28 Sept 2023
Event	Twenty-sixth European Conference on Artificial Intelligence (ECAI 2023) - ICE, Krakow, Poland Duration: 30 Sept 2023 → 4 Oct 2023 https://ecai2023.eu

Publication series

Name	Frontiers in Artificial Intelligence and Applications
Volume	372
ISSN (Print)	0922-6389
ISSN (Electronic)	1879-8314

Conference

Conference	Twenty-sixth European Conference on Artificial Intelligence (ECAI 2023)
Abbreviated title	ECAI 2023
Country/Territory	Poland
City	Krakow
Period	30/09/23 → 4/10/23
Internet address	https://ecai2023.eu

Keywords

neuro-symbolic AI
procedural semantics
visual dialogue
personal dynamic memory
computational construction grammar

Access to Document

10.3233/FAIA230544

FAIA-372-FAIA230544Final published version, 811 KBLicence: CC BY-NC

Cite this

Verheyen, L., Botoko Ekila, J., Nevens, J., Van Eecke, P., & Beuls, K. (2023). Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks. In K. Gal, A. Nowé, G. J. Nalepa, R. Fairstein, & R. Rădulescu (Eds.), ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings: ECAI 2023 (pp. 2419 - 2426). (Frontiers in Artificial Intelligence and Applications; Vol. 372).. https://doi.org/10.3233/FAIA230544

Verheyen, Lara ; Botoko Ekila, Jérôme ; Nevens, Jens et al. / Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks. ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings: ECAI 2023. editor / Kobi Gal ; Ann Nowé ; Grzegorz J. Nalepa ; Roy Fairstein ; Roxana Rădulescu. Amsterdam, 2023. pp. 2419 - 2426 (Frontiers in Artificial Intelligence and Applications).

@inproceedings{f28b58ab35e340028a223e926809ad7f,

title = "Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks",

abstract = "This paper introduces a novel approach to visual dialogue that is based on neuro-symbolic procedural semantics. The approach builds further on earlier work on procedural semantics for visual question answering and expands it on the one hand with neuro-symbolic reasoning operations, and on the other hand with mechanisms that handle the challenges that are inherent to dialogue, in particular the incremental nature of the information that is conveyed. Concretely, we introduce (i) the use of a conversation memory as a data structure that explicitly and incrementally represents the information that is expressed during the subsequent turns of a dialogue, and (ii) the design of a neuro-symbolic procedural semantic representation that is grounded in both visual input and the conversation memory. We validate the methodology using the reasoning- intensive MNIST Dialog and CLEVR-Dialog benchmark challenges and achieve a question-level accuracy of 99.8% and 99.2% respectively. The methodology presented in this paper responds to the growing interest in the field of artificial intelligence in solving tasks that involve both low-level perception and high-level reasoning using a combination of neural and symbolic techniques.",

keywords = "neuro-symbolic AI, procedural semantics, visual dialogue, personal dynamic memory, computational construction grammar",

author = "Lara Verheyen and {Botoko Ekila}, J{\'e}r{\^o}me and Jens Nevens and {Van Eecke}, Paul and Katrien Beuls",

note = "Funding Information: The research reported on in this paper received funding from the EU{\textquoteright}s H2020 RIA programme under grant agreement no. 951846 (MUHAI), from the Research Foundation Flanders (FWO) through a postdoctoral grant awarded to PVE (grant no. 76929) and from the Flemish Government under the {\textquoteleft}Flanders AI Research Program{\textquoteright}. Publisher Copyright: {\textcopyright} 2023 The Authors.; Twenty-sixth European Conference on Artificial Intelligence (ECAI 2023), ECAI 2023 ; Conference date: 30-09-2023 Through 04-10-2023",

year = "2023",

month = sep,

day = "28",

doi = "10.3233/FAIA230544",

language = "English",

isbn = "978-1-64368-436-9",

series = "Frontiers in Artificial Intelligence and Applications",

pages = "2419 -- 2426",

editor = "Kobi Gal and Ann Now{\'e} and Nalepa, {Grzegorz J.} and Roy Fairstein and Roxana R{\u a}dulescu",

booktitle = "ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings",

url = "https://ecai2023.eu",

}

Verheyen, L, Botoko Ekila, J, Nevens, J, Van Eecke, P & Beuls, K 2023, Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks. in K Gal, A Nowé, GJ Nalepa, R Fairstein & R Rădulescu (eds), ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings: ECAI 2023. Frontiers in Artificial Intelligence and Applications, vol. 372, Amsterdam, pp. 2419 - 2426, Twenty-sixth European Conference on Artificial Intelligence (ECAI 2023), Krakow, Poland, 30/09/23. https://doi.org/10.3233/FAIA230544

Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks. / Verheyen, Lara; Botoko Ekila, Jérôme; Nevens, Jens et al.
ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings: ECAI 2023. ed. / Kobi Gal; Ann Nowé; Grzegorz J. Nalepa; Roy Fairstein; Roxana Rădulescu. Amsterdam, 2023. p. 2419 - 2426 (Frontiers in Artificial Intelligence and Applications; Vol. 372).

Research output: Contribution in Book/Catalog/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks

AU - Verheyen, Lara

AU - Botoko Ekila, Jérôme

AU - Nevens, Jens

AU - Van Eecke, Paul

AU - Beuls, Katrien

N1 - Funding Information: The research reported on in this paper received funding from the EU’s H2020 RIA programme under grant agreement no. 951846 (MUHAI), from the Research Foundation Flanders (FWO) through a postdoctoral grant awarded to PVE (grant no. 76929) and from the Flemish Government under the ‘Flanders AI Research Program’. Publisher Copyright: © 2023 The Authors.

PY - 2023/9/28

Y1 - 2023/9/28

N2 - This paper introduces a novel approach to visual dialogue that is based on neuro-symbolic procedural semantics. The approach builds further on earlier work on procedural semantics for visual question answering and expands it on the one hand with neuro-symbolic reasoning operations, and on the other hand with mechanisms that handle the challenges that are inherent to dialogue, in particular the incremental nature of the information that is conveyed. Concretely, we introduce (i) the use of a conversation memory as a data structure that explicitly and incrementally represents the information that is expressed during the subsequent turns of a dialogue, and (ii) the design of a neuro-symbolic procedural semantic representation that is grounded in both visual input and the conversation memory. We validate the methodology using the reasoning- intensive MNIST Dialog and CLEVR-Dialog benchmark challenges and achieve a question-level accuracy of 99.8% and 99.2% respectively. The methodology presented in this paper responds to the growing interest in the field of artificial intelligence in solving tasks that involve both low-level perception and high-level reasoning using a combination of neural and symbolic techniques.

AB - This paper introduces a novel approach to visual dialogue that is based on neuro-symbolic procedural semantics. The approach builds further on earlier work on procedural semantics for visual question answering and expands it on the one hand with neuro-symbolic reasoning operations, and on the other hand with mechanisms that handle the challenges that are inherent to dialogue, in particular the incremental nature of the information that is conveyed. Concretely, we introduce (i) the use of a conversation memory as a data structure that explicitly and incrementally represents the information that is expressed during the subsequent turns of a dialogue, and (ii) the design of a neuro-symbolic procedural semantic representation that is grounded in both visual input and the conversation memory. We validate the methodology using the reasoning- intensive MNIST Dialog and CLEVR-Dialog benchmark challenges and achieve a question-level accuracy of 99.8% and 99.2% respectively. The methodology presented in this paper responds to the growing interest in the field of artificial intelligence in solving tasks that involve both low-level perception and high-level reasoning using a combination of neural and symbolic techniques.

KW - neuro-symbolic AI

KW - procedural semantics

KW - visual dialogue

KW - personal dynamic memory

KW - computational construction grammar

UR - http://www.scopus.com/inward/record.url?scp=85175816134&partnerID=8YFLogxK

U2 - 10.3233/FAIA230544

DO - 10.3233/FAIA230544

M3 - Conference contribution

SN - 978-1-64368-436-9

T3 - Frontiers in Artificial Intelligence and Applications

SP - 2419

EP - 2426

BT - ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings

A2 - Gal, Kobi

A2 - Nowé, Ann

A2 - Nalepa, Grzegorz J.

A2 - Fairstein, Roy

A2 - Rădulescu, Roxana

CY - Amsterdam

T2 - Twenty-sixth European Conference on Artificial Intelligence (ECAI 2023)

Y2 - 30 September 2023 through 4 October 2023

ER -

Verheyen L, Botoko Ekila J, Nevens J, Van Eecke P, Beuls K. Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks. In Gal K, Nowé A, Nalepa GJ, Fairstein R, Rădulescu R, editors, ECAI 2023 - 26th European Conference on Artificial Intelligence, including 12th Conference on Prestigious Applications of Intelligent Systems, PAIS 2023 - Proceedings: ECAI 2023. Amsterdam. 2023. p. 2419 - 2426. (Frontiers in Artificial Intelligence and Applications). doi: 10.3233/FAIA230544

Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Meaning and understanding in human-centric AI

Cite this

Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Meaning and understanding in human-centric AI

Cite this