Adnen Abdessaied

Institute for Visualisation and Interactive Systems, University of Stuttgart

+49 711 685 60052 Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building, Room 01.035 Google Scholar

Biography

Adnen Abdessaied is a PhD student in the Perceptual User Interface group since March 2021. He holds a Bachelor's and Master's degree in Simulation Technology (SimTech) from the University of Stuttgart. The latter was obtained with an overall verdict of very good with distinction and won the Outstanding Work Award from the Industrial Consortium SimTech (IC SimTech). In his Bachelor's studies, he focused on numerical mathematics and classical machine learning techniques. In his master's studies, he got involved in deep multi-modal memory-enhanced neural networks that are inspired by the human brain. His research interests lie in the design and interpretability of multi-modal hybrid Neuro-Symbolic AI models.

Teaching

2022

Mensch-Computer Interaktion (Teaching Assistant)Bachelor

2021

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

---

Supervision

2022

Attention-Based Reasoning over Multimodal Embeddings in Video-Grounded Dialog (M.Sc.) Manuel Hochmeister

 

Multimodal Feature Fusion via Graph Neural Networks for Visual Dialog (M.Sc.) Nishan Chatterjee

---

Open Thesis Projects

 

Diagnostic Dataset for Video Dialog based on CLEVRERMaster

 

Multimodal Models Meet Unbiased Scene GraphsMaster

---

Publications

(Abdessaied et al., 2024; Abdessaied et al., 2024; Abdessaied* et al., 2022; Abdessaied et al., 2022)

1. 

   OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog

   Adnen Abdessaied, Manuel Hochmeister, Andreas Bulling

   Proc. 31st Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pp. 1–11, 2024.

   Abstract Links BibTeX Project

   We present the Object Language Video Transformer (OLViT) – a novel model for video dialog operating over a multi-modal attention-based dialog state tracker. Existing video dialog models struggle with questions requiring both spatial and temporal localization within videos, long-term temporal reasoning, and accurate object tracking across multiple dialog turns. OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns. In stark contrast to previous works, our approach is generic by nature and is therefore capable of learning continuous multi-modal dialog state representations of the most relevant objects and rounds. As a result, they can be seamlessly integrated into Large Language Models (LLMs) and offer high flexibility in dealing with different datasets and tasks. Evaluations on the challenging DVD (response classification) and SIMMC 2.1 (response generation) datasets show that OLViT achieves new state-of-the-art performance across both datasets.

   Paper: abdessaied24_coling.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/OLViT

   @inproceedings{abdessaied24_coling, author = {Abdessaied, Adnen and von Hochmeister, Manuel and Bulling, Andreas}, title = {OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog}, booktitle = {Proc. 31st Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)}, year = {2024}, pages = {1--11} }
2. 

   VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs

   Adnen Abdessaied, Lei Shi, Andreas Bulling

   Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5805–5814, 2024.

   Abstract Links BibTeX Project

   We propose VD-GR – a novel visual dialog model that combines pre-trained language models (LMs) with graph neural networks (GNNs). Prior works mainly focused on one class of models at the expense of the other, thus missing out on the opportunity of combining their respective benefits. At the core of VD-GR is a novel integration mechanism that alternates between spatial-temporal multi-modal GNNs and BERT layers, and that covers three distinct contributions: First, we use multi-modal GNNs to process the features of each modality (image, question, and dialog history) and exploit their local structures before performing BERT global attention. Second, we propose hub-nodes that link to all other nodes within one modality graph, allowing the model to propagate information from one GNN (modality) to the other in a cascaded manner. Third, we augment the BERT hidden states with fine-grained multi-modal GNN features before passing them to the next VD-GR layer. Evaluations on VisDial v1.0, VisDial v0.9, VisDialConv, and VisPro show that VD-GR achieves new state-of-the-art results across all four datasets

   Paper: abdessaied24_wacv.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/VDGR

   @inproceedings{abdessaied24_wacv, author = {Abdessaied, Adnen and Shi, Lei and Bulling, Andreas}, title = {VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs}, booktitle = {Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, year = {2024}, pages = {5805--5814} }

1. 

   Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA

   Adnen Abdessaied*, Ekta Sood*, Andreas Bulling

   Proc. of the 7th Workshop on Representation Learning for NLP (Repl4NLP), pp. 1–12, 2022.

   Abstract Links BibTeX Project

   [Equal contribution by the first two authors.] We propose the Video Language Co-Attention Network (VLCN) – a novel memory-enhanced model for Video Question Answering (VideoQA). Our model combines two original contributions: A multimodal fast-learning feature fusion (FLF) block and a mechanism that uses self-attended language features to separately guide neural attention on both static and dynamic visual features extracted from individual video frames and short video clips. When trained from scratch, VLCN achieves competitive results with the state of the art on both MSVD-QA and MSRVTT-QA with 38.06% and 36.01% test accuracies, respectively. Through an ablation study, we further show that FLF improves generalization across different VideoQA datasets and performance for question types that are notoriously challenging in current datasets, such as long questions that require deeper reasoning as well as questions with rare answers.

   Paper: abdessaied22_repl4NLP.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/vlcn

   @inproceedings{abdessaied22_repl4NLP, author = {Abdessaied*, Adnen and Sood*, Ekta and Bulling, Andreas}, title = {Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA}, booktitle = {Proc. of the 7th Workshop on Representation Learning for NLP (Repl4NLP)}, year = {2022}, pages = {1--12} }
2. 

   Neuro-Symbolic Visual Dialog

   Adnen Abdessaied, Mihai Bâce, Andreas Bulling

   Proceedings of the 29th International Conference on Computational Linguistics (COLING), pp. 1–11, 2022.

   Abstract Links BibTeX Project Oral presentation

   We propose Neuro-Symbolic Visual Dialog (NSVD) —the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog —a relative improvement of more than 10% over the state of the art —while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.

   Paper: abdessaied22_coling.pdf

   Code: Contact Author for further information

   @inproceedings{abdessaied22_coling, author = {Abdessaied, Adnen and Bâce, Mihai and Bulling, Andreas}, title = {Neuro-Symbolic Visual Dialog}, booktitle = {Proceedings of the 29th International Conference on Computational Linguistics (COLING)}, year = {2022}, pages = {1--11} }