Symbolic Computation in Deep Neural Networks

No Thumbnail Available
Journal Title
Journal ISSN
Volume Title
Perustieteiden korkeakoulu | Master's thesis
Date
2022-07-29
Department
Major/Subject
Machine Learning, Data Science and Artificial Intelligence
Mcode
SCI3044
Degree programme
Master’s Programme in Computer, Communication and Information Sciences
Language
en
Pages
48+2
Series
Abstract
Studying symbolic computation in deep neural networks (DNNs) is essential for improving their explainability and generalizability. Whether DNNs can conduct symbolic computation is a long-standing topic of controversy. One analysis is that DNNs, as connectionist models, only process associative relations and are unlikely to have higher-level cognitive abilities. However, their success in complex tasks such as language modeling has raised interest in whether symbolic computation is involved. I investigate the presence of symbolic computation in DNNs by testing the performance of state-of-the-art Transformer networks BERT and T5 in reasoning and vocabulary generalization experiments. Our results show that the model has good performance in systematic and zero-shot vocabulary generalization but is easily disturbed by task-specific vocabularies. Based on both theoretical argumentation and empirical results, I propose the interpretation that Transformers accomplish the tasks by two main associative and hence not genuinely symbolic methods via 1. embedding similarity between tokens, 2. (in)sensitivity to positional features irrespective of vocabulary.
Description
Supervisor
Asokan, N.
Thesis advisor
Gröndahl, Tommi
Keywords
deep neural network, Transformer, symbolic computation, systematic generalization, zero-shot generalization
Other note
Citation