Theses and Dissertations
Advisor
Ramkumar, Mahalingam
Committee Member
Pillai, Nisha
Committee Member
Chen, Zhiqian
Date of Degree
8-7-2025
Original embargo terms
Immediate Worldwide Access
Document Type
Graduate Thesis - Open Access
Major
Computer Science
Degree Name
Master of Science (M.S.)
College
James Worth Bagley College of Engineering
Department
Department of Computer Science and Engineering
Abstract
This thesis introduces a novel interpretability framework for transformer encoder models by hypothesizing that their internal embedding updates define a state transition system. We propose that transformers implicitly learn token-to-token influence dynamics, which can be analyzed using graph-theoretic methods. To validate this, we construct transition graphs from embedding changes and rank token importance using the PageRank algorithm. We develop two transformer models from scratch: a self-supervised model that incorporates serotype tokens to learn contextualized embeddings, and a supervised model initialized with these embeddings. Cosine similarity analysis of their token influence patterns reveals strong structural alignment, with values exceeding 93%. This architecture-aware approach reframes attention as a dynamic transition process, providing deeper insight into how information propagates within transformer models. Our findings high- light the potential of graph-based analysis for uncovering interpretable structures in modern deep learning architectures.
Sponsorship (Optional)
USDA
Recommended Citation
Nassiri, Ramak, "Unboxing the black box: Graph-based interpretability in transformer models" (2025). Theses and Dissertations. 6674.
https://scholarsjunction.msstate.edu/td/6674
