MINDS CDT, 2023
Doctoral Researcher, University of Southampton
Sam Hilton-Jones is a doctoral researcher working on how transformer models, the architecture behind most modern large language models, handle reasoning over long sequences of text.
As these models are asked to process longer and longer inputs, questions arise about how reliably they can maintain focus, identify relevant information and reason correctly. Current understanding of how transformers manage this is limited, and that gap matters: if we do not understand how attention and reasoning work at scale, it becomes harder to predict when and why these models fail, or how to make them better.
Sam’s research investigates the internal dynamics of attention within transformer models as sequence length increases. He studies how the model’s ability to focus on relevant parts of the input changes under these conditions, and how reasoning algorithms emerge from the transformer’s internal structure. The approach is grounded in rigorous theoretical analysis, working toward a clearer picture of what is happening inside these models rather than treating them as black boxes.
His paper, Modelling Attention with Aitchison Geometry: Token Distinguishability and Temperature Scaling, was published at ICML 2026, the International Conference on Machine Learning. It presents a mathematical framework for understanding how transformers distinguish between different tokens when processing long sequences, and how temperature scaling shapes that process.
Sam is due to graduate in 2027.