MINDS CDT doctoral researcher | University of Southampton | MINDS CDT, 2023
Understanding how transformers reason across long sequences
Sam Hilton-Jones is using mathematical analysis to investigate how transformer models, the architecture behind most modern large language models, focus and reason as their inputs grow longer. His work aims to help researchers see more clearly what is happening inside these models, rather than treating them as black boxes.
Professional and research links: LinkedIn | ICML 2026 paper | Paper on OpenReview
Research at a glance
| Research focus | Long-context reasoning and attention in transformer models |
| MINDS theme | Agent-Based Adaptive Systems |
| Methods | Mathematical and theoretical analysis of model behaviour |
| Technologies | Transformers, attention mechanisms, large language models |
| Featured output | ICML 2026 conference paper |
| Current status | Doctoral researcher, due to graduate in 2027 |
How do transformers reason across long sequences?
Transformer models are the architecture behind most modern large language models. To produce an answer, a transformer has to decide which parts of its input matter most. It does this through an attention mechanism, which gives each word or token in the input an importance score.
As inputs grow longer, these scores become much smaller. That makes it hard to tell whether the model is still making useful choices about which information to use, or whether it is losing focus on the information that matters. Current understanding of how transformers manage this is limited, and that gap makes it harder to predict when and why these models fail, or how to improve them.
Sam’s research investigates how a transformer’s ability to focus and reason changes as the number of tokens in a sequence increases, and how reasoning algorithms emerge from the model’s internal structure. His work is grounded in rigorous mathematical analysis of the dynamics operating inside transformer models.
Why the research matters
Explaining what models learn
A clearer mathematical picture of attention could help researchers understand what transformer models are learning and how they arrive at their outputs.
Identifying limitations
Understanding how a model’s focus changes with input length may help reveal where and why long-context reasoning becomes less reliable.
Supporting more dependable AI
The work could support more dependable approaches for models that need to work with large amounts of information, and gives developers a measure they can use to check how clearly a model is choosing between inputs.
Research approach
Sam studies transformers through theoretical analysis, building mathematical models that describe how attention behaves as sequences grow. Several ideas are central to his current work:
- Measuring attention in relative terms. A transformer’s attention scores always add up to a whole, so Sam compares them with each other. This separates scores that shrink simply because the input is longer from scores that shrink because the model has stopped making clear choices.
- Aitchison geometry. A branch of mathematics designed for data made up of parts of a whole, such as proportions. It offers a natural way to compare attention scores through the ratios between them.
- Token distinguishability. How clearly a model can tell apart, and choose between, the different tokens in its input.
- Temperature scaling. A way of adjusting how sharply attention scores are separated. Sam’s work examines how strongly the scores need adjusting to help a model make clear choices.
Research and outputs
Featured output: ICML 2026 conference paper
Modelling Attention with Aitchison Geometry: Token Distinguishability and Temperature Scaling
Sam Hilton-Jones, Timothy Norman and Zhanxing Zhu. International Conference on Machine Learning (ICML) 2026, Seoul.
The paper presents a mathematical framework for understanding how transformers distinguish between tokens when processing long sequences, and how temperature scaling shapes that process. It shows that, under idealised conditions, the relative differences between attention scores do not disappear as inputs grow longer. For real-world conditions, it sets out how strongly the scores need adjusting to help a model make clear choices, and reports that this relative measure gives useful information to people developing and deploying AI models.
Read the paper
Presenting at ICML 2026
In July 2026, Sam travelled to Seoul, South Korea, to present his first conference poster at ICML 2026. The conference brought him into conversation with researchers whose work he had been reading, and papers he discussed with their authors are now shaping the approach he takes in his current project.
Connect with Sam
Interested in large language models, transformer attention or the mathematics of AI? Connect with Sam through his professional profile, or contact the MINDS team about a wider research or collaboration enquiry.
Not sure whether this is the right research area? The MINDS team can help direct your enquiry.