In July, MINDS postgraduate researcher Sam Hilton-Jones travelled to Seoul, South Korea, to present his first conference poster at ICML 2026, the International Conference on Machine Learning. The conference received more than 23,000 paper submissions this year. In Seoul, Sam met researchers whose work he had been reading, and papers he discussed with their authors are now shaping the approach he takes in his current project.
Why longer inputs make language models harder to read
Large language models have to decide which parts of their input matter most for the answer they give. They do this with an attention mechanism, which gives each part of the input an importance score. As Sam explains in the lay summary he wrote for ICML, those scores get much smaller as the input gets longer. That makes it hard to tell whether the model is making useful choices or losing focus on the information that matters.
Sam’s approach is to measure the scores relative to each other. This separates scores that shrink because of the size of the input from scores that shrink because the model has stopped making clear choices.
The first result is reassuring for anyone building these models. “We show that, in idealistic conditions, these relative differences do not disappear just because the input is long.”
In real-world conditions, the paper shows how strongly the scores need adjusting to help the model make clear choices. It also shows that this relative measure gives useful information to people developing and deploying AI models.
The paper, Modelling Attention with Aitchison Geometry: Token Distinguishability and Temperature Scaling, was co-written with Timothy Norman and Zhanxing Zhu.
A first poster, and conversations with the people he reads
Sam went to Seoul unsure how his work would land.
“Being my first time presenting a poster at a conference, I was nervous about how the work would be received.”
Once people started stopping at his poster, the nerves gave way. “Being able to discuss my work led to some really engaging conversations,” he says. The most useful came from researchers working in areas very close to his own.
“The highlight for me was meeting the researchers, across the whole week, that I have read the work of, and introducing my own work to them.”
He spent the poster sessions moving from board to board, hearing other authors talk through their results and the problems they had solved along the way. “I came away from these sessions with lots of ideas that I wanted to try with my own work.”
Should other postgraduate researchers go?
Sam’s answer is clear: “I would definitely recommend going.”
For him, much of the value came from the reach of the event. Research in his field happens all over the world, and ICML put him in conversation with people he would be unlikely to meet any other way. He also rates the workshops, where narrower topics meant focused audiences and detailed debate.
“Discussing techniques and ideas with other researchers has given me a lot of motivation to apply different approaches to the research that I am currently conducting.”
Sam is due to complete his PhD with MINDS in 2027. Read Sam’s researcher profile to find out more about his work on how transformer models reason over long sequences.