MINDS CDT doctoral researcher | University of Southampton | MINDS CDT, 2023
Teaching transformer models to approximate solutions to differential equations
Xiyue Fan is investigating whether transformer models, the type of AI behind large language models, can learn from mathematical expressions and help approximate solutions to differential equations in a form that may be easier to inspect than numerical output alone.
Research at a glance
| Research focus | AI-assisted approximation of differential equations |
| MINDS theme | Task-Optimised Devices and Systems |
| Methods | Transformer models, symbolic mathematics, sequence modelling |
| Wider relevance | Science and engineering problems described by differential equations |
| Expected completion | 2027 |
Teaching AI to work with the mathematics of change
Differential equations describe how quantities change over time or space. They are used throughout physics, engineering, biology and climate science. Many cannot be solved exactly, and finding good approximate solutions can be difficult or computationally expensive.
Many current machine-learning approaches are data-driven, such as Physics-Informed Neural Networks (PINNs) and Neural Operators (NOs), while others combine numerical and symbolic data through methods such as symbolic regression. These approaches can lack interpretability or generalise poorly, and some remain sensitive to noisy data or face challenges in searching efficiently for a solution.
Xiyue is training a transformer model from scratch on purely symbolic data. Transformers are the architecture behind large language models, which learn patterns in sequences of words. In Xiyue’s work, the sequences are mathematical expressions. She is exploring deep representation techniques that could help the model learn better representations of these expressions, with the aim of producing more accurate approximate solutions.
Why this research matters
Supporting complex modelling
Where exact solutions are hard to find, AI-assisted methods could support researchers modelling physical, biological or environmental systems.
Solutions that can be inspected
Working with symbolic expressions may make it easier to see how a solution is represented, compared with a system that returns only a number.
Understanding the limits of AI
The work explores how far transformer methods can go with mathematical problems, which could help show where they are useful and where they fall short.
Research approach
Xiyue’s approach treats mathematics as a language. Equations and their solutions are written as symbolic expressions, such as x2 + sin(x), and converted into sequences of symbols that a transformer can process, much as a language model processes a sentence.
The model learns patterns in mathematical expressions. It does not reason about mathematics in the way a human mathematician does, and its outputs are approximations that need checking. A symbolic output may make it easier to check a proposed solution against the original equation.
Research keywords: Transformers, differential equations, symbolic AI, sequence modelling, mathematical approximation
Research and outputs
Featured output: Journal of Physics D: Applied Physics paper
Xiyue is one of eight co-authors of Temporal dynamics of TiOx memristor for reservoir computing applications, published in Journal of Physics D: Applied Physics, volume 59, article 015101 (2026).
This work sits alongside her transformer research. It studies memristors, electronic components whose resistance depends on the signals they have previously received, and how their behaviour over time could support reservoir computing, a hardware approach to processing time-dependent data.
Read the paper in Journal of Physics D: Applied Physics | View the University of Southampton repository record
Research profile
Related MINDS research
Reservoir computing in hardware is also explored by MINDS researcher Lianghong Ye, who is investigating whether tiny mechanical resonators can process time-dependent data. Explore Lianghong Ye’s research
Connect with Xiyue’s research
Interested in transformer-based mathematical reasoning, symbolic AI or differential equations? Contact the MINDS team about Xiyue’s research or a wider collaboration enquiry.