Western researchers have developed a novel technique using math to understand exactly how neural networks make decisions - a widely recognized but poorly understood process in the field of machine learning.
Many of today's technologies, from digital assistants like Siri and ChatGPT to medical imaging and self-driving cars, are powered by machine learning. However, the neural networks - computer models inspired by the human brain - behind these machine learning systems have been difficult to understand, sometimes earning them the nickname "black boxes" among researchers.
"We create neural networks that can perform specific tasks, while also allowing us to solve the equations that govern the networks' activity," said Lyle Muller, mathematics professor and director of Western's Fields Lab for Network Science, part of the newly created Fields-Western Collaboration Centre. "This mathematical solution lets us 'open the black box' to understand precisely how the network does what it does."
The findings were published in the high impact journal PNAS, in collaboration with international researchers including University of Amsterdam's machine learning research chair Max Welling.
Lyle Muller, mathematics professor and director of Western's Fields Lab for Network Science, part of the Fields-Western Collaboration Centre. (Jeff Renaud/Western Communications)
'Seeing things' by segmenting images into parts
The Western team, which included Muller, post-doctoral scholars Luisa Liboni and Roberto Budzinski and graduate student Alex Busch, first demonstrated this new advancement on a task called image segmentation - a fundamental process in computer vision where machine learning systems divide images into distinct parts, like separating objects in an image from the background.
Starting with simple geometric shapes like squares and triangles, they created a neural network that could segment these basic images.
Muller and his collaborators next used a mathematical approach, which they previously developed to study other networks, to investigate how the new network performed this segmentation task when analyzing these simple images.
The mathematical approach allowed the team to understand precisely how each step of the computation occurred. Somewhat surprisingly, the team then found the network could also segment - or see and interpret - a handful of natural images, like photographs of a polar bear walking through the snow or a bird in the wild.
"By simplifying the process to gain mathematical insight, we were able to construct a network that was more flexible than previous approaches and also performed well on new inputs it had never seen," said Muller, a member of the Western Institute for Neuroscience.
"What's particularly exciting is that this is just the beginning, as we believe this mathematical understanding can be useful far beyond this first example." - Lyle Muller, mathematics professor and director of Western's Fields Lab for Network Science
The implications of the work extend beyond image processing.
In a related study published in Communications Physics in 2024, Muller and his team developed a similar "explainable" network that could perform various tasks, from basic logic operations to secure message-passing and memory functions.
In another collaboration with physiology and pharmacology professor Wataru Inoue and his research team at Schulich School of Medicine & Dentistry, they even successfully connected their network to a living brain cell, creating a hybrid system that bridges artificial and biological neural networks.
"This kind of fundamental understanding is crucial as we continue to develop more sophisticated AI systems that we can trust and rely on," said Budzinski, a post-doctoral scholar in the Fields Lab for Network Science.
