How do you intuitively know that you can walk on a footpath and swim in a lake? Researchers from the University of Amsterdam have discovered unique brain activations that reflect how we can move our bodies through an environment. The study not only sheds new light on how the human brain works, but also shows where artificial intelligence is lagging behind. According to the researchers, AI could become more sustainable and human-friendly if it incorporated this knowledge about the human brain.
When we see a picture of an unfamiliar environment -- a mountain path, a busy street, or a river -- we immediately know how we could move around in it: walk, cycle, swim or not go any further. That sounds simple, but how does your brain actually determine these action opportunities?
PhD student Clemens Bartnik and a team of co-authors show how we make estimates of possible actions thanks to unique brain patterns. The team, led by computational neuroscientist Iris Groen, also compared this human ability with a large number of AI models, including ChatGPT. "AI models turned out to be less good at this and still have a lot to learn from the efficient human brain," Groen concludes.
Viewing images in the MRI scanner
Using an MRI scanner, the team investigated what happens in the brain when people look at various photos of indoor and outdoor environments. The participants used a button to indicate whether the image invited them to walk, cycle, drive, swim, boat or climb. At the same time, their brain activity was measured.
"We wanted to know: when you look at a scene, do you mainly see what is there -- such as objects or colors -- or do you also automatically see what you can do with it," says Groen. "Psychologists call the latter "affordances" -- opportunities for action; imagine a staircase that you can climb, or an open field that you can run through."
Unique processes in the brain
The team discovered that certain areas in the visual cortex become active in a way that cannot be explained by visible objects in the image. "What we saw was unique," says Groen. "These brain areas not only represent what can be seen, but also what you can do with it." The brain did this even when participants were not given an explicit action instruction. 'These action possibilities are therefore processed automatically," says Groen. "Even if you do not consciously think about what you can do in an environment, your brain still registers it."
The research thus demonstrates for the first time that affordances are not only a psychological concept, but also a measurable property of our brains.
What AI doesn't understand yet
... continue reading