Shelly Fan in Singularity Hub:
“Attention is all you need.”
This 2017 breakthrough idea transformed AI. The concept of self-attention became the foundation of today’s chatbots. Claude, Gemini, and ChatGPT are all large language models (LLMs), AI systems designed to focus on the matter at hand while filtering out distractions. The results have been remarkable. From brainstorming recipes to generating code, apps, websites, and content, LLMs are being woven into our lives at breakneck speed. But now, a City University of New York team and collaborators are asking: How closely does AI self-attention resemble human attention? It’s not just academic curiosity. AI researchers have long looked to the brain for ideas to improve machine intelligence. In turn, AI models have offered new ways to investigate the brain. Comparing artificial and biological attention could inspire AI that concentrates more like us.
In their study, the team asked multiple chatbots to complete a classic psychology test of attention and cognitive control. Participants are shown the word for a color—such as “red”—written in either the same or a different color than the one the word describes. The challenge is to name the ink color while ignoring the word itself. On short word lists, the chatbots performed at a high level. But as the tasks grew longer, their focus faltered. Instead of naming the ink color, they increasingly defaulted to reading the word. Under more demanding conditions—ones that also trip up people—their performance nearly collapsed.
More here.
Enjoying the content on 3QD? Help keep us going by donating now.