Berkeley researchers find that AI language understanding is improving, chatbots can analyse and diagram complex sentences like trained linguists, challenging long-held beliefs about language understanding being uniquely human.
AI chatbots can now ‘reflect on how language is organised’, according to new research from UC Berkeley, suggesting that large language models (LLMs) are beginning to master metalinguistics and natural language understanding — the uniquely human ability to think about and analyse language itself.
‘Our new findings suggest that the most advanced LLMs are beginning to bridge that gap’, said Gašper Beguš, Berkeley Associate Professor of Linguistics and lead author of the study. ‘Not only can they use language, they can reflect on how language is organised.’
Can AI understand complex language structures?
The study, due to appear in IEEE Transactions on Artificial Intelligence, tested multiple versions of OpenAI’s ChatGPT alongside Meta’s Llama 3.1. Beguš and his team fed them 120 complex sentences, instructing them to analyse each, identify specific linguistic features and create ‘syntactic trees’ — visual diagrams that break down sentence structures.
For you
Be part of something bigger, join BCS, The Chartered Institute for IT.
In the ambiguous sentence ‘Eliza wanted her cast out’, only OpenAI’s o1 model detected the dual meaning — whether Eliza wanted someone expelled, or a physical cast removed — and accurately diagrammed it. ‘That was revealing and signalled the improvements the models were making’, Beguš noted.
The researchers also explored recursion — the human ability to embed phrases within phrases, leading to potentially infinite nesting, as in ‘The dog that chased the cat that climbed the tree barked loudly.’ First theorised by Noam Chomsky, recursion has long been viewed as a defining feature of human language separating us from other animals.
Testing AI language comprehension
To test this, Beguš’ team asked the AI models to spot recursion in sample sentences, identify its type, and add another recursive clause. Faced with the sentence ‘unidentified flying objects may have conflicting characteristics’, OpenAI’s o1 detected the recursive structure — ‘flying’ modifying ‘objects’ and ‘unidentified’ modifying ‘flying objects’ — diagrammed it, and further expanded the sentence to: ‘Unidentified recently sighted flying objects may have conflicting characteristics.’
‘In these models, we have one of the rare things that we thought was human-only,' Beguš observed. The team concluded that ‘o1 significantly outperformed all others,’ hinting that AI is edging closer to human-like language comprehension.
Read more about the study's findings on the UCS Berkely News website.
