New research identifies fundamental weaknesses in AI language models

An academic at a Welsh university has co-authored a new study arguing that the most significant shortcomings of artificial intelligence stem from a fundamental difference between how humans and machines acquire knowledge.

Published in the journal AI & Society, the paper, ‘More Problems of Large Language Models in Comparison with Human Knowledge’, identifies 13 fundamental weaknesses in large language models (LLMs) such as ChatGPT and argues that human knowledge cannot be reduced to data alone.

The study, part of an internal research collaboration called the “Knowledge Observation Group”, is co-authored by Dr Simon Thorne, Senior Lecturer in Computer Science at Cardiff Metropolitan University.

It argues that human knowledge is developed through socialisation, education, and participation in communities, while LLMs are trained on vast quantities of internet text without experiencing the structured social processes that help people develop judgement, morality and an understanding of evidence.

The authors suggest this distinction helps explain why AI systems can appear fluent and authoritative while still producing inaccurate or even fabricated information.

Dr Simon Thorne said: “Large language models are exceptionally good at generating convincing language, but this is not the same thing as genuine understanding.

“Our research shows that many of the weaknesses we see in AI today can be traced back to the fact that that LLM learning approaches are not the equivalent to human socialisation or participation in social life.”

The paper suggests that humans develop a “moral compass” and what the researchers call a “classroom sensibility” from an early age.

These experiences teach people facts, as well as how to judge whether information is reliable, when evidence is sufficient, and when it is appropriate to admit uncertainty.

Current LLMs lack these foundations, instead operating with what they describe as a “response imperative” – a tendency to provide an answer whether or not the system has a reliable basis for producing that answer.

The study also raises concerns about the extent to which AI systems can be shaped by those who train, align, or modify them, highlighting the potential for political, commercial and ideological influence to affect outputs.

The authors argue that transparency and critical scrutiny will become increasingly important as AI systems become embedded in education, healthcare, business and public services.

The researchers emphasise that their findings do not diminish the usefulness of LLMs. Instead, they argue that greater recognition of the gap between human and machine intelligence is necessary for the safe and reliable deployment of AI technologies.

Dr Thorne added: “The challenge facing society is not simply how to build more powerful AI systems. It is how to ensure we understand their limitations, the sources of their knowledge, and the risks that arise when fluent language is mistaken for expertise.

“We have proposed a framework which we hope will provide researchers, policymakers and technology developers with a more systematic way of assessing AI systems and anticipating potential risks as the technology continues to evolve.”

More Problems of Large Language Models in Comparison with Human Knowledge is available to read here.

---

Get more trusted Welsh news

Choose Nation.Cymru as a preferred source in Google News to see more of our journalism.