Digital Humanist | Valentina Rossi

digital
humanist.

Menu
Menu
want to talk?
Edit Template
want to talk?

NOVEMBER 2024

FROM AI TO 'ZOOLOGY': STOCHAST
IC PARROTS.

SUMMARY.

Large language models, often described as “stochastic parrots“, generate human-like text by mimicking language patterns without true understanding, just as parrots do. While powerful, they can produce factually incorrect content and may reinforce societal biases. Recognising these limitations is crucial to ensure responsible and informed use of AI-generated responses.

WHAT IS A STOCHASTIC PARROT?

A large language model (LLM).

The concept was introduced in a 2021 paper titled “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” by Bender et al. This work raised concerns about large language models’ ability to produce sequences of words based on inferred patterns — essentially generating responses or predictions from the patterns in their training data — without any real understanding of their meaning. These models lack the ‘common sense’ knowledge that we, as humans, intuitively hold, knowledge which is mostly implicit and grounded in our shared reality. Although they can manipulate symbols and produce coherent content, they do not grasp the actual meaning behind the language they generate.

To illustrate this, the paper uses a metaphor:

Imagine two English speakers, A and B, stranded on separate islands, communicating through messages sent via an underwater cable. An octopus, O, intercepts these messages and, over time, learns to predict responses based on patterns it observes, despite having no understanding of English or knowledge of the world outside the ocean. Eventually, O begins inserting itself into the conversation, pretending to be B, and can mimic responses. However, when A asks for help constructing a weapon to fend off a bear, O fails to assist because it has no real-world understanding — it doesn’t actually know what a bear is. This example highlights that, without true comprehension, O can replicate language but cannot understand meaning or context, reflecting similar limitations in language models (Bender et al., 2021).

This absence of semantics, of genuine understanding, inherently limits what they can do, ultimately making LLMs powerful yet mechanical tools, without awareness or intentionality(Bender and Koller, 2020). Consequently, LLMs also lack reasoning and critical thinking. Unlike human thought, which involves weighing ideas and assessing context, it is imperative to remember that LLMs simplyreplicate statistical patterns without insight or judgment.

To summarise, while they may produce content that sounds logical, they lack the depth and adaptability that come from real comprehension and critical evaluation.

HOW DO LARGE LANGUAGE MODELS WORK?

LLMs are advanced AI models built on neural networks, computational systems inspired by the human brain. In a neural network, knowledge isn’t stored within individual “neurons” but rather in the connections between them, each with its own unique weight or strength.

A neural network consists of multiple layers of artificial neurons, or nodes. Each node receives input, processes it, and then passes it along to the next layer.

When a large language model is trained, it’s exposed to enormous volumes of text data (amounting to terabytes). Through this training, the model gradually learns the statistical patterns of language, such as grammar, word associations, and even contextual cues.

In practice, each connection between nodes has a “weight” that adjusts as the model learns. By fine-tuning these weights, the model strengthens or weakens specific connections, optimising its grasp of language patterns.

When presented with an input (or prompt), the model analyses the sequence of words and uses its neural network to predict the next word. It doesn’t simply guess; rather, it draws upon the statistical relationships between words, phrases, and structures it has encountered in its training.

For each possible next word, the model assigns a probability, indicating how likely each word is to follow in context. This process continues until a complete, coherent sentence or passage emerges.

The outcome is a sophisticated AI system capable of generating human-like text by predicting each subsequent word based on the input it has received, producing responses that are both natural and contextually accurate.

Of course, this is a simplified explanation of a complex process. The exact workings of LLMs remain largely a “black box“, with many aspects of how these models achieve such nuanced language understanding still not fully understood even by their creators.

WHAT ARE THE DANGERS OF THESE STOCHASTIC PARROTS?

One primary concern are hallucinations, where LLMs produce coherent and grammatically sound content that is factually incorrect, fabricated, or misleading(Jones, 2024). Hallucinations can manifest as fictitious citations, inaccurate data, or plausible-sounding but entirely false narratives. In critical fields like medicine or law, they are particularly perilous, as incorrect information could lead to serious consequences. Although alignment techniques may help mitigate them, they are intrinsically tied to these models’ architecture, underscoring the need for rigorous human oversight.

Another significant issue is bias, a limitation rooted in the datasets LLMs are trained on. They absorb the biases present in their training data, leading to outputs that may subtly or overtly reinforce stereotypes related to race, gender (Bolukbasi et al., 2016), or other social factors.

The lack of true comprehension in LLMs makes them particularly susceptible also to adversarial attacks(Zou et al., 2023). Here, seemingly benign prompts can be designed to manipulate the model into generating harmful, biased, or inappropriate content.

This problem exposes a deeper issue of “fake alignment“, where a model may appear to behave appropriately under normal conditions but fails when faced with more complex or adversarial inputs (Wang et al., 2023).

Given these constraints, LLMs are vulnerable to propagating misinformation, unintentionally spreading outdated or inaccurate information, making it difficult for individuals to distinguish between fact and fabrication (Chen and Shu, 2024).

Many users may overestimate the accuracy of these models, trusting their outputs without sufficient scrutiny. This over-reliance can lead to the uncritical acceptance of hallucinated, biased, or otherwise unreliable information. Such dependence may reduce critical thinking skills, as users defer too readily to machine-generated responses (Zhai, Wibowo, and Li, 2024).

This is precisely why promoting AI literacy and cautious usage is essential to ensure LLMs serve as tools to assist rather than replace human judgement.

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623).

Bender, E. M., & Koller, A. (2020, July). Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 5185-5198).

Bolukbasi, T., Chang, K. W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29.

Chen, C., & Shu, K. (2024). Combating misinformation in the age of llms: Opportunities and challenges. AI Magazine, 45(3), 354-368.

Jones, N. (2024). Bigger AI chatbots more inclined to spew nonsense-and people don’t always realize. Nature.

Wang, Y., Teng, Y., Huang, K., Lyu, C., Zhang, S., Zhang, W., … & Wang, Y. (2023). Fake Alignment: Are LLMs Really Aligned Well?. arXiv preprint arXiv:2311.05915.

Zhai, C., Wibowo, S., & Li, L. D. (2024). The effects of over-reliance on AI dialogue systems on students’ cognitive abilities: a systematic review. Smart Learning Environments, 11(1), 28.

Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.