Cognitive processes are not something that we generally pay much attention to until something goes wrong, but they cover the entire scope of us ingesting sensory information, the processing and recalling thereof, as well as any resulting decisions made based on such internal deliberation.
Within that context there has also long been a struggle between those who feel that it’s fine for humans to rely on available technologies to make tasks like information recall and calculations easier, and those who insist that a human should be perfectly capable of doing such tasks without any assistance. Plato argued that reading and writing hurt our ability to memorize, and for the longest time it was deemed inappropriate for students to even consider taking one of those newfangled digital calculators into an exam, while now we have many arguing that using an ‘AI’ is the equivalent of using a calculator.
At the root of this conundrum lies the distinction between that which enhances and that which hampers human cognition. When does one merely offload tasks to a device or object, and when does one harm one’s own cognition?
Surrender Versus Offloading
Cognitive offloading is the practice of shifting cognitive tasks to external aids, and it is thought to make learning complex tasks easier. In contrast to rote memorization of facts like dates of events and formulas, if we consider books to be an external memory storage device, then we can offload such precise memorization to their pages and only require from students that they are capable of efficiently finding information, as well as the judging of these on their merits.
An often misquoted anecdote here pertains to Albert Einstein, who was was once asked why he couldn’t cite the speed of sound from memory. To this he responded with a curt:
[I do not] carry such information in my mind since it is readily available in books. …The value of a college education is not the learning of many facts but the training of the mind to think.
With this statement Einstein makes a clear case for the benefits of cognitive offloading in the sense that rote memorization does not enhance one’s cognition. Similarly, the ability to solve complicated equations and sums without so much as the use of pen and paper is fairly irrelevant when a slide rule and a digital calculator can offload all that work. As a benefit these devices tend to be more precise, faster and very accessible.
It is still important to have an intuitive feeling for whether a calculation is in the expected range, and one should never assume that what is written in a book is the absolute truth. That in a nutshell is the key difference between cognitive offloading and cognitive surrender. If you have entered a series of values into your calculator, the result seems off and you re-type them to be sure, that’s cognitive offloading.
If, however, you accept the outcome of such a calculation, or a text as written without a second thought, that constitutes surrendering an essential part of your cognitive processes to an external source. If we thus replace ‘calculator’ in this context with ‘LLM chatbot’ or an ‘AI summary’, the same caveat applies. Perhaps more so as at least a calculator is fully deterministic and can be proven to be mathematically correct.
So if that’s the case, and modern-day ‘AI’ isn’t really what it’s often cracked up to be, why would a presumably intelligent human being end up accepting their outputs like the literal gospel?
External Cognition
A recent study (DOI link) by Steven D. Shaw and Gideon Nave of the University of Pennsylvania investigated the prevalence of cognitive surrender in the context of LLM chatbots, looking for instances where users are seen to blindly accept the generated answers.
In this study, Shaw et al. had three groups of volunteers take a standardized test, during which one group had to rely purely on their own wits, the second group could use an LLM chatbot which gave correct answers, while a third group also had access to this chatbot, but for them it gave wrong answers.
System 3 facilitates cognitive surrender. (Credit: Shaw et al., 2026)
Perhaps unsurprisingly, the test subjects used the chatbot quite a lot when available, with predictable results. In the ‘tri-system theory of cognition’ that Shaw et al. propose in the paper, the external cognitive system (‘System 3’) is that of the chatbot, whose output is clearly being accepted verbatim by a significant part of the test subjects. If said chatbot output is correct, this is great, but when it’s not, the test results massively suffer.
Where this is worrisome outside of such a self-contained tests is that people are exposed to endless amounts of faulty LLM-generated text, such as for example in the form of ‘AI summaries’ that search engines love to put front and center these days. Back in 2024, for example, Avram Piltch over at Tom’s Hardware compiled a amusing collection of such faulty outputs, some of which are easier to spot than others.
Ranging from the health effects of eating nose pickings to the speed difference between USB 3.2 Gen 1 and USB 3.0, to classics like adding Elmer’s glue to pizza sauce, it’s generally possible to find where on the internet a ridiculous claim was scraped from for the LLM’s dataset, while other types of faulty output are simply due to an LLM not possessing any intelligence or essentials like grasping what a context is.
Meanwhile other types of output are clearly confabulations, a fact which ought to be obvious to any intelligent human being, and yet it seems that so much of it passes whatever sniff test occurs within the cognitive capabilities of the average person.
Making Decisions
Anterior cingulate gyrus. (Credit: BodyParts3D, Wikimedia)
In the generally accepted model of cognitive decision making we see two internal systems: the first is the fast, intuitive and emotion-driven system. The second is the deliberate and analytical system, which tends to take a backseat to the first system in general, but could be said to be checking the homework of the first.
Although psychology is hardly an exact science, in the scientific fields of systems neuroscience and cognitive neuroscience we can find evidence for how decisions are made in the primate brain – including those of humans – with various cortices involved in the decision-making process. Fascinating here is the activity observed in the parietal cortex where a decision is not only formed, but also apparently assigned a degree of confidence.
Lesions in the anterior cingulate cortex (ACC) have been linked to impaired decision making and the arisal of impulse control issues, as the ACC appears to be instrumental in error detection. Issues in the ACC are thus more likely to result in faulty or flawed decisions and judgements passing by uncorrected. Incidentally, the ACC was found to be heavily affected by environmental tetraethyl lead contamination, underlying the theory that leaded gasoline was responsible for a surge in crime until this additive was discontinued.
Knowing this, we can thus say with a fairly high degree of confidence that the concept of human cognition is very much determined by the physical wiring in the pinkish-white goo that constitutes our brains. A good demonstration of this is the effect of ethanol on the brain, as well as the intense cravings that accompany addictions.
Implanted electrodes in the rostrodorsal anterior cingulate cortex (rdACC) for alcoholism treatment. (Credit: Sook Ling Leong et al., 2020, Neurotherapeutics)
Abnormal activity in the ACC has for example been associated with alcohol addiction, with an implant suggested to adjust said neural activity as detailed in a 2020 Neurotherapeutics study by Sook Ling Leong et al. In this study the eight treatment-resistant alcoholics had electrodes inserted into part of their ACC to provide direct stimulation, leading to a self-reported 60% drop in cravings.
As ethanol can freely pass through the blood-brain barrier, it is free to start binding with GABA receptors and induce the release of dopamine along with a range of other neurological effects that initially induce a feeling of relaxation and well-being, but also suppresses activity in various cortices, including the ACC. Effectively ethanol thus reduces one’s cognitive prowess and with it the ability to recognize flawed decisions.
From this we can thus deduce that activity in the ACC is not only essential for decision-making, but it also illustrates how the pinkish goop in our skulls is a fascinating biochemistry and neurochemistry experiment in which the addition or subtraction of certain substances and poking it with electrodes can induce a wide variety of cognitive outcomes.
Experiments aside, we started our lives off with the baseline that we were born with (‘nature’) and the various neuroplastic alterations made as we grew up (‘nurture’), which along the way led to various cognitive outcomes that we may or may not regret as adults. This leaves us free to learn from our mistakes and do better in as far as neuroplasticity allows.
Asking Why
It’s often said that the most valuable skill in life that adults tend to lose as we mature out of innocent childhood is the incessant ability to ask ‘Why?’. By questioning everything and wanting to know everything, we not only display curiosity, but also nurture the cognitive skills of our brain. If instead our environment pushes back against this, it can harm the development of such cognitive skills, even if the pushback doesn’t rise to the level of childhood trauma.
As a certified ‘nerdy kid’ back in the day who went through all the motions of being bullied, shoved into proverbial lockers and other types of physical abuse at school for having the nerve to like books, science and other ‘nerdy’ things that involved being curious, it’s hard not to feel the social pressure to simply comply and not question things. As an adult such social pressure only gets worse, with skills like critical thinking generally discouraged.
Of course, said critical thinking is exactly what we need when confronted with new technologies and the temptation to simply surrender that cognitive burden instead of asking questions. Yet when cognitive surrendering can have real consequences that may affect not just your own life but also those of others, it’s pretty much a basic survival skill to weapon yourself against it.
In a world where things like politics, idols, religion, and advertising exist, the rise of this purported ‘AI’ in the form of LLM-based chatbots with their often very convincingly human-like and authoritative outputs seem to have hit the same weaknesses that unscrupulous religious leaders and scammers exploit, with sometimes tragic consequences.
Although it’s clear that believing some factual misinformation generated by a chatbot is a far cry from deciding to take fatal actions based on a dialog with said chatbot, it also highlights the importance of retaining your critical thinking skills. Although we often like to think otherwise, people aren’t fully rational beings whose cognitive processes belong completely to themselves.
Answering the question of when we harm our own cognition, it would seem that while we can generally trust a calculator, an LLM-based chatbot is not nearly as reliable or benign. Caution and awareness of the risk of cognitive surrendering are thus well-warranted.
Instagram Engineering
Shopify Engineering
