epistemic status: ongoing
I re-read 1984 by George Orwell, probably, once every two years. Having recently done so, the notion that societies dominated by a state, regardless of ideology, are totalising. This book has always helped reaffirm my libertarian persuasions. More so helped this year with a reading of the Origins of Totalitarianism by Hannah Arendt. So, imagine a government that no longer needs to tell its citizens what to think directly. Instead, it acquires influence over the systems they consult to understand events, assess evidence, compose arguments, and make decisions. Its preferred assumptions arrive in the language of helpful explanation. Citizens continue to ask questions and receive articulate answers. Yet the range of conclusions they can readily reach begins to narrow.
This introduction might prompt you (no pun intended) to anticipate this essay is in relation to controversial “Digital Duty of Care” legislation currently debated in Australia: it isn’t (although if you’re interested in this debate, I recommend this interview). Instead, the introduction is a political future I fear with convergence of cognitive dependence and fully fledged adoption of large language models. A future regime doesn’t need a neural implant to achieve this outcome. It only need the tooling to sufficiently influence a population, and that population be unable to evaluate the output of that tooling.
Use of language is inherited. From our family, friends, teachers, books, and media. This ties intellectual capacity to reason with assistance of these diverse and balanced sources. On face value, a LLM supports this level of information acquisition, parsing, and synthesis. Problems, like my introduction, have been explored in a recent paper to arise when we fiend for the answer and neglect any real labour of reason.
A recent paper titled Large-Language Models as a Cognitive Virus caused a stir online. Ricard Solé and colleagues modelled mathematical diffusion of LLM dependence across three buckets of user. People who aren’t interacting, those who do interact but as a tool within a broader self directed system of reasoning, and those who rely on LLMs for reasoning end to end. The framework emphasises movement of groups through these buckets, exploring methods of reversal as well as highlighting progression toward majority dependence. Although the authors avoid the implication of a specific threshold of no return, I think it is concerning that crossing a rubicon of dependence is a non-zero likelihood. A person might retain the biological capacity for independent reasoning while losing the habits, confidence, time, and institutional support needed to exercise it. If education and employment come to presume continuous AI assistance, proceeding unaided could become prohibitively difficult. At that point, original thought would remain possible, but sustaining and defending it might become untenable for many people. Dependence would be reinforced by the environment in which they must function.
I must emphasise that dependence alone does not produce totalitarianism. Orwell and Arendt clarify for us that danger arises when dependence converges with concentrated control, directed by harsh ideology, rule systems. An LLM’s response is shaped by learned parameters, training data, subsequent training, instructions, retrieved material, and deployment choices. Its weights do not contain a single coherent political doctrine waiting to be expressed. Nevertheless, choices across this system can influence which explanations are available, which sources receive attention, how disagreement is framed, and which requests are refused. The crucial question is what happens when the intermediary becomes difficult to challenge. Someone who consults a model while retaining independent expertise can identify omissions and dispute its framing. Someone who relies on the same model to supply the explanation, formulate the criticism, and judge the criticism has a weaker route to correction.
Consider a hypothetical authoritarian state with substantial control over domestic AI provision. It could influence the production and distribution of models through state ownership, procurement, licensing, or pressure on providers. Control need not extend to every parameter or every conversation. Defaults used across schools, workplaces, and public services could give the state considerable reach. Across training and deployment, the regime could privilege approved sources and reward answers consistent with its political commitments. It could shape what counts as a credible authority, an acceptable criticism, or a responsible recommendation. Selective emphasis, missing context, and unequal standards of scrutiny could gradually favour the official account.
A fly wheel of hyperideological feedback wouldn’t take long to emerge. By hyperideological beliefs, I mean commitments insulated from ordinary correction. Positions whose evidence is continually favoured and whose alternatives are dismissed before receiving fair consideration. A totalising regime and its LLM tooling could help create this insulation while presenting itself as patient, balanced, and practical. A user would encounter political selection within the ordinary work of understanding a question.
The hypothetical feedback loop of hyperideology is straightforward. People delegate intellectual work because the system is useful. Reduced practice and disappearing alternatives make independent evaluation harder. A politically captured system then shapes the interpretations available to increasingly dependent users. If those interpretations justify further restrictions on competing sources and institutions, the conditions for correction deteriorate again.
This is an extension of the dependence argument, not a result established by the cognitive virus model. It would require evidence about sustained use, changes in independent competence, institutional concentration, and political influence over deployed systems. Nor would model diversity automatically solve the problem. Several LLMs might still share important sources, incentives, or state imposed restrictions. Equally, a single widely used LLM would not necessarily create political dependence if people retained strong skills, accessible primary sources, and institutions capable of challenging it. A totalitarian outcome would require more than biased answers. It would involve power over education, public communication, association, and dissent. Cognitive dependence could make that broader project easier by weakening people’s capacity to recognise and contest the accounts supplied to them.
The hypothesis would be weakened if sustained LLM use reliably strengthened unaided reasoning, if users could readily detect political steering, and if independent alternatives remained viable. It would become more concerning if dependence deepened while those alternatives disappeared. The freedom at stake is the practical ability to form a judgement that survives disagreement with one’s tools. A society may continue to produce fluent essays, sophisticated arguments, and apparently lively debate while losing some of that ability. Intellectual output would then conceal a growing dependence on whoever controls its infrastructure.