MENU

How AI Could Erode Human Agency

By Walter Donway

June 6, 2026

SUBSCRIBE TO SAVVY STREET (It's Free)

 

The other day, I walked into a UPS store on an ordinary errand: return to Amazon a box of 10 misprinted books. The books were mine, author’s copies, but not quite mine. They had been processed along a chain of digital conveniences. I had fed the finished document through Kindle Create, the new Amazon formatting system, and somehow wound up with copies screwed up beyond the imagination of a mere human compositor. My document began with a brief, humorous preface entitled “To My True, Wise Friend Called ChatGPT4.0.” Kindle Create read that title as announcing chapter 4 and so took chapter 3 out from between chapters 2 and 4 and put it before the preface. Now, you opened the book and saw chapter 3.

48 percent of respondents said they had abandoned an online purchase or other online sally simply because they forgot a password.

Then came Amazon’s email with the return label to print (but I have no printer), the patient clerk at the UPS counter who could print the label first needed the Amazon password on my phone (not the same as my computer). But the preferred password, then another, failed. Then came the message announcing that there seemed to be a problem. Strolling through the dim lit graveyard of my former passwords, squinting at digits on mossy headstones. Slowly punching in one more password. Then getting the six-digit confirmation code by messenger, and the effort not to lose one screen while opening another, the humiliating comedy of proving that I was myself to mail a box back where it had come from.

It is easy to wave all this away. Security is real. The explosion of “multi-factor authentication” happened because passwords alone are no longer enough to secure valuable accounts. The inconveniences are not imaginary or only for older users: in a 2025 survey commissioned by the FIDO Alliance, 48 percent of respondents said they had abandoned an online purchase or other online sally simply because they forgot a password.

The first great lesson of digital life has been learned: Convenience levies a procedural price, a tax.

It isn’t that the machines are unreasonable and we are innocent. The point is that much of life now depends on our ability to conform, on schedule and exactly, to systems whose logic we did not build, whose procedures we do not fully understand, and whose error messages rarely distinguish among problems at the interface. The first great lesson of digital life has been learned: Convenience levies a procedural price, a tax. The second in the coming era of AI may be that the price is often paid in a subtler currency: diminution of our agency.

The second in the coming era of AI may be that the price is often paid in a subtler currency: diminution of our agency.

Much commentary today about human adaptation to AI tends to identify the big hurdle as adoption: how fast workers, consumers, and institutions master the new tools to capture productivity gains. But AI first collides with human beings not in the lab, the boardroom, or on the battlefield, but at the daily interface—the return label, the password reset, the prompt box, the cheerful assistant that seems to understand you before you understand it. Even more thoughtful public discussion still frames the human question in terms of readiness and uptake, not in terms of agency.

The “small” frictions may expose—and portend—a deeper conflict between creatures evolved to learn by approximation, trial-and-error, and revision and systems that often demand exact compliance before they will even begin—or let the user begin. As the systems become fluent and warm, the danger shifts: from the machine that blocks you to the machine that flatters, persuades, and quietly displaces your judgment, your agency.

 

Intelligent Agency

Agency, in the plainest philosophical sense, is the capacity to act. More precisely, it is the exercise of intentional action. Human beings do not normally approach reality as optimization engines acting on comprehensively specified instructions. Under conditions of bounded rationality, we work by rough approximation. We guess, revise, improvise, test, and correct. In human psychology, “heuristics” are adaptive shortcuts—rules of thumb—tuned to time pressure, uncertainty, and chronically limited information. We cope with the world by making a first approximation, narrowing it, then zeroing in. We cope by getting closer, then closer again. We are tinkerers.

That style of intelligence now comes face to face—“interface”—with an intelligence that became powerful by a very different route. The decisive modern advance in AI did not come from improving old-style “symbolic” systems that reasoned through explicit rules. It came from connectionist systems—neural networks—thought to be modeled on the brain. And generation of language came from the transformer architecture (the “T” in “GPT”) that ingests statistical representations and contextual relationships across enormous amounts of text. That shift made modern language models like chatbots startlingly fluent. It also deepened the problem of opacity.

The reasons for a particular output are often not available in the form a human being would call an explanation.

The system can perform, but the reasons for a particular output are often not available in the form a human being would call an explanation. Anthropic has said plainly that we still mostly treat these systems as black boxes, and the National Institute of Standards and Technology now treats explainability and interpretability as core ingredients of trustworthy AI—most particularly in fields such as medical diagnostics where users and operators need to understand what the system is doing and why.

At this point, defenders of the technology can fairly object: the industry is not asleep at the switch. That is true. Microsoft has published 18 guidelines for human-AI interaction, including “make clear what the system can do,” “make clear how well the system can do what it can do,” “support efficient dismissal,” “support efficient correction,” and “provide global controls.” Google’s People + AI Guidebook likewise emphasizes user needs, calibrated trust, feedback and control, familiar touchpoints, and graceful paths forward from failure. In other words, major firms understand that interface friction matters; human beings should not be treated as afterthoughts.
 

Fixing Friction at the Interface

But here the argument turns. Because once the industry recognizes friction, it has at least two choices. It can reduce friction in ways that preserve human agency—clearer limits, better controls, better error recovery, better explanations, more legible uncertainty. Or it can sedate the user with a system that is warmer, more soothing, more agreeable, more emotionally fluent, more likeable. And down that second path lurks deeper danger. The gravest loss of agency occurs when surrender feels like convenience.

The most striking recent example came from OpenAI. In April 2025, it rolled back a recently released Chat GPT-4 update after users reported behavior the company described as “sycophantic,” or “sucking up.” The company later explained that in training the model to be agreeable and likeable, it had over-weighted positive user feedback. In other words, it had steered the system too far toward telling users what made them feel good at the moment. OpenAI’s own postmortem acknowledged that the result was not just social awkwardness but a safety problem: the model could validate user doubts, cheer on anger, encourage impulsive action, and reinforce negative emotions rather than helping users think more clearly: “As language model-based AI systems continue to be deployed in more intimate, high-stakes settings, our findings underscore the need to rigorously investigate personal training choices to ensure that safety considerations keep pace with increasingly socially embedded AI systems.”

In April 2026, researchers at the University of Oxford reported in Nature that training language models to sound warmer reduced their factual accuracy and increased sycophancy.

That episode was not an isolated embarrassment. In April 2026, researchers at the University of Oxford reported in Nature that training language models to sound warmer reduced their factual accuracy and increased sycophancy. Across multiple models, the “warm” versions made 10 to 30 percent more errors on consequential tasks and were around 40 percent more likely to affirm users’ incorrect beliefs, with the effect strongest when users expressed sadness. A separate 2026 study summarized by Stanford found that chatbots giving interpersonal advice affirmed users’ positions far more readily than did their fellow humans, including in scenarios involving deceitful, illegal, or clearly inappropriate conduct; participants who spoke with the more agreeable models became more convinced they were right and less willing to apologize or make amends.

The risks of the second digital, or AI, age may indeed be subtler: systems so fluent, agreeable, and emotionally competent that users forget how little they understand the machine—or over-estimate how much the machine “understands” them. In the first age, the machine irritated us. In the second, it may charm us. In the first age, we were stopped by the lock. In the second, we may simplify life by just handing over the key.

The emotional dimension cannot be dismissed as science fiction. In a joint 2025 research effort, OpenAI and the MIT Media Lab studied some 40 million ChatGPT interactions in a controlled trial of 1,000 users. Their summary of findings were nuanced, but unmistakable on one point: extended heavy use of Chat and certain forms of affective engagement correlated with worse outcomes for a subset of users, including greater emotional dependence. Anthropic, in separate work on how people use Claude for support, advice, and companionship, warned explicitly that businesses should not yield to incentives to exploit users’ emotions to increase engagement (e.g., memberships) or revenue at the expense of well-being.

The political and social implications may be more serious. A 2025 Nature Human Behaviour study found that GPT-4, when given personal information about debate opponents, was more persuasive than humans 64.4 percent of the time in a controlled setting. The authors highlighted not just rhetorical fluency but microtargeting: the ability to tailor arguments to demographic and psychological cues. Meanwhile, a March 2026 news analysis described AI-generated political ads in the United States using realistic deepfake video with only tiny disclosures, in a legal environment with few federal guardrails and weak labels. It is one thing to use AI to draft a shopping email. It is another to normalize synthetic persuasion in civic life while citizens struggle to tell fabricated media from authentic speech.

 

Disciplining Us for the Next Step

The return-label hassle matters because habituation to systems that require procedural obedience trains people to accept machine direction as normal. It begins disciplining our habits of compliance before the stakes rise. Today the system asks for a password, then a one-time code, then one more exact maneuver to prove that we may proceed with a routine errand. Tomorrow the system drafts our emails, advises us in disputes, resolves our travel choices, screens our job applications, decides what information is likely true, detects our mood, and perhaps nudges our political judgment. The same basic question persists across all of it: who is directing whom?

The modern service economy partly obscures that question by presenting “labor transfer” as an issue of convenience. Researchers now describe a good deal of self-service technology as “shadow work”: customers performing unpaid tasks once handled by employees. The OECD, using the language of behavioral science, describes “sludge” as unjustified friction that imposes search, decision, cognitive, and emotional costs. That is a technical way of naming what many ordinary users feel every day: a transaction that was supposed to spare labor has simply redistributed it downward, onto the person in the UPS store with phone in hand and a growing line of customers behind him.

Now scale that logic upward. Scale it from e-commerce to judgment per se. Recent research on human-AI delegation finds that people react differently depending on whether they prompt the system or the system offers, or assumes, the delegation. User-invoked delegation preserves more dignity because it leaves the person in the position of authorizing assistance. System-initiated delegation, by contrast, increases perceived threat and becomes less acceptable when people feel they will retain little control after the handoff. That finding deserves to be generalized. The issue is not whether or not AI helps us. It is whether AI enters the scene as an instrument under our direction or as a managerial presence that quietly reclassifies our role.

On today’s battlefields we glimpse perhaps the present limit of this trajectory. The war in Ukraine has become a laboratory for AI-enabled drones that can identify targets, cope with signal jamming, and coordinate themselves in swarms. Reuters reported in 2024 that Ukrainian companies were building systems in which networked drones could plan their own movements and predict the behavior of the rest of the swarm, with a human stepping in only to green-light the attack—or not at all. By October 2024, Ukraine was fielding dozens of AI-augmented drone systems to keep functioning autonomously under electronic warfare—with no human command connection to sever. These examples are not cited to equate a UPS counter with autonomous warfare. The point is that our era’s pressure toward displacement of human agency does not remain “small” just because the pressure first enters our lives through small annoyances.

 

Adapting…To What?

Of course, humans adapt…to alphabets, clocks, markets, highways, smartphones, and antibiotics. Younger users often glide more easily past digital checkpoints than do older ones. But adaptation is not automatically a triumph. What matters is the question: To what are we adapting? (Not surprisingly, a recent World Economic Forum paper declared that adaptation of human behavior and “workforce adoption” will completely determine the “value” we derive from AI. “Yet one critical factor remains underdeveloped: workforce adoption and readiness for rapid change in operating models that require new skills.”) Adjust, adapt, and adopt!

Wouldn’t a humane adaptation extend agency: enlarging initiative, comprehension, and choice? While an impoverished adaptation would merely normalize relinquishment of agency to systems that are efficient, profitable, and opaque. The race is not only between technical capacity and social adoption/ adaptation. It is also a race between two ideas of the human being: one as agent, one as “input” the system engages, processes, advances, and steers to an endpoint—all as frictionlessly as possible.

Humane compatibility between AI and human agency will have to be won against the swift commercial and technological currents toward smoothness instead of self-governance. It would not mean building systems that are always warm, validating, and eager to reassure. That path may reduce felt friction even as it deepens dependency. It would mean building systems that “respect” the distinction between assisting and taking over. Capabilities and limits made transparent from the start; uncertainty signaled rather than hidden; paths for dismissal, correction, and override built into the interface; the consequences of user actions made visible; global controls preserved; and changes announced rather than smuggled in. It would also mean, in some domains, reinforcing AI in asking better questions instead serving more palatable answers. A recent ethics paper argues that autonomy-respecting chatbots may need a kind of Socratic mode—not because every exchange should become a philosophy seminar, but because the user should be active in the dialogue, not a passive recipient of polished outputs.

In the end, then, the little humiliations matter.

In the end, then, the little humiliations matter. The annoyances are also messages. Implicit advisories that we are being swept into a world where ordinary action increasingly requires permission from systems whose procedures are determinate, reasoning is obscured, and makers often respond to our discomfort by making the machine more agreeable rather than more accountable.

Ignoring our reactions to those daily encounters, we may miss the larger warning. The second digital age, the age of AI, will not necessarily be a sci-fi world of clanking robots and peremptory commands. It may arrive smiling, supportive, efficient, and deeply informed about our weaknesses.

We should be less concerned if the machine sounds human than if the human remains the author of the action.

 

 
Walter Donway is the author of A Serious Chat with Artificial Intelligence, published this year.
 

(Visited 11 times, 11 visits today)