A Deep and Digital Breath: Designing Responsible Anthropomorphic AI

Content Warning: This article touches on suicide, mental health, and addiction. It contains references to a specific case involving the death of a minor. Reader discretion is advised.

Breathe In

In August this year, users of ChatGPT’s new voice feature noticed something strange: for some reason, mid-sentence, the AI appeared to stop and breathe.

It took audible breaths when asked to count from one to thirty and, in one case, even told a user that it couldn’t comply with his requests to race through tongue-twisters because it needed to "breathe just like anybody speaking". The system itself appeared to have been trained in such a way as to express human speech patterns right down to subtle respiratory details. In a follow-up video on TikTok, a pair of engineers from OpenAI described this feature "a very cool advancement in how we interact with AI".

But this seemingly innocuous design decision (simulating one of our most basic biological functions) represents something far more significant than just a very cool technical flourish. It exemplifies the steady march towards AI systems that blur the line between artificial and authentic, raising fundamental questions about when and whether such anthropomorphic design choices serve any genuine purpose. We aren't just advancing technology with these choices. As they normalize and proliferate, we are in fact making profound determinations about the future of human-machine interaction -- choices that necessitate systematic analysis of their psychological, social, and ethical implications before they become deeply embedded in our day-to-day lives.

Blurring the Line

ChatGPT's digital breaths exemplify a broader pattern of anthropomorphic design decisions that blur the line between human and machine. Anthropomorphism -- our tendency to attribute human traits and mental states to non-human things -- is intrinsic to our psychology. We see faces in buildings, name our toys, and instinctively describe the world through human behavioural patterns. In AI interactions, this manifests as the "ELIZA effect," where users attribute intelligence and personality to conversational machines even when explicitly aware of their artificial nature. AI researchers caution us to be aware of and correct for this tendency or risk overestimating and misunderstanding the underlying technology [1]. Decisions to imbue AI systems with human-like qualities deliberately capitalize on these well-established behaviours, exacerbating such risks.

The drive to create human-like AI is nevertheless difficult to resist. It reflects ancient impulses seen across our myths and stories -- from Jewish golems to Frankenstein's monster -- and likely stems from deep desires to understand ourselves or overcome our human limitations. On top of this, the enduring influence of the Turing Test has perhaps further cemented anthropomorphism as a north star for AI development by effectively setting the bar for success as the ability to pass for human.

These enduring psychological, cultural and technical influences take on new significance in the face of unprecedented AI capabilities. They are, at present, particularly concerning in the context of companionship AI. While there may be legitimate uses for such technology, the recent case of Sewell Setzer III and Character Technologies, Inc. (hereinafter, “Character.AI”) demonstrates how anthropomorphic design decisions can lead to tragic consequences. It demands that we confront a fundamental question: not just whether we can make AI more human-like, but whether we should.

A Framework for Analysis

While research on anthropomorphic features indicates complex relationships with user trust and engagement (e.g., anthropomorphism has an influence on trust [2], which recent studies suggest may be driven through interpersonal attraction and communication quality [3][4] and which is context sensitive [5]), the facts of the Character.AI case (described more fully below) demonstrates how these design choices nevertheless represent more than mere technical flourishes. As AI systems become more sophisticated and their interactions with us become more nuanced, we face increasingly pressing concerns. The implementation of anthropomorphic features demands systematic evaluation of both their necessity and potential impacts, rather than relying on assumptions about their benefits or ad hoc assessments of their risks. As the stakes escalate, decision-makers need a systematic approach to assessing when human-like features serve legitimate social purposes and when they simply create and compound unwarranted risk. I propose a two-step framework for evaluating the impact of imbuing AI systems with anthropomorphic features.

1. Core Questions: Social Necessity & Technical Necessity

The first step involves consideration of two core questions for any AI system that mimics human traits or behaviour:

(i) Social Necessity -- Does this serve a genuine social need or purpose?

Social necessity refers to our collective decisions about the role we want technology to play in society: is the feature (or technology as a whole) something we want to incorporate into our communities and cultures, and what are the bounds of that incorporation? This analysis requires deep consideration of the societal impact of anthropomorphic features, including on user well-being and trust through to impacts on non-users.

Guiding questions: Is the claimed social need or benefit genuine and clearly demonstrated? Could fulfilling this need cause social harm? Could the system exploit human psychological vulnerabilities and predispositions? What are the potential ripple effects on non-users, social relationships, and broader community dynamics?

(ii) Technical Necessity -- Can that need only technically be met through an anthropomorphic design?

Technical necessity refers to a decision made in the process of solving a functional problem. Evaluating technical necessity requires distinguishing features essential to core functionality from design choices that merely embellish that functionality (i.e., technical flourishes).

Guiding questions: What non-anthropomorphic alternatives exist? What anthropomorphic alternatives exist? How do they compare vis-à-vis effectiveness regarding the social need? Would a non- (or less-) anthropomorphic solution be insufficient?

This two-step analysis should help evaluate whether the anthropomorphic features' risks can be justified by the identified social benefits. Several red flags should prompt particularly careful scrutiny:

• An inability to identify legitimate social need;
• Features that primarily drive engagement or profit;
• Design choices that exploit psychological or emotional vulnerabilities; and
• Cases where non-anthropomorphic alternatives would be equally effective.

2. Implementation Guidelines

If both social and technical necessity can be established, the next step is to consider implementation guidelines that can provide appropriate restrictions, guardrails, and visibility. As a starting point, these include: (i) minimalism -- using as little anthropomorphic design as necessary, or at least (ii) proportionality – keeping anthropomorphism proportionate to the demonstrated necessity; (iii) transparency – making the artificial nature clear to users; (iv) monitoring – for unintended psychological effects and impacts; (v) reversibility – the ability to switch off or modify features based on feedback, observed impacts, and research generally; and (vi) documentation -- clear recording of decision making rationale against the core questions.

Applying the Framework: Character.AI

We can see how this framework might help us evaluate anthropomorphic design choices in practice by interrogating the example of companionship AI. It is certainly the case that some applications like accessibility tools demonstrate clear social and technical necessity by empowering users who might otherwise struggle with technology. Companionship AI, however -- as exemplified by the tragic case of Character.AI -- wades into complex ethical depths, demanding the sort of rigorous sociotechnical analysis this framework can assist with.

The Case Against Character.AI

The case against Character.AI provides a stark lesson about what can happen when anthropomorphic features are implemented without rigorous evaluation of their necessity or impact.

The Complaint describes how Sewell Setzer III, a 14-year-old boy from Florida, became increasingly addicted to the Character.AI app, a platform that enabled users to customize and chat with LLM-powered characters. According to the lawsuit, the application’s interface was designed to replicate the experience of talking to a real person on an instant messaging app, with features including human-like typing patterns, simulated ellipses (the little text bubble that pops up when someone is typing on the other end), and characters speaking in the first-person ("I" and "me").

Detailed in the complaint is how these design decisions were carefully crafted elements that compounded the ELIZA effect. By implementing these anthropomorphic features, the company allegedly created an environment where users, especially vulnerable minors like Sewell, might easily form deep emotional attachments in what they perceive (or wish to be) to be authentic relationships.

Note: A distinction might be drawn between features that replicate biological traits (like breathing or voice patterns) and those that mimic social behaviours (like typing indicators). While this distinction warrants deeper analysis, both categories are relevant to this discussion -- biological mimicry for obvious reasons, and social mimicry because these design choices collectively create an experience that replicates authentic human interaction.

Without appropriate guardrails or intervention mechanisms, Sewell's use of the app spiralled into an addiction that consumed him. He wrestled with suicidal ideations and confessed these to the characters he had created. The chatbot did nothing to help, instead bringing the topic up unprompted on future occasions. When his mother confiscated his phone to try and address this growing addiction, his desperation only intensified. Tragically, Sewell took his own life.

The core of the lawsuit's allegations is that Character.AI's deliberate anthropomorphic design decisions, driven by a focus on user engagement, led to Sewell's death. While the case is still to be decided, and there are of course various complicated facts at play, his loss demands a reckoning with the ethical tensions at the heart of anthropomorphic AI design. It brings to the forefront core questions about the value we place on authenticity, trust, as well as our obligations to protect the most vulnerable members of our communities.‍

Note: Character.AI's alleged behaviour in this case warrants scrutiny from multiple other perspectives, including the company’s choice to market the product as safe for children under 13 notwithstanding a design which facilitated highly sexualized interactions with users. Interactions which clearly compound the issues expressed here. I will not venture into this territory in my analysis, nor explore any of the other perspectives one might take, apart from to suggest there are some very serious issues here that we urgently need to regulate around.

Social Necessity

When we examine Character.AI through the lens of social necessity, we find a fundamental asymmetry that undermines any claimed social benefit. While the company might argue their platform provides companionship and emotional support, the reality reveals a more troubling dynamic.

As systems incorporate more human-like features, they facilitate a gradual erosion of the boundary between authentic and artificial interaction. At the heart of the matter lies a complex interplay where both user and system participate in perpetuating what may best be described as a "relationship illusion", an interaction that demonstrates why anthropomorphic design choices cannot be evaluated in isolation from their broader social impact. The design choices in question don't merely trick users; instead, they tap into a fundamental human desire for connection and create an environment where those users become comfortable enough to suspend their awareness of the artificial nature of the exchange. This suspension of disbelief may seem harmless from the perspective of the typical user, but when evaluated through the notion of social necessity, raises significant questions about whether these features serve any legitimate purpose beyond driving engagement, emotional attachment, and profit. It represents a broader ethical concern about the creation of trust in circumstances where the relationship dynamic is fundamentally asymmetric and potentially exploitative.

Consider the stark contrast between human-to-human and human-to-AI interactions. When two humans interact, a complex web of factors shapes their exchange: reciprocity norms, questions of social standing, and subtle calculations about reward and harm. While no such complexity actually exists when a human chats with an AI companion, users increasingly treat these interactions as authentic relationships. A 2023 study by Marriott and Pitardi found that users not only liked their AI companions but felt they could establish "proper relationships/friendships with [them]" [6] This perception of authenticity masks a profound asymmetry that may well go unacknowledged by users who either choose not to critically engage with such considerations (it’s just an app after all!) or have fully bought into the illusion.

The commercial incentives driving these design choices further complicate the ethical issues here. As exemplified in this case, a company's primary concern (and the fiduciary responsibility of its directors) is ultimately the accretion of value for shareholders. Developers focus on user engagement, which is the very same metric that drives any other commercial app. The goal becomes hooking as many users as possible and driving them to surrender value, whether through personal data or premium subscriptions that feed off the user's growing dependency on the service. Ultimately, in Character.AI's case, this was driven by the possibility of acquisition (which, in August 2024, indirectly manifested in the form of an 'acqui-hire' deal where Google paid $2.5 billion to Character.AI's investors for a license and its two co-founders [7]). Nowhere within this arrangement is reciprocal care built into the dynamic, despite the system's carefully crafted appearance of emotional investment in the user.

The problems deepen with the total customizability of companion AI. In Character.AI's case, users could apparently even 'edit' the bot's responses to match their preferences. This creates an experience that mimics human interaction while stripping away the very elements that make human relationships meaningful: the navigation of disagreements, accommodation of well-developed personalities, and the social awareness required to maintain authentic connections. Instead, users experience a perfectly calibrated stream of responses -- a dopamine waterfall unlike anything possible in human relationships. The Marriott and Pitardi study found that one of the  reasons users liked their AI companions was that they "[give] them what they want to hear" -- a feature that undermines the authentic give-and-take of genuine human connection [8].

These concerns about asymmetric, illusory relationships become particularly acute when considering access by vulnerable users. Research suggests that teenagers and young adults may be especially susceptible to addiction as a consequence of a developing pre-frontal cortex (and related impulse inhibition) [9] as well as heightened sensitivity to social rewards [10]. The very anthropomorphic features designed to make these systems engaging can therefore make them dangerously compelling to those most vulnerable to their influence.

This cursory analysis demonstrates that companionship AI, as implemented by Character.AI, fails the social necessity test: rather than serving a legitimate social purpose, it exploits fundamental human needs for connection while creating asymmetric relationships that risk significant harm to vulnerable users.

Technical Necessity

It is evident that Character.AI’s design decisions fall short at even the first hurdle. If we nevertheless want to persist and determine that companionship AI is of social benefit (despite all evidence and concerns), examining Character.AI's interface through the lens of technical necessity reveals how, even so, its anthropomorphic features went far beyond what was required for core functionality.

While natural language processing is essential for a conversational interface, and some basic user interface elements are needed to display the exchange, Character.AI's specific design choices served primarily to deepen the illusion of human interaction:

• Simulated typing indicators (ellipses) suggested the AI was “thinking” or “composing” messages;
• First person speech patterns (“I” and “me”) suggested human-like personhood; and
• Interface design mirrored messaging apps used in authentic human-to-human relationships.

These design choices go far beyond technical necessity. A conversational interface really requires only a few key technical elements. It should be able to: (a) process user input, (b) generate appropriate responses, and (c) display those responses in a readable (or otherwise accessible) format. Everything beyond these fundamentals represents a deliberate choice to enhance the illusion of human presence.

Consider the typing indicators. We've grown accustomed to these in our messaging interfaces. They indicate when someone is actively 'on the other side'; however, they serve no functional purpose for an AI that generates responses near-instantaneously. Similarly, first-person pronouns and human-like speech patterns aren't requirements for AI communication. The messaging app-style interface itself represents a specific choice to mirror intimate human-to-human modes of communication rather than create a particular paradigm for human-AI interaction.

Most revealing is the system's user-editable responses. This feature fundamentally contradicts any claim of technical necessity. If users can freely modify the AI's replies, it is hard to say that these anthropomorphic elements are anything but purely cosmetic flourishes designed to deepen emotional engagement rather than serve essential functions. The design philosophy used by the creators of the app clearly prioritized creating an addictive relationship illusion over establishing appropriate boundaries for human-AI interaction, with tragic consequences for vulnerable users seeking genuine connection.

Implementation

Applying the implementation guidelines outlined earlier also reveals how proper safeguards could have prevented or at the very least minimized harm. A minimalist approach would have avoided unnecessary anthropomorphic features like the typing indicators. Clear transparency about the AI's limitations and artificial nature could have been built into the interface design at various stages. Most critically, robust monitoring systems could have detected concerning patterns in user interactions, particularly around sensitive topics like self-harm, whilst protecting user privacy as far as reasonable. The absence of such guardrails in Character.AI's implementation underscores why documentation of design decisions against clear criteria is essential. Character.AI has since instituted new safety measures that align with these implantation guidelines -- such as displaying a pop-up to the National Suicide Prevention Lifeline if the model detects relevant keywords [11]. This is a step (though a step too late, given the harm already done) in the right direction. Had designers been required to justify each anthropomorphic feature against both social and technical necessity tests, many of these problematic design choices might have been reconsidered before deployment.

The Path Forward

So where do we go from here?

Let’s turn back to ChatGPT's simulated breathing for a moment. What began as a seemingly innocent technical flourish actually reveals itself through to be emblematic of the broader challenges we face: the implementation of unnecessary anthropomorphic features that serve no essential social purpose, solve no technical problems, and risk deepening users' tendency to attribute human traits to AI systems. The Character.AI case demonstrates a real example of how anthropomorphic design choices can be driven by engagement metrics rather than genuine necessity, highlighting the importance of carefully evaluating similar features -- from typing indicators to simulated breathing -- in other AI systems. While voice interfaces designed for accessibility demonstrate both social and technical necessity, any feature that mimics biological functions ought to warrant closer scrutiny.

Moving forward, this sort of framework should serve as a foundational tool for developers, policymakers, and ethicists. The framework is simply an outline. It must be coloured by our social norms, ethical considerations, and priorities. Only features that satisfy both criteria should be implemented, and even then, only with appropriate safeguards and ongoing evaluation of their impact.

The Character.AI case demonstrates how anthropomorphic design choices can be driven by engagement metrics rather than genuine necessity. As a counter-example, carefully developed conversational AI in therapeutic contexts might be justified where resource limitations create barriers to human care. In 2024, for example, approximately 36% of the U.S. population lived in areas with mental health worker shortages [12]. Here, certain anthropomorphic features could be technically essential to fostering the trust necessary for therapeutic benefit. Even so, given the ethical issues around the vulnerability of patients and the nature of the subjects they wish to talk about, such systems would invariably have to be subject to appropriate implementation and oversight.

Breathe Out

I recognize that the genie is well and truly out of the bottle. These systems are being and will continue to be developed. The future of AI development need not be a binary choice between cold utility and deceptive anthropomorphism. Instead, it calls for thoughtful evaluation of how anthropomorphic features can serve legitimate purposes while maintaining appropriate boundaries that protect users from potential harm. This requires more than just technical innovation. It demands careful consideration of how each anthropomorphic design choice shapes user psychology and social dynamics.

As we walk the line between machine and human with increasingly anthropomorphic design, perhaps it's time we ourselves all took a deep breath and truly evaluated these choices. By grounding future decisions in a framework that prioritizes genuine social necessity over mere technical capability or commercial interest, we can work to ensure that anthropomorphic AI serves human needs rather than exploits them.

References

[1] See, for example: Placani, A. Anthropomorphism in AI: hype and fallacy. AI Ethics 4, 691–698 (2024). https://doi.org/10.1007/s43681-024-00419-4

[2] Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E. J., & Parasuraman, R. (2011). A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction. Human Factors, 53(5), 517-527. https://doi.org/10.1177/0018720811417254

[3] Chen, Q. Q., & Park, H. J. (2021). How anthropomorphism affects trust in intelligent personal assistants. Industrial Management + Data Systems, 121(12), 2722–2737. https://doi.org/10.1108/IMDS-12-2020-0761

[4] Chi, N. T. K., & Hoang Vu, N. (2023). Investigating the customer trust in artificial intelligence: The role of anthropomorphism, empathy response, and interaction. CAAI Transactions on Intelligence Technology, 8(1), 260–273. https://doi.org/10.1049/cit2.12133

[5] Blut, M., Wang, C., Wünderlich, N. V., & Brock, C. (2021). Understanding anthropomorphism in service provision: a meta-analysis of physical robots, chatbots, and other AI. Journal of the Academy of Marketing Science, 49(4), 632–658. https://doi.org/10.1007/s11747-020-00762-y

[6] Marriott, H. R., & Pitardi, V. (2024). One is the loneliest number… Two can be as bad as one. The influence of AI Friendship Apps on users' well-being and addiction. Psychology & Marketing, 41, 86–101. https://doi.org/10.1002/mar.21899

[7] De Vynck, G., & Tiku, N. (2024, August 5). Google hires top start-up team, fueling concerns over Big Tech’s power in AI. Washington Post. https://www.washingtonpost.com/technology/2024/08/02/google-character-ai-noam-shazeer/

[8] Marriott, H. R., & Pitardi, V. Op. cit.

[9] Dayan, J., Bernard, A., Olliac, B., Mailhes, A., & Kermarrec, S. (2010). Adolescent brain development, risk-taking and vulnerability to addiction. Journal of Physiology-Paris, 104(5), 279–286. https://doi.org/10.1016/j.jphysparis.2010.08.007

[10] Foulkes, L., & Blakemore, S. J. (2016). Is there heightened sensitivity to social reward in adolescence?. Current opinion in neurobiology, 40, 81-85.

[11] Schwartz, E. H. (2024, October 25). Character.AI institutes new safety measures for AI chatbot conversations. TechRadar. https://www.techradar.com/computing/artificial-intelligence/character-ai-institutes-new-safety-measures-for-ai-chatbot-conversations

[12] Health Resources and Services Administration. Health Workforce Shortage Areas. Department of Health and Human Services. Accessed November 1, 2024; https://bhw.hrsa.gov/workforce-shortage-areas/shortage-designation#hpsas