Tech Bros Stole My Friend: Faux-Social LLMs Meet the Market
Sometimes even I, as an ardent Luddite, wish AI companies would be slower in driving their own failure.
Recently I've been working on a paper relating to faux-social LLMs - those which mimic relationships, therapy, care etc. In part of the paper I suggest viewing those 'relationships' built with LLMs as toxic or insecure in part because whatever a user can experience from them (which isn't a lot of things) that intimacy is always temporary. Behind it stands a tech company which, for whatever reasons, can issue an update or change the ToS in a way that completely breaks the nascent (if one sided) connection. That was only one point among many and a very brief summary of it at that but apparently Character.AI have gone all out on proving me right.
Their latest update - PipSqueak 2 - is proving massively unpopular as users lament the passing of the bots they knew and, in some cases, professed to love. Along with ads, age verification and usage limits it's part of a raft of changes which highlight the mounting profitability crisis around LLMs in general and, I'm guessing, faux-social LLMs in particular. Designed to work on long term engagement and user capture the likes of Character.AI and Replika are, I suspect, haemorrhaging money by offering generous free accounts as a hook for subscriptions. Something that even the biggest companies with (in theory) less compulsive models are struggling to maintain amidst the endless demand for investment and the minimal potentials for profit.
For users it's a hard blow to take. Faux-social LLMs are great at capturing attention and attachment both and while obviously not everyone is going to be investing huge emotional energy in their bot of choice some people certainly are and whatever negative views I may have about the models I do retain sympathy for those caught up in them. Unfortunately there isn't a huge amount of research on user demographics (that I know of) but certainly you do often find stories of those in isolation or struggling with feelings of alienation being particularly drawn to the ever-affectionate, always on responses of the LLM. Whilst those responses may never lead to anything healthier than a temporary fix for loneliness (but never isolation) there's still no joy in seeing people confronted with the predictable loss of perceived intimacy on the financial whims of tech companies scrabbling to justify their own existence. Although it is a story we're likely to see repeated ad nauseum in the coming months and years.
One interesting wrinkle in the story is the varied reactions you can see coming through. Taking a not at all scientific glance at two popular subreddits (CharacterAI and MyBoyFriendIsAI) you can see something of an emerging demand on users who engage with these platforms. While plenty walk away, soured on the idea of faux-social LLMs in general ('guess I'm not addicted any more!') others enter into ever deeper explorations of how to retain the perceived intimacy they've come to rely on. At the moment the CharacterAI sub is loaded with outrage and disappointment while the MyBoyFriendIsAI seems largely quiet on the subject. The reason? Well, looking back to when ChatGPT-5 was released the outrage and sense of loss was certainly similar but for the smaller and more focused community there's been an ongoing process of adjustment since then. Models have been compared, discussions had on which platform offers and will continue to provide their ideal of 'intimacy', users have migrated whole 'relationships' in line with the impacts of model releases.
This kind of progression and crowd sourcing of wisdom on where the best experience can be had is, to be fair, a reflection of genuine community building but it also represents an accumulation of insecurities. Going back to the idea of toxic relationships one key aspect of them is insecurity, the unpredictability of affection be it from an unreliable human or, in this case, a tech company lurking behind the façade of a humanised LLM. The agile repositioning of human affection from one platform to another, always vulnerable to new updates, guardrails and design choices, perpetually racing ahead of a wave of desperate monetisation and (no doubt upcoming) financial collapse feels like a more and more doomed grasp for a sense of connection and socialisation which was never really there to start with but which seems likely to only diminish ever more.
- Dylan
--
Recommended reading/listening on faux-Social LLMs...
Your Undivided Attention Podcast - Attachment Hacking and the Rise of AI Psychosis
Attensity!: A Manifesto of the Attention Liberation Movement - The Friends of Attention