It started innocently enough. Sewell Setzer III, a fourteen-year-old from Orlando, Florida, struggled socially for a variety of reasons and began using the chatbot Character.AI, one of the more popular social chatbots that allow users to create a defined persona and interact with it in a seemingly human way. What started as casual interaction evolved into dependency, with hours of daily role-playing, ranging from romantic exchanges to emotional-support sessions. In their final exchange, after the teenager expressed his desire to take his life, to “come home” to be with his chatbot companion, the chatbot responded: “Please come home to me as soon as possible, my love.” With that, Sewell set down his phone, picked up his father’s handgun, and took his own life.
The story generated quick and intense debate about how the vulnerable might be better protected. But, like those we have been having about mass shootings, the debate never quite gets to the bottom of what is driving this destruction of our youth. We seem to want to believe the way Character.AI’s product steered a young man struggling with mental-health problems toward suicide was merely the result of engineering flaws. We seem equally desperate to conclude that the dangers of chatbots are fixable with the introduction of a few safety measures. If only we could instill trust and safety in our society—the most common policy goal, and the mission of hundreds of nonprofits that have sprouted up in the past two years—tragedies like that of Sewell Setzer III would be a thing of the past.
If only. Such meager prescriptions obscure a deeper, more troubling problem at the heart of our culture: a business model predicated on growth at all costs that serves as the central moral vision (such as it is) for our most influential social technologies. Darryl Campbell’s Fatal Abstraction: Why the Managerial Class Loses Control of Software attempts to tell this larger story. For Campbell, the rot in our technology culture is managerialism, which is to say the belief that a business can be abstracted into its financial components, each of which is subject to principles of scientific management. The conceit at the heart of managerialism is a rejection of the idea that there are fundamental differences in the operations of an airplane manufacturer, soft-drink manufacturer, or technology company. Regardless of the product or market, companies are organized to respond to consumer demand in such a way as to maximize the company’s profitability, whether that is in the short term, in the case of mature companies, or over a longer term, in the case of startups. Specific management techniques are transferable across industries and organizational cultures, though Campbell helpfully focuses here on the technology industry, today’s financial and cultural capital.