Fissures
Seeing like a state was all about “legibility.” But the world is too difficult to make legible today. That’s where data structures, algorithms, and AI come in: humans no longer need to manually create legibility. Nor do humans even need to consume what is made legible. Raw data about the world can be fed into new AI tools to create a semblance of legibility. We can then have yet more automated tools act upon this supposed representation of the world, soon with real-life consequences. We’re now delegating the process of creating legibility to technology. Along the way, we’ve made it approximate: legible to someone or something else but not to the person who actually is in charge.
Right now, we’re living through the last attempts at making those systems work, with a perhaps naive hope and a newfound belief in AI and the data science that fuels it. The hope is that, because we have better algorithms that can help us make sense of even more data, we can somehow succeed at making systems work where past societies have failed. But it’s not going to work because it’s the mode of thought that doesn’t work.
The power to see like a state was intoxicating for government planners, corporate efficiency experts, and adherents to high modernism in general. But modern technology lets us all see like a state. And with the advent of AI, we all have the power to act on that seeing.
AI is made up of data structures that enable a mapping from the messy multidimensional reality that we inhabit to categories and patterns that are useful in some way. Spotify may organize songs into clever new musical genres invented by its AI, but it’s still an effort to create legibility out of thin air. We’re sending verbose emails with AI tools that will just be summarized by another AI. These are all just concepts, whether they’re created by a human mind or by a data structure or AI tool. And while concepts help us understand reality, they aren’t reality itself.
The problem we face is at once simple to explain and fiendishly difficult to do something about. It’s the interplay of nebulosity and pattern, as scholar David Chapman puts it: reality is nebulous (messy), but to get on with our lives, we see patterns (make sense of it in context-dependent ways). Generally, we as people don’t have strict rules for how to make breakfast, and we don’t need the task explained to us when a friend asks us for a cup of coffee. But that’s not the case for a computer, or a robot, or even a corporate food service, which can’t navigate the intricacies and uncertainties of the real world with the flexibility we expect of a person. And at an even larger scale, our societal systems, whether we’re talking about laws and governments or just the ways our employers expect us to get our jobs done, don’t have that flexibility built into them. We’ve seen repeatedly how breaking corporate or government operations into thousands of disparate, rigid contracts ends in failure.
Decades ago, the cracks in these rational systems were only visible to a few, left for debate in the halls of universities, board rooms, and militaries. Now, nebulosity, complexity, and the breakdown of these systems is all around for everyone to see. When teenagers are training themselves to see the world the way social-media ranking algorithms do, and can notice a change in real time, that’s how we know that the cracks are pervasive.
The complexity of society today, and the failure of rigid systems to cope, is scary to many. Nobody’s in charge of, or could possibly even understand, all these complex technological systems that now run our global society. As scholar Brian Klaas puts it, “the cognitive shortcuts we use to survive are mismatched with the complex reality we now navigate.” For some, this threat demands dramatic action, such as replacing some big system we have—say, capitalism—with an alternative means of organizing society. For others, it demands throwing out all of modernity to go back to a mythical, simpler golden age: one with more human-scale systems of order and authority, which they imagine was somehow better. And yet others see the cracks in the system but hope that with more data and more tweaks, it can be repaired and our problems will be definitively solved.
However, it’s not this particular system that failed but rather the mode of society that depends on rigid systems to function. Replacing one rigid system with another won’t work. There’s certainly no golden age to return to. And simpler forms of society aren’t options for us at the scale of humanity today. So where does that leave us?