Transhumanism in a Technofeudal Society

1 Blindsight, Transhumanism, and Surveillance Capitalism

I can not stop thinking about Peter Watts novel Blindsight (2006). It keeps me up at night thinking about its implications and what it says about humanity. Let me tell you about it.

The novel's main character Siri is a "synthesist". Siri has the remarkable ability to translate ideas without having to actually understand them himself. Instead, he prompts his interlocutors to speak, and rather than attempting to process the words directly, he observes their "topology", the connections between the speakers language, their body movements and their interactions with their environment. Keaton describes it like understanding the shape of an underwater object by observing the waves it creates on the surface. Others compare it to the "Chinese Room" thought experiment.

I let this concept sit with me for a while. Pondering what it must be like to process information without understanding. Then I realised, he was just describing surveillance capitalism.*

The systems which observe us do not understand us. They don't need to. Targets Customer Loyalty Program does not know what a baby is, but, it knows when a particular customer will need to buy baby stuff, even before the customer. The system doesn't know the customer is pregnant. It doesn't know what pregnant is. It doesn't need to know. It just sees the connections in the customers purchase history, recognises the pattern and acts on it.

1.1 Prediction =/= Understanding

It is interesting the way Watts draws a distinction between predictive ability and understanding. In the Machine Learning world they often talk about the ability for a model to generalise out of distribution and whether or not it learned a causal model for the system it is predicting. That's the jargon for whether or not a model understands cause-and-effect to make its predictions, or whether it is merely noticing patterns. This is kind of like the difference between how a plant seed "knows" to sprout once it gets wet, and a human who understands that plants need rain to grow.

The models which control out feeds, manage this based solely on the patterns we imprint on the world. Without any self-awareness, any consciousness, the models cannot understand the systems they are modelling.

In the novel, Watts shows intelligent agents do not need consciousnesses - or self-awareness - to make goals and act on them within a society. Even empathy can be substituted by modelling conscious beings' incentives as a reward maximisation algorithm. There is one exception to this, which Watts argues in the appendix

"Art might be a bit of an exception. Aesthetics seem to require some level of self-awareness—in fact, the evolution of aethestics might even be what got the whole sentience ball rolling in the first place. When music is so beautiful if makes you shiver, that's the reward circuitry in your limbic system kicking in: the same circuitry that rewards you for fucking an attractive partner or the same gorging on sucrose. It's a hack, in other words; your brain has learned how to get the reward without actually earning it through increased fitness. It feels good, and it fulfills us, and it makes life worth living. But it also turns us inward and distracts us. Those rats back in the sixties, the ones that learned to stimulate their own pleasure centers by pressing a lever: remember them? They pressed those levers with such addictive zeal that they forgot to eat. They starved to death. I've no doubt they died happy, but they died. Without issue."

This is the paradox of our age:

• We built systems that predict our aesthetic preferences with terrifying accuracy
• Lacking any self-reflection, they are missing any intuitive understanding of that beauty they curate
• Yet they've learned to trigger our reward circuitry more reliably than any artist They succeed in controlling us — our decisions, our desires — even without a causal model of human psychology. The machines don't understand why the memes make us laugh. They just know which lever to pull.

Socrates famously argued "the unexamined life is not worth living"? Our lives seem pretty well examined these days. Just not necessarily by ourselves.

1.2 Where the "Self" Ends

Let’s turn back to Blindsight for a second. One of Siri Keeton’s most important revelations was that it was impossible to understand his colleague Robert Cunningham in isolation. Cunningham is a weird character, while Keeton had his brain modified, Cunningham had everything modified. His visual, olfactory, and tactile systems weren’t confined to his body anymore, instead extending into a plethora of electro-mechancial instruments.

"He had sacrificed half of his neocortex for the chance to see x-rays and taste the shapes hiding in cell membranes, he had butchered one body to become a fleeting tenant of many. Pieces of him hid in the sensors and manipulators that lined the [prison]'s cages"

Keeton's mistake was trying to model Cunningham as a closed system which acted on its environment. However, Cunningham didn’t just operate the machinery. He inhabited it. Pieces of him were scattered throughout the vessel, hidden in its sensors and manipulators, and it was only once Siri learned to model Cunningham's entire distributed system, that he figured out how to read him.

It sounds strange until you remember what it feels like to learn how to skate, or ski, or drive. At some point, the board or the car or the bike stops being something you're controlling, and starts being something you're part of. Your brain builds a new body map that includes the wheels under your feet. That’s what Cunningham did with his own consciousness—just with a much bigger machine.

Trans-humanism - the idea of human driven evolution - is usually pitched as a concern of the future. However, the evidence seems to show, we have made significant changes to the nature of humanity already. Involution has been recorded in a number of areas of human life. For example, we know that upon the introduction of writing, humans lose their biological memorization ability. Pre-literate societies we able to memorize songs and poems which lasted for days at a time, now most people can't memorise a phone number. Similarly, we can readily observe that, pre-reading children are far more likely to poses eidetic memory than reading adults. Put simply, we forgot how to remember.

I should note here, this is not a consequence of natural selection acting on our DNA, this time-frame is far too short for that. It is instead a consequence of neuroplasticity. Our brains are incredibly good at adapting to their environment. When a skill is no longer useful, our brain will happily discard it.

The age of computing put this trend into turbo mode. I am aware it is a very kids these day thing to point out, but it is true, people have forgotten how to navigate without GPS. Other studies have shown that easy access to Google has worsened problem solving abilities within the population. Hell, this offloading intelligence to computers is actively encourage in some circles with the concept of a "second brain" - the argument being to offload the job of remembering to the computer, leaving your brain to focus on thinking. (I even wrote a blogpost how I practice this idea)

We have already extended our humanity into silicon. I'm not the first to admit to feeling like a part of me is missing when I don't have my phone or computer with me. It is not the metal I am missing. Instead it is like there is some real part of "me" which exists in the information embedded in all those electrons distributed around the world. In the same way that I struggle to navigate without GPS, I struggle to think without access to the internet, and I cannot remember without access to my notes.

I'm not a particularly online person, but I believe people when they say the "online" version of themselves is more real than the offline. How could it not for people who are limited in their offline abilities? The disabled or otherwise marginalised groups are afforded community bonds onlline much stronger than is possible in a physical world which is shrinking around them. Surely the "you" which represents the majority of your social life can become the "real" "you"?

So I would argue, that "trans-humanism" is not something we need to worry about in the future. It has already started. In fact, it began as soon as we started learning to write.

1.3 Transhumanism in a technofeuadal world

Here is the danger. The tools I use to think, remember, and connect aren't mine; they're leased from corporations. They can be taken away and controlled by forces unaware of and indifferent to our needs and desires. We sped up our trans-human evolution without first checking who owned the means of computation and then left ourselves vulnerable to systems of control which we have no power over.

We leave imprints of ourselves onto our computers with our presence online, and the machines use the patterns of these imprints to understand our motivations, manufacture our desires and control our actions.

The machines of The Matrix, never needed to understand humans to control them. They just needed to model our incentives well enough to convince us we were content

It seems clear to me that the current crop of tech-oligarchs cannot be trusted with their sovereign control over the means of computation. These systems are too entangled with our humanity to be left in the hands of anyone whose incentives are misaligned with our own.

1.4 Information control within a Feudal World

History has shown us what happens when the means of information access is restricted. The darkest days of Catholic history were marked by the Church's insistence that only clergy and academics could be trusted to interpret the Bible. For most of Christian history, the Bible was written ONLY in Latin. The knowledge within was filtered through the words of priests who claimed to understand God's will. The laity received information pre-interpreted and shaped by the Church's incentives.

This enabled the worst behaviors of the Church. Abuse ran rampant because the victims couldn't access the scriptures that condemned such acts. Indulgences were sold as legitimate penance because people couldn't read for themselves that salvation couldn't be bought.

Today, Google doesn't show you the raw internet—it shows you search results filtered through its algorithms. Facebook doesn't show you what your friends actually posted—it shows you what the engagement algorithm thinks will keep you scrolling. TikTok doesn't even pretend that you are in control of what you consume. They know best.

Like the medieval Church, Big Tech argues this gate-keeping is for your own good. The internet is too vast, too chaotic, too full of misinformation for ordinary people to navigate. You need their algorithms to filter out the noise, to protect you from harmful content. Only their trained engineers and data scientists can be trusted to keep you safe.

What will happen to us when the knowledge of the internet becomes locked behind the control of Google, Meta and Amazon? Can we afford to constrain our knowledge to the "Gospel Of Economics" according to Bezos? Or allow our faith in Google to guide us to our destination?

The medieval system only cracked when information was democratized. Martin Luther's translation of the Bible into German, enabled by Gutenberg's printing press, broke the Church's monopoly. Suddenly, people could read for themselves what the text actually said. They discovered that salvation couldn't be bought, that clergy weren't divinely appointed intermediaries, that much of what they'd been told was self-serving interpretation rather than divine truth.

When will we crack the next system? How long can we afford to let them control the tools we use to think?

1.5 Who knew heaven was a place on Earth?

The final anecdote from the story I want to end on, is in the man-made construct of HEAVEN. Billions of the people in Blind-sight willingly abdicate their physical agency to upload themselves into Heaven. It doesn't work for everyone, some plug themselves into this virtual reality, and all they see is pixels.

Is this something we would do? Is there a significant difference between a human uploading themselves into a digital utopia, and a rat starving itself to death because it is so addicted to its heroine it forgot to eat?

Blindsight was published in 2006. That September, Facebook opened to everyone, and the world was never the same.

It wasn't long before his team and his competitors perfected their poison. For all our progress as humans, it turns out that we could be captured by 30 second vertical format videos. No different from "those rats back in the sixties"

2 Further Reading

• Blindsight is actually free. Watts released it into the public domain. You can read it online or download the pdf or ePub. here is the rest of his public domain works
• This essay was also strongly inspired by Mark Rowland's book "The Happiness of Dogs", which discuses philosophy of mind and consciousness extensively.
• My anxieties around the control over the "Means of Computation" lie thanks to Cory Doctorows Book "The Internet Con", and was supplemented by Yannis Varoufakis' book "Technofeudalism"
• The Matrix is a great freaken movie. Go watch that again.
• Shout out to my high-school religion class, thats where I learned about Council of Trent. Just putting that out there while I am listing my influences. You were a great teacher Ms O'Connor
• Go watch that "San Junipero" episode of blackl mirror again. That episode was great.
• I linked to a couple of "a_lillian"'s videos. They are great, I wish I could write as well as them.

*I highly that was his intention, since to be honest, I don't know think the concept had even been formulated by 2006. But that is exactly how modern surveillance capitalism works.