No, "AI" will never become conscious

In the late 20th century, there was a gorilla named Koko who made waves by displaying an apparent ability to talk to humans in sign language. This would've had shocking implications for life as we know it, but the science determined that what she was doing wasn't actually "using language". Through reinforcement learning, she had come to realize that mimicking the hand signals of her human handlers would result in certain outcomes. She knew that making the hand signal for "food" might convince the humans to give her food, but she didn't have the brain structures that would allow her to understand that the hand sign represented food in an abstract way. She could learn word associations, but she couldn't understand the grammar and syntax required for the communication to actually be a "language". We did not discover Gorilla sapiens.

That's not to say it's impossible; whatever mutation made human brains develop language could happen again, and if it does, we should take it seriously. I don't think this will happen the way it appeared with Koko; we'll probably notice an animal communicating in a slightly more complex way, but it might take eons before the communication becomes something we recognize as language. Or we'll discover a heretofore unknown species of animal that has already undergone this transformation. Extremely unlikely, but it could happen.

So what about the opposite? If a conscious animal can't spontaneously learn to use language, can a language-using being develop consciousness? Some people think it's possible; some claim to believe it's already happened, that currently-existing LLMs are already conscious. This is a possibility I'm more skeptical about.

First of all, I contest the idea that LLMs are "using language" at all. They're doing something that looks a lot to us like using language, an act that's fooling a lot of people; but so did Koko. The way LLMs learned language is basically a high-speed, highly specialized version of the way Koko "learned" a "language". We made a computer program simulate an animal that "wants things", then taught it to mimic human language in a way that "gets it what it wants". If you had a virtual Koko whose brain was the same but worked a trillion times faster, she might also be able to "learn language" to a similar degree.

Ted Chiang wrote the definitive debunking of AI consciousness, so I won't rehash his analysis, go read it. (if the full article won't load, try disabling javascript.) But I think continuing the animal comparison will illuminate why "AI consciousness" is not only impossible, but a concept that only makes sense in a religious framework.

We don't know exactly when or how consciousness first appeared, but we do know why, because the "why" of evolution is always the same: consciousness evolved because it helped an animal survive. Some organism mutated some feature resembling a prototypical brain, with some primordial spark of awareness; this (somehow) helped it survive and reproduce in greater numbers than the purely stimulus-based animals around them. Over millions of years, the organism kept outcompeting the ones in the niche without this mutation, and as they kept reproducing, the mutation kept mutating. Brains got bigger and the spark grew along with them.

This is how evolution works in the natural world: (mutations + suitability)*(time) = [differences]. The most useful differences within a particular niche become the dominant ones.

Evolution is a useful metaphor for machine learning, because some concepts from evolution map neatly onto self-improvement algorithms. However, this metaphor has become load-bearing for the AI industry, whose existence depends on making sure we confuse the map for the territory forever. If self-improvement algorithms are sort of like evolution, and computers can "evolve" a zillion times faster than organic life, then AI will probably "evolve" into SUPER AI within the next 5 years, or if not, then definitely the next 10, trust me bro.

The problem is that a metaphor is only a metaphor. In reality, machine learning is different from evolution in important ways.

Have you ever played with the Genetic Algorithm 2D Car Thingy? If not, check it out, it's a fun little toy. Amorphous masses of circles and polygons appear on the left. The circles rotate clockwise. If a circle touches the ground, it will propel the mass of polygons it's attached to forward, until gravity and the shape of the terrain forces it to stop.

In round 1, most of the masses don't get very far. Maybe one or two will get lucky and be generated with two wheels touching the ground. These will go the farthest, but still quickly fall over because their shape is random, not optimized for traversing the hills and valleys in front of them.

Once all the "cars" stop, the universe resets. A new batch of "cars" is generated, using traits from the most successful "cars" in round 1. In this round, the cars that went the furthest reappear with new mutations, and if the mutations make them more car-like, they'll do a little better. This process repeats, and after a few generations, the masses start to resemble cars. Their wheels get closer in size to each other and orientated toward the ground; their bodies become longer and sleeker; their weird random appendages flatten out. Meaningless bundles of shapes were given the same goal we have: to survive as long as possible. And, thanks to a computer simulation, we can watch them evolve to achieve that goal. It's super cool!

Here's a question you might ask: the cars would be able to survive longer if the gravity wasn't so strong, right? But it remains stubbornly stuck at Earth gravity (9.8 m/s) unless the human operator decides to intervene. So why don't the cars learn how to hack the simulation and turn the gravity down? Wouldn't that be a huge evolutionary advantage?

Well no, you probably wouldn't ask that question, because it's very silly. Obviously the cars aren't literally evolving: it's a metaphor. It's a neat simulation, but the only random mutations the cars can undergo are the ones programmed by humans. The programmer could give them the ability to change gravity, but you could let the simulation run for infinity years and they wouldn't develop the ability on their own, I don't care how many cosmic rays flip how many bits.

An LLM is the same thing as a 2D genetic car, except instead of being programmed to evolve cars that go right, they're programmed to evolve chatbots that produce plausible text. The chatbots that produce less plausible text were outcompeted and died off; the ones remaining are very good at producing plausible text, and that's all they can ever be good at.

So-called "agentic AI" is just the same text generators connected to API calls. The text they're trained on is code that makes API calls, and the LLMs generate code that will plausibly produce similar actions. Instead of spewing text into a chat box, they spew it into a command-line interface. They didn't develop the ability to "do" anything, not by any meaningful definition of agency.

To evolve consciousness, an LLM would have to be programmed to evolve consciousness, and to do that, we would need to know what consciousness is and how to program it.

The only reason the cars can evolve is because there are physical principles we understand and can simulate. We know how a wheel works. We understand how gravity affects momentum. We can make a simplified model of the physical world and watch the cars adapt to it.

The only reason an LLM can evolve is because we know what plausible text looks like. We have trillions of examples. Language follows rules of grammar and structure that a computer can emulate. Human evaluators can weed out the models that produce less plausible text.

We don't know what consciousness is. We don't know why we have it. We don't know why some animals have it and others don't. We don't know if it can be quantified. We don't have a model of consciousness or know if one can be created.

Singularitans follow a science-fiction religion that believes, without evidence or even a theory of action, that LLMs will spontaneously mutate and evolve not only consciousness, but God-like super intelligence. They believe this because computers are fast enough and powerful enough to get good at doing tasks that humans program them to do very quickly. Whatever mechanism by which they expect computers to get from A to Z, it's no more grounded in reality than believing God will look down upon a computer from Heaven and, in His infinite compassion, bless it with an eternal soul.

Believing an LLM can become conscious should be just as embarrassing as believing a bunch of circles and polygons can learn how to jailbreak their simulation and hack its gravity. They're so good at evolving into cars, I know they'll be able to figure out javascript. I swear bro, just give them 5 or 10 years.