In the late 20th century, there was a gorilla named Koko who made waves by displaying an apparent ability to talk to humans in sign language. This would've had shocking implications for life as we know it, but the science determined that what she was doing wasn't actually "using language". Through reinforcement learning, she had come to realize that mimicking the hand signals of her human handlers would result in certain outcomes. She knew that making the hand signal for "food" might convince the humans to give her food, but she didn't have the brain structures that would allow her to understand that the hand sign represented food in an abstract way. She could learn word associations, but she couldn't understand the grammar and syntax required for the communication to actually be a "language". We did not discover Gorilla sapiens.
That's not to say it's impossible; whatever mutation made human brains develop language could happen again, and if it does, we should take it seriously. I don't think this will happen the way it appeared with Koko; we'll probably notice an animal communicating in a slightly more complex way, but it might take eons before the communication becomes something we recognize as language. Or we'll discover a heretofore unknown species of animal that has already undergone this transformation. Extremely unlikely, but it could happen.
So what about the opposite? If a conscious animal can't spontaneously learn to use language, can a language-using being develop consciousness? Some people think it's possible; some claim to believe it's already happened, that currently-existing LLMs are already conscious. This is a possibility I'm more skeptical about.
First off, I dispute the idea that LLMs are "using language" at all. They're doing something that looks, to a casual observer, a lot like using language; but lest we forget, so did Koko. The way LLMs learned language is basically a high-speed, highly specialized version of the way Koko "learned" a "language". We made a computer program simulate an animal that "wants things", then taught it to mimic human language in a way that "gets it what it wants". If you had a virtual Koko whose brain was the same but worked a trillion times faster, she might also be able to "learn language" to a similar degree.
Ted Chiang wrote the definitive debunking of AI consciousness, so I won't rehash his analysis, go read it. (if the full article won't load, try disabling javascript.) But I think continuing the animal comparison will illuminate why "AI consciousness" is not only impossible, but a concept that only makes sense in a religious framework.
We don't know exactly when or how consciousness first appeared, but we do know why, because the "why" of evolution is always the same: consciousness evolved because it helped an animal survive. Some organism mutated some feature resembling a prototypical brain [edit: this isn't entirely correct, see this comment from TNSBWM] with some primordial spark of awareness; this (somehow) helped it survive and reproduce in greater numbers than the purely stimulus-based animals around them. Over millions of years, the organism kept outcompeting the ones in the niche without this mutation, and as they kept reproducing, the mutation kept mutating. Brains got bigger and the spark grew along with them.
This is how evolution works in the natural world: (mutations + suitability)*(time) = [differences]. The most useful differences within a particular niche become the dominant ones.
Evolution is a useful metaphor for machine learning, because some concepts from evolution map neatly onto self-improvement algorithms. However, this metaphor has become load-bearing for the AI industry, whose existence depends on making sure we confuse the map for the territory forever. If self-improvement algorithms are sort of like evolution, and computers can "evolve" a zillion times faster than organic life, then AI will probably "evolve" into SUPER AI within the next 5 years, or if not, then definitely the next 10, trust me bro.
The problem is that a metaphor is only a metaphor. In reality, machine learning is different from evolution in important ways.
Have you ever played with the Genetic Algorithm 2D Car Thingy? If not, check it out, it's a fun little toy. Amorphous masses of circles and polygons appear on the left. The circles rotate clockwise. If a circle touches the ground, it will propel the mass of polygons it's attached to forward, until gravity and the shape of the terrain forces it to stop.
In round 1, most of the masses don't get very far. Maybe one or two will get lucky and be generated with two wheels touching the ground. These will go the farthest, but still quickly fall over because their shape is random, not optimized for traversing the hills and valleys in front of them.
Once all the "cars" stop, the universe resets. A new batch of "cars" is generated, using traits from the most successful "cars" in round 1. In this round, the cars that went the furthest reappear with new mutations, and if the mutations make them more car-like, they'll do a little better. This process repeats, and after a few generations, the masses start to resemble cars.
Their wheels get more oriented towards the ground; their bodies become longer and sleeker; their weird random appendages shrink and flatten out. Meaningless bundles of shapes were given the same goal we have: to survive as long as possible. And, thanks to a computer simulation, we can watch them evolve to achieve that goal. It's super cool!
Here's a question you might ask: the cars would be able to survive longer if the gravity wasn't so strong, right? But the simulation is stuck at Earth gravity (9.8 m/s) unless the human operator decides to intervene. So why don't the cars learn how to hack the simulation and turn the gravity down? Wouldn't that be a huge evolutionary advantage?
Well no, you probably wouldn't ask that question, because it's very silly. Obviously the cars aren't literally evolving: it's a metaphor. It's a neat simulation, but the only random mutations the cars can undergo are the ones programmed by humans. The programmer could give them the ability to change gravity, but you could let the simulation run for infinity years and they wouldn't develop the ability on their own, I don't care how many cosmic rays flip how many bits.
An LLM is the same thing as a 2D genetic car, except instead of being programmed to evolve cars that go right, they're programmed to evolve chatbots that produce plausible text. The chatbots that produce less plausible text were outcompeted and died off; the ones remaining are very good at producing plausible text, and that's all they can ever be good at.
So-called "agentic AI" is just the same text generators connected to API calls. The text they're trained on is code that makes API calls, and the LLMs generate code that will plausibly produce similar actions. Instead of spewing text into a chat box, they spew it into a command-line interface. They didn't develop the ability to "do" anything, not by any meaningful definition of agency.
To evolve consciousness, an LLM would have to be programmed to evolve consciousness, and to do that, we would need to know what consciousness is and how to program it.
The only reason the cars can "evolve" is because there are physical principles we understand and can simulate. We know how a wheel works. We understand how gravity affects momentum. We can make a simplified model of the physical world and watch the cars adapt to it.
The only reason an LLM can "evolve" is because we know what plausible text looks like. We have trillions of examples. Language follows rules of grammar and structure that a computer can emulate. Human evaluators can weed out the models that produce less plausible text.
We don't know, in a material sense, what consciousness is. We don't know why we have it. We don't know why some animals have it and others don't. We don't know if it can be quantified. We don't have a model of consciousness or know if one can be created. All we can know about consciousness is what we experience and infer. This is the hard problem of consciousness.
Singularitans follow a science-fiction religion that believes, without evidence or even a theory of action, that LLMs will spontaneously mutate and evolve not only consciousness, but God-like super intelligence. They believe this because computers are fast enough and powerful enough to get good at doing tasks that humans program them to do very quickly. Whatever mechanism by which they expect computers to get from A to Z, it's no more grounded in reality than believing God will look down upon a computer from Heaven and, in His infinite compassion, bless it with an eternal soul.
Believing an LLM can become conscious should be just as embarrassing1 as believing a bunch of circles and polygons can learn how to jailbreak their simulation and hack the gravity variable. They're so good at evolving into cars, I know they'll be able to figure out javascript. I swear bro, just give them 5 or 10 years.
-
But that said, if you were on team "AI is/will be conscious" and you find this post convincing enough to change your mind, I'm not going to make fun of you. We all believe embarrassing things at various points in our lives, and ideally grow out of it. I don't think mocking sincere personal growth is cool or helpful. ↩
If regression to the mean could find consciousness, then you wouldn't need llms to find it?
Llms sometimes, ahh, ape the shape of consciousness, or at least the way very verbal people are conscious. I assume. Maybe. The BSer class is most impressed by them. If llm is told to take notes and use the notes to work through problems, they become a bit better at problem solving. It's like improving someone's working memory so they can make smarter decisions. Except that it's not like human memory at all. I recently listened to a science friday podcast about AIs making us out of practice as we offload skills to it I think that's affecting my argument there. Even stuff like image recognition which is the older more responsible relative of the automatic uncredited art collage machines.
What I think is is llms are a new way for computers to be dumb very, very, fast. And right now I don't know about any more prosocial use for it than doing thankless secretary work like meeting notes.
This is well-written, and I believe that your conclusion is fundamentally correct.
But it's a little confusing to both say that consciousness cannot occur in a machine unless the machine's programmed to evolve consciousness... and that we don't know what consciousness is.
You can't have it both ways! If we don't know what it is or where it comes from... then you can't rule out the possibility that it can occur spontaneously in something that "acts like" it has it. We draw a line between things that "are" and "are not" conscious (which, personally, I find pretty arbitrary but there are others who prefer a more-concrete definition), but the fuzzier that line is, the harder it is to define not only what it means to cross it... but whether or not a particular thing has crossed it at all!
Don't get me wrong: I'm not arguing that LLMs are on the path to consciousness. LLMs aren't even on the path to AGI, which would (if we developed such a thing, if it's even possible) be immeasurably more-sophisticated than LLMs are and could definitely cause legitimate debate about the nature of consciousness! But LLMs are a parlour trick: a whole lot of (admittedly complex!) statistics that mashes together sentences based on what it's seen before: data-in, data-out.
We don't know if being a simpler, data-in, data-out kind of organism was part of OUR path to consciousness. But if it was, it's far-removed from anything an LLM's shown itself even remotely capable of so far!
So yeah: I agree with you. Except for the pair of arguments that (a) a thing is not-conscious and (b) we don't know what consciousness is, which seem self-contradictory. I suspect that we - or you at least - DO have some concept of what consciousness is (to satisfy argument (a))... it's just hard to define (to satisfy argument (b)).
And yeah: the machines don't have it yet, whatever it is.
@Dan Q: Thank you. Consciousness emerged via the same process language did: biological changes that occurred over eons. It's not going to happen spontaneously any more than Koko was going to spontaneously develop the ability to use language because she "acted like" she was using language.
We don't know exactly what sapience is, or why we have it and gorillas don't, but we can infer that it requires brain structures that a gorilla doesn't have.
We don't know exactly what consciousness is, or why we have it and computers don't, but we can infer that it requires brain structures that a computer doesn't have.
We won't accidentally make a computer conscious because we're trying to teach it to think, any more than we were going to accidentally make a gorilla sapient because we tried to teach her to sign.
More succinctly: that consciousness can be an emergent property of simulated language is an extraordinary claim. Like all such claims, we shouldn't believe it without extraordinary evidence.
"and to do that, we would need to know what consciousness is and how to program it."
By that same token (no pun intended), you need to know what it is to claim that we cannot program consciousness. Everything points at the brain as the producer of consciousness via some kind of information processing being done by neurons. This is evident by studying patients that had brain damage in different areas and observing how different "features" of cognition and subsequently consciousness are turned off and on.
If we can conclude consciousness is some kind of information processing, the next question is how did it evolve, which would lead directly to what the algorithm and architecture is. This could then be implemented in any substrate that can support that kind of information processing, i.e. it could be some massively parallel algorithm, even more than current LLMs with GPUs.
I completely agree LLMs are nowhere near though, and they're just pattern matchers that give a strong illusion because humans have theory of mind and project agency unto it. It becomes clear after interacting with one enough that it does not reciprocate this theory of mind and the illusion breaks.
@proc0: I don't think it's a given that we could design a machine architecture for consciousness by looking at different kinds of brain damage, because two patients with differently-damaged brains have almost everything else in common: they both have optical nerves and eardrums and olfactory systems that feed the brain sensory information in roughly the same way. They both have little hairs that stand up when they sense danger, and circulatory systems that pump extra blood when the brain experiences certain emotions. They both have endocrine systems that pump all sorts of fun hormones around the body. They both have hundreds of neurotransmitters and neuroactive peptides acting on the brain in ways we struggle to even identify. Even if we can eventually make a computer that works like a brain, I think it's naive to think that'll be meaningful at all without all the other stuff.
I think the glib thing to say is LLMs are Goodheart's law applied to the Turing Test.
OK it's possible to approach this from another angle. What is consciousness? We call humans conscious, and that is justification of many rights we have that animals don't have (or those that like to be logically consistent do). Humans are all considered conscious, though kids have less rights. Some developmentally disabled people get less rights too, in conservatorships. And elderly disabled. Also Brittany Spears. The system is prone to abuse and indignity. This is why we have few levels of rights/consciousness legally, simpler rules mean fewer loopholes. Even if people are conscious by degrees, that doesn't mean we deserve dignity by degrees.
We also try to treat humans better than animals even when humans aren't conscious. Someone with brain damage that will never wake up gets a more dignified death than most chicken dinners.
@Lisa: I feel the difference in human/animal treatment comes down to sapience more than sentience, and I think it's justified to treat sapient as higher priority than not-sapient in some regards. But yeah the fact that non-sapient beings are still conscious and can experience suffering is a reckoning I expect will be unpleasant for a lot of us if we can fix the planet-burning, civilization-collapsing problems in time to reckon it. I don't like the idea of doling out rights based on arbitrary thresholds of sapience, or consciousness. Personally I'm not conscious for at least 6-7 hours every night, so if someone goes cherry-picking I might be in trouble
what I was trying to get at, but got kinda side tracked, is that consciousness is not one thing, but varied, many. This is I think why when you mention what counts as conscious you always invite arguments, we're all kinda differently aware of ourselves.
The evolutionary advantage of consciousness is when you have a copy of the universe you can run simulations in, you can get more from the universe. If you're a little better able to simulate your competition than they can simulate you, then you get more than them. Thus there's evolutionary pressure to get smarter, and when the smartness level rises, there's pressure to get even smarter... this is why we spend so many calories on brains. There mystery solved all I had to do was erase so much detail that you can't predict much with the theory. Does fit with how much theory of mind projection we have though.
A brain cell isn't smart. A brain center out of brain context isn't smart. There may already be a way to combine all the ways computers are dumb fast into consciousness, but we have a harder time figuring that out than figuring out hands. (It's so amazing that it's hard to engineer robotic hands and it's also hard for AIs to draw them! What incredible hand knowledge do we not know we know because they run instinct not consciousness? claps ) There's lots of theories of course, probably more theories than there are minds. The testable ones treat us more like black boxes, because it's damn horrifying to think of breaking up a brain into machines one could replace individually. And LLMs are only good at things that can be tested to be trained...
LLMs could be a component of consciousness. Adding more compute to LLMs will not produce sonsciousness.
Correct. The idea that agency would come bolted on atop sufficient intelligence is just silly. The same architecture at larger scale isn’t suddenly going to gain agency.
One correction. No prototypical brain was needed for awareness. Indeed, what you’re describing was clearly done by Cyanobacteria when they invented their circadian clock. A remarkable, and ancient system, it allowed them to move from merely reacting to sunlight to anticipating it. In the lab, it has been shown conclusively that having an internal anticipatory rhythm that matches the external solar cycle literally lets Cyanobacteria survive better. Critically, if you take the cynaobacteria with clock proteins from when they first evolved them, when the earths rotation period was 19 hours, and put them in 24 hour lab cycles, they get outcompeted by their modern counterparts.
But switch the lab to a 19 hour cycle, and the ancient Cyanobacteria outcompete modern ones!
This right here is the core of biological awareness. As multicellular beings evolved, we see complex clocks evolve to help the cells negotiate their shared timing, so that by the time you reach mammals, each cell is still an autonomous clock, but entrains itself to a master oscillator deep in the brain.
Current AI, of course, lacks any internal sense of time. Ask an agent to do a task, and it won’t know how long it took it, nor can it give good estimates. Two agents doing similar tasks can take wildly different times, and if one depends on the other, it’ll wait stalled if the other stalls.
This is what biology solved billions of years ago, from the bottom up. Basal awareness is already internalized and entrainable to external cycles, well before you got any cell that can be called a neuron.
The neuron abstraction used in LLMs is simply from a time when we don’t know each neuron in our brain has an internal clock that, among other things, gates and modulates synaptic activity. These are continuous systems, the “basal” cognition of temporal rhythms, and the more “advanced” linguistic or other kinds of cognition that the same neurons enable.
And so, larger models won’t suddenly get agency. They won’t self organize into behavior where they can oscillate in attunement with the environment. Nor will they magically find a software solution to the Von Neumann bottleneck.
@TheNotSoBlindWatchmaker: Thanks for the additional context!
@lisa
That’s incorrect. A brain cell out of the brain isn’t smart in the same way. And yes, in complex multicellular beings like us, each single cell has lost the competency to survive just anywhere.
However, take a brain cell and put it in a dish where it gets the nutrients it needs (labs do this all the time). These cells are damn smart. They can find chemical gradients. They can be given electrical input and will, depending on the type of neuron they are, fire in specific ways.
Indeed, take a cluster of such neurons and other brain cells called an organoid, a mini brain. These are quite smart. In fact, there’s an Australian company that’s found a way to use such organoids to do computing, and they do it for a lot less electricity than modern LLMs.
I’d recommend you give Michael Levin’s work a read. He’s done a lot of work to show that cognition goes all the way down to single cells. Not just neurons but single bacteria are incredibly smart. Certainly, all living systems show intelligence that no current LLM is able to match, in many contexts.
By that explanation, people could not evolve consciousness, because we don't understand it, but we have. From that point the discussion becomes theological - god gave it to us, which will only appeal to a subset of the audience. To the scientifically minded readers - an alternative is needed.
I'd personally go with "we don't understand consciousness" but that there are a number of coincident factors and chance that make it possible. Then the discussion becomes AI can create consciousness, but it could take longer than our lifetime to do it. On the other hand, AI "thinks" faster than we do, so we might condense a million years of evolution into a thousand years or so. As things improve, the time needed becomes less and less. I don't know how long it is - but I think it is possible. Just from one instant to the next unlikely.
@Paul: I address many of the points you bring up in the post you're responding to.
So is the reason we currently regard AI as non-conscious that we still can’t clearly define the boundaries of consciousness, let alone measure it by any objective standard?
@Aikilan: No, the heuristics we have are enough to definitively induce that AI isn't conscious. Our understanding of consciousness doesn't need to be complete to make that inference. See the Ted Chiang article I linked.