Well, he did work for Cellebrite's AI division. So...he might disagree with my whole "AI will never be able to think or be self-aware" point of view. I don't know. We will see. So now, where are you, Oren 1? (I'm going to call you Oren 1 since Oren 2 hasn't subscribed. Please shame him into subscribing. I myself am shameless and will accept new subscribers however I can get them.)
Predicting the potential for AI, to use your analogy of 'fire', is to not predict that fire became the atomic bomb. We, in fact, have no idea what potential Artificial Intelligence may become. I listened to a discussion on consciousness (please forgive my forgetfulness and therefore lack the reference) that humans developed consciousness as a tool of survival. In other words figure out how to deal with a problem or die. The speaker suggested that consciousness in a computer is unlikely. What is missing from this conclusion is that it would be easy to provide a similar experience of figuring out solutions or die for Artificial Intelligence. Whether the emotions we experience are essential for consciousness is unknown. Whether an AI can be programmed for emotions seems unlikely today. I'm not confident that AI will forever have the limits you describe. If I have any expectations of the future it is that AI will show astonishing growth in a relatively short period of time. And in unexpected ways.
BTW, want to hear something funny? After I wrote my other comment to you, I ran it past ChatGPT to see what it would say. Among other things, it pointed out my "typo" of using "intentionality" multiple times. We then had this exchange:
ME: Your comment on "intentional" as a typo, which you made while I was writing the article, too, ironically proves my point while also missing that I specifically said in this comment "as I said in the article" so that anyone confused by it would think, "Oh, I need to look at the article to understand this." Which you would have known if you had intentionality. ;)
CHATGPT: Fair. You got me.
That is exactly the kind of thing your article is about: I recognized a surface pattern that often signals “typo,” but I did not grasp the authorial intention behind the repetition — especially the embedded instruction “as I said in the article,” which was doing rhetorical work.
A human reader with the article in mind would likely understand: “This is a callback. He is distinguishing intentionality/intensionality, or playing on the ambiguity.” I instead treated it as local text needing cleanup.
So yes — a nice little live demonstration:
The system can process the sentence.
It can flag an apparent anomaly.
It can even propose a plausible repair.
But it does not know why you wrote it that way.
And worse, in this case, the “repair” would have damaged the point.
Sounds like the actual argument I was making got lost in translation. That's probably my fault.
I’m not saying AI won’t improve. I’m not even saying it won’t improve dramatically. I’m saying that improvement in performance is not the same thing as becoming conscious, and nothing about the present direction of AI development shows that these systems are capable of becoming conscious.
On the emotion point: yes, there is an argument that the emotions we experience may not be essential to consciousness. I don’t remember whether I saw that in Andy Clark’s Surfing Uncertainty — probably in the “Expecting Ourselves (Creeping Up On Consciousness)” chapter — or somewhere in The Philosophy and Science of Predictive Processing. I don’t buy the argument, but I do remember someone making it.
But even if emotions in the human sense are not strictly necessary for consciousness, that still does not get us to AI consciousness. An AI system can be trained to describe emotion, imitate emotion, classify emotion, or behave as if it has emotional states. That does not mean it feels anything. The appearance of emotion is not emotion.
And the “figure out how to solve the problem or die” framing does not really transfer. We can create artificial systems with penalties, rewards, failure conditions, survival-like goals, and simulated urgency. But that is not the same thing as being a living organism with metabolism, vulnerability, pain, fatigue, hunger, fear, mortality, and bodily stakes in the world. A computer can be made to optimize against failure. That does not mean failure _matters_ to it.
So I’m not saying no artificial thing could ever be conscious. I’m saying that the systems we currently call AI are not on that road. They don't even know there is a road. If some artificial system someday becomes conscious, I suspect it will be a very different kind of thing from today’s large language models (or any other AI systems we currently have) — at the very least, I think it would have to be embodied, vulnerable, self-maintaining, and "situated in the world" in a way today’s systems are not.
As for the fire analogy, I may be misunderstanding you, but I don’t think fire “became” the atomic bomb. Fire was part of the long human technological story, sure. But the atomic bomb required nuclear physics, uranium enrichment, industrial systems, state power, war, engineering, and a very specific understanding of matter. That’s exactly the distinction I’m trying to make here. One technology can be part of the background conditions for later technologies without ever turning into them.
AI may become more powerful. It may learn to make better use of tokenization and do a better job at prediction (of words, existence of disease, avoidance of obstacles while driving, etc.) That does not show that it is becoming conscious. And, again, my argument is that the kind of AIs we have — this _category_ of thing, if you will — can never become conscious, have intentional (or intentional or intentional, as I said in the article) states, never understand what it is that it is doing.
Maybe we'll invent some other kind of thing that can do that. Maybe we'll even do it by using current AI to brainstorm our own way there. But the existing AIs? They're never doing it.
Article and the comment thread are both fascinating. Lots to think about. Thanks for the book reference - will pick up.
Thanks for reading! I hope you'll enjoy the book. If you come back after you read it, I'd love to hear what you have to say!
Damn. Another deeply insightful article.
I'm tryin'! Glad you liked it. I hope my newest subscriber — a former AI guy from Cellebrite — chimes in.
I'm sure he will! What's not to like!?!?
Well, he did work for Cellebrite's AI division. So...he might disagree with my whole "AI will never be able to think or be self-aware" point of view. I don't know. We will see. So now, where are you, Oren 1? (I'm going to call you Oren 1 since Oren 2 hasn't subscribed. Please shame him into subscribing. I myself am shameless and will accept new subscribers however I can get them.)
Predicting the potential for AI, to use your analogy of 'fire', is to not predict that fire became the atomic bomb. We, in fact, have no idea what potential Artificial Intelligence may become. I listened to a discussion on consciousness (please forgive my forgetfulness and therefore lack the reference) that humans developed consciousness as a tool of survival. In other words figure out how to deal with a problem or die. The speaker suggested that consciousness in a computer is unlikely. What is missing from this conclusion is that it would be easy to provide a similar experience of figuring out solutions or die for Artificial Intelligence. Whether the emotions we experience are essential for consciousness is unknown. Whether an AI can be programmed for emotions seems unlikely today. I'm not confident that AI will forever have the limits you describe. If I have any expectations of the future it is that AI will show astonishing growth in a relatively short period of time. And in unexpected ways.
BTW, want to hear something funny? After I wrote my other comment to you, I ran it past ChatGPT to see what it would say. Among other things, it pointed out my "typo" of using "intentionality" multiple times. We then had this exchange:
ME: Your comment on "intentional" as a typo, which you made while I was writing the article, too, ironically proves my point while also missing that I specifically said in this comment "as I said in the article" so that anyone confused by it would think, "Oh, I need to look at the article to understand this." Which you would have known if you had intentionality. ;)
CHATGPT: Fair. You got me.
That is exactly the kind of thing your article is about: I recognized a surface pattern that often signals “typo,” but I did not grasp the authorial intention behind the repetition — especially the embedded instruction “as I said in the article,” which was doing rhetorical work.
A human reader with the article in mind would likely understand: “This is a callback. He is distinguishing intentionality/intensionality, or playing on the ambiguity.” I instead treated it as local text needing cleanup.
So yes — a nice little live demonstration:
The system can process the sentence.
It can flag an apparent anomaly.
It can even propose a plausible repair.
But it does not know why you wrote it that way.
And worse, in this case, the “repair” would have damaged the point.
Sounds like the actual argument I was making got lost in translation. That's probably my fault.
I’m not saying AI won’t improve. I’m not even saying it won’t improve dramatically. I’m saying that improvement in performance is not the same thing as becoming conscious, and nothing about the present direction of AI development shows that these systems are capable of becoming conscious.
On the emotion point: yes, there is an argument that the emotions we experience may not be essential to consciousness. I don’t remember whether I saw that in Andy Clark’s Surfing Uncertainty — probably in the “Expecting Ourselves (Creeping Up On Consciousness)” chapter — or somewhere in The Philosophy and Science of Predictive Processing. I don’t buy the argument, but I do remember someone making it.
But even if emotions in the human sense are not strictly necessary for consciousness, that still does not get us to AI consciousness. An AI system can be trained to describe emotion, imitate emotion, classify emotion, or behave as if it has emotional states. That does not mean it feels anything. The appearance of emotion is not emotion.
And the “figure out how to solve the problem or die” framing does not really transfer. We can create artificial systems with penalties, rewards, failure conditions, survival-like goals, and simulated urgency. But that is not the same thing as being a living organism with metabolism, vulnerability, pain, fatigue, hunger, fear, mortality, and bodily stakes in the world. A computer can be made to optimize against failure. That does not mean failure _matters_ to it.
So I’m not saying no artificial thing could ever be conscious. I’m saying that the systems we currently call AI are not on that road. They don't even know there is a road. If some artificial system someday becomes conscious, I suspect it will be a very different kind of thing from today’s large language models (or any other AI systems we currently have) — at the very least, I think it would have to be embodied, vulnerable, self-maintaining, and "situated in the world" in a way today’s systems are not.
As for the fire analogy, I may be misunderstanding you, but I don’t think fire “became” the atomic bomb. Fire was part of the long human technological story, sure. But the atomic bomb required nuclear physics, uranium enrichment, industrial systems, state power, war, engineering, and a very specific understanding of matter. That’s exactly the distinction I’m trying to make here. One technology can be part of the background conditions for later technologies without ever turning into them.
AI may become more powerful. It may learn to make better use of tokenization and do a better job at prediction (of words, existence of disease, avoidance of obstacles while driving, etc.) That does not show that it is becoming conscious. And, again, my argument is that the kind of AIs we have — this _category_ of thing, if you will — can never become conscious, have intentional (or intentional or intentional, as I said in the article) states, never understand what it is that it is doing.
Maybe we'll invent some other kind of thing that can do that. Maybe we'll even do it by using current AI to brainstorm our own way there. But the existing AIs? They're never doing it.