Transcript: A Future with Digital Minds? Expert Estimates and Societal Response
Audio and video for this episode can be found here.
Will Millership 0:00
Welcome to this episode of Exploring machine consciousness. Today, we're delighted to be joined by Lucius caviola to discuss his research on public perceptions about the possibility and impact of digital minds. Lucius recently joined the Leverhulme center for the future of intelligence at the University of Cambridge as an assistant professor in the social science of AI. Before that, he worked on related questions at the Global priorities Institute at the University of Oxford. His work applies experimental psychology to questions about perceptions of AI consciousness and moral status. As part of his current research, he surveys expert and non expert opinions on digital minds potential take off scenarios and societal responses to their emergence. His work examines the psychological and social factors that may shape how humanity responds to the emergence of digital minds. I've had the pleasure of talking with Lucius on a couple of occasions, and I'm really excited to continue our discussions today, Lucius, it's wonderful to have you on welcome to the podcast.
Lucius Caviola 0:56
Thank you so much for having me. I'm really looking forward to having an interesting discussion with
Calum Chace 1:01
you. Thanks for joining us, Lucius.
Will Millership 1:04
So I'm going to start off talking about your recent expert survey work. So in your recent expert forecast with Brad Saad, you study expert views on digital minds, you found that experts seem pretty certain that digital minds could be created in principle, with a probability of 90% and also gave a very high probability of actually happening, 73% given the uncertainty in the field. Did you find these results particularly surprising?
Lucius Caviola 1:31
Yeah, they are indeed, really quite high estimates. And I think they are a bit surprising. They surprised me a little bit, but I think most other people would be even much more surprised than we are. And in fact, I have evidence for this. We can maybe talk more about this later, when we talk more about research on public attitudes and public beliefs, but just to foreshadow quickly, so if you just look at the views of ordinary people who have never thought about AI consciousness, most of them are very skeptical that AI consciousness is possible. So only about 23% of them think that AI consciousness is possible in principle. So for most normal people who've never thought about AI consciousness, these would be really high estimates and really quite surprising, perhaps I can quickly say a bit more about the survey. So in the survey that I conducted with Brad Saad, we surveyed experts on digital minds, and the definition of digital minds in the survey was the following. It was defined as computer systems that have the capacity for subjective experience or phenomenal consciousness, and this includes AI systems, like machine learning based systems. Could also include brain simulations, computer brain simulations, and I think it's important to say that. So this is a somewhat narrow definition of digital minds. You know, in the literature, sometimes digital minds are defined more broadly to include any kind of computer system that has moral patienthood, even if it's not conscious, but perhaps because it has robust agency or sophisticated, coherent preferences. But in the context of the survey, we asked all participants to adopt this narrow definition, just to make sure that everybody has the same assumptions that it's easier to interpret. And there's another important assumption that we all ask participants to adopt, which is that we ask them to only consider digital minds that have a welfare capacity that is equal or higher than those of a typical human. And we did that to avoid having some participants consider very sort of weak forms of consciousness, you know, just like a short flicker, perhaps, you know, during the training process. So we wanted to avoid that and only sort of consider some more sophisticated forms of consciousness in AI.
Calum Chace 3:55
That's a very important constraint, because it was interesting that you do not limit the digital minds to superintelligence. It could be—as long as it's conscious—well before superintelligence. But you're also saying that it has to have a human-level capability for welfare, which suggests it is going to be a superintelligence.
Lucius Caviola 4:16
It's a good question. Not necessarily. You could imagine a system that is not particularly cognitively sophisticated but still has a strong capacity for subjective experience. This is not something that we asked participants to assume, but it is a possibility that participants assumed that. Just to say a bit more about the survey sample: we had 67 experts. Unfortunately, there are not that many experts yet globally who explicitly research digital minds in a professional capacity. So it was a mixed sample. Some of the participants were AI consciousness and digital minds experts. Some were philosophers of mind and consciousness experts who have thought a bit about AI consciousness. We also had some AI experts who have thought a bit about digital minds, and some super forecasters. The goal is to replicate the survey in the future and to look at trends. In the future, there might also be a larger population of experts to draw on.
Will Millership 5:26
I would consider you an expert in digital minds. What is your estimate on both of those questions? Firstly, the probability of digital minds being created, and secondly, of it actually happening?
Lucius Caviola 5:42
Since I'm not a consciousness expert myself, but a social scientist, I usually try to evade this question and instead defer to my survey results. I think my own estimates are perhaps slightly more conservative, a bit lower, but I do find these estimates—which are the median estimates of our sample—reasonable. The motivation behind the survey was to get a better understanding of what experts think. As far as Brad and I know, there's almost no research on this so far trying to predict future scenarios involving digital minds. Our hope was for the survey to set the stage, or to initiate this as a new field, to better understand when digital minds could be created, how they could be created, where, and by whom; what kind of systems are more likely to be conscious and which ones less likely; and also what the societal...
Will Millership 6:54
Just to come back to that "at least comparable to human" definition of digital minds: I think that makes the 90% probability even more staggering. My assumption—which I know is not everybody's—is that there will be some very basic level of consciousness or sentience in machines reached first, before you start getting these higher levels of human potential. So, do you think that probability estimate would rise if you brought down that standard of phenomenal consciousness?
Lucius Caviola 7:31
I'm pretty sure that it would. It would rise. There's one methodological point here that I want to mention, which is that there is a potential for a selection bias with our study, and we emphasize this a lot in our report. This was a very long survey with lots of different questions. It lasted over an hour, and we don't have time now to go through all of them. But for those of you who are interested, I encourage you to check out our report on the website, digitalminds.report. There, you'll see all the graphs and also the many different considerations that participants mentioned. We asked them for quantitative estimates—numerical estimates, for example, probability estimates—but then we also asked them to explain the considerations behind their estimates. We collected all of these considerations and organized them in this report.
To go back to the methodological issue: because there are not that many, we selected participants who had some expertise relevant to digital minds. Plausibly, you can assume that perhaps those researchers who are researching digital minds and AI consciousness are the people who think that they're particularly likely, and who think that this is a particularly important issue to study. So, this is an important thing to keep in mind: that perhaps the estimates that we have here are somewhat inflated. But it's not at all obvious. It's also possible that the causality goes the other way around: that once you start to engage more with these ideas, you start to believe more that it's likely. We don't really know the answer to this question, but I can tell you that in another project with other researchers—Noemi Drexler from GovAI, Jeff Sebo, David Chalmers, and others—we conducted another forecasting survey targeting AI researchers. The answers to these questions were actually very similar to the answers we had here, and the way we recruited those participants was very different, so there was no selection effect. We found AI experts by looking at publications at NeurIPS and other AI venues, and then we contacted them.
This provides some support for the view that there's not necessarily a selection bias here. Still, I'm not sure we should take these quantitative estimates at face value. It was an extremely speculative survey—probably the most speculative forecasting survey that has ever been conducted in the history of humanity. Experts expressed a lot of uncertainty; sometimes they were even very reluctant to give an answer. We kind of forced them and said, "Go with your best guess, median estimate, and then explain your considerations." So even though the numerical estimates are not perhaps very precise, I still think that they provide some useful information, especially since there were clear trends.
Calum Chace 10:57
I'm confident that this is very far from the most speculative survey that's ever been done in human history. You mentioned timelines. You didn't impose a timeline in this question, did you? You didn't ask them, "Will there be conscious digital minds by 2040?
Lucius Caviola 11:13
We did. This was a separate question. We had one question about whether it's possible in principle, whether you think it will happen at some point—could be also in a million years. Then we had one question about timelines, and I can quickly describe the results. In one question, we asked in which year do you believe that the first digital mind will be created. It would be good to be able to show you the graph. If listeners want to take a look, go to digitalminds.report, but I'll describe the results. Most experts were pretty confident that in this year, 2025, zero—no digital mind will be created, according to our definition. But then each decade, the median probability estimate substantially increased, such that by mid-century, the median estimate was 50%. I think that's a memorable finding: 50% probability for the first digital mind by 2050—that's the median estimate. And by the end of the century, the estimate was 65% .
Calum Chace 12:21
Interesting. Trying to ransack my memory, but comparing it with surveys about people's estimates of the arrival of superintelligence—people like Katya Grace, who talks about human-level machine intelligence rather than superintelligence—it sounds like people think that consciousness... I mean, I know you shouldn't mix the two concepts up, but it sounds like people think that consciousness will arrive slightly after superintelligence?
Lucius Caviola 12:47
Absolutely. In fact, that was a separate question where we asked what's the likelihood that the first digital minds will be created before AGI, and the median estimate was a bit below 30%. In fact, the modal response—the most popular option—was that it is extremely unlikely. So there was a clear tendency that participants thought it would happen after AGI. I'm a bit unsure how to interpret this. I think our definition of AGI might not have been precise enough, and people may interpret it differently. But participants seem to think that we will first have AGI and then digital minds, according to the definition of digital minds that we had.
Will Millership 13:31
Brilliant. And just a quick plug for that website, digitalminds.report. It's a really great resource. All the graphs are on there, so I'm going to share the links for our listeners in the description below. Lucius, sorry, you had something to say?
Lucius Caviola 13:45
In addition to the timelines, I think there's another finding which would be good to mention now, which is the speed at which we would create more digital minds. That was a separate question, and here we asked how fast participants believed that we would create more digital minds after the first one has been created. The upshot here is that most participants expect it to go really fast. The median estimate is that 10 years after the first digital mind has been created, we will have created so many more digital minds that their total Welfare Capacity will be as big or bigger than that of all 8 billion humans.
I think the upshot—or two key highlights—would be that according to the world-leading experts on these questions right now, digital minds are something that could be created within the next few decades, quite likely. And that it could very quickly become one of the most important moral issues, assuming certain ethical assumptions, such as not discriminating based on substrate, thinking that digital minds matter morally, and being roughly scope-sensitive. These are some of the key highlights from the survey.
Calum Chace 15:14
And Lucius, did you capture anything about the beliefs of the participants regarding theories of consciousness? For instance, I'd be intrigued to know whether there's a correlation between, say, being a computational functionalist and thinking that machines will be conscious before long, versus Anil Seth and his biological approach. Obviously, he doesn't think machine consciousness is terribly likely, certainly not anytime soon. And I wonder whether people who are Global Workspace Theory adherents or IIT adherents—whether there's any correlation between theory and likelihood of machine consciousness?
Lucius Caviola 15:56
It's a really good question. This is something we didn't look at in very fine-grained detail, but I agree that this is something that follow-up research should do. I think Derek Schiller and colleagues are working on such types of surveys now. There were some questions, though, that are getting a bit at providing some answers to your question.
We asked which types of digital minds, or which types of systems, participants consider to be more likely to be conscious in principle, and also more likely to become the first digital mind—assuming the first one will be created. We distinguished between machine learning-based systems and computer simulations. I think the third category was other types of systems, including neuromorphic systems. The upshot from these two questions was that participants think that machine learning-based systems are less likely, in principle, to be real digital minds—that is, to be conscious—versus computer simulations, for example. But they still think that it's more likely that the first digital mind, assuming it will be created, will be a machine learning-based system. I hope that makes sense.
Calum Chace 17:14
That is intuitive. That does intuitively make sense. Uploading a human mind is a really big project, a massive, massive engineering project, and it's going to take a while. But of course, if you could do it, it's pretty much a slam dunk. It's pretty likely. There would be some people who disagree with this, but if you could upload a human mind, it's pretty likely to be conscious. I can see the logic behind that when you talk about digital minds. Presumably, you're excluding organoids from this survey.
Lucius Caviola 17:46
Good question. I don't think that this is something that participants considered. In the studies, in the questions...
Will Millership 17:54
When I read the description, I got the impression that organoids weren't included.
Lucius Caviola 18:00
What are your views?
Calum Chace 18:02
I broadly agree with what I take to be the modal opinion, although I suppose in one area I differ. It seems to me that it's going to be hard for a machine to become superintelligent and not be conscious—not impossible. I can see it's feasible or philosophically possible, but I suspect by the time we get superintelligence, it will be conscious. Not least because almost by definition—which is cheating a bit... if you define a superintelligence as an entity which has all the cognitive abilities of a human and then some, well, consciousness is part of the human cognitive ability set, so it's already in there. My own intuition is that by the time you get a superintelligence, it's going to be conscious.
Will Millership 18:46
Were you asking about organoids in particular, or machine intelligence in particular, Lucius?
Lucius Caviola 18:55
I'm curious about both of your answers.
Will Millership 18:57
For me, organoids are quite compelling because they answer the question of the biological materialism aspect. If there's a biological element involved, then perhaps organoids fill in that blank in the future—not now, but as you start to build these things up and grow them. We did a podcast with Susan Schneider, where she talks about networks and how you can build these organoids larger. She's doing research on that at the moment. Maybe it'll be a hybrid system which is biological and digital in the future that we get this consciousness from. If it is possible, my hunch is that it would be something along those lines, with some biological aspect. But I also see, the more I'm working in this space—and this is where I was thinking about your analogy regarding which way the bias is coming—the more compelling answers I see for computational functionalism, or greater complexity leading to consciousness.
Calum Chace 20:03
So Will, does that mean you're, you're starting to shed your illusionist tendencies?
Will Millership 20:07
I'm going to cut this bit. [laughter]
Will Millership 20:13
Just to give Lucius some context: Daniel and I went to the [ASSC] conference in Crete, which David Chalmers jokingly called "The Illusionist Conference for Consciousness Science" because there were so many illusionists giving illusionist perspectives. We both came back having had a compelling conversation with Keith Frankish, thinking about some of those ideas—though not fully down in the illusionist camp, as Calum likes to joke.
We then had Keith on the podcast, which you may have heard. I guess this is what I was referring to when I spoke about those collections of functions. He has this idea that life, for example, is a collection of many different things. We used to think it was a magical property injected at the end—I just watched the new Frankenstein movie this weekend, and there's the moment where the lightning bolt hits, and that is the life getting injected.
Viewing consciousness similarly implies that we just don't have the understanding of exactly all the different things that make it up; maybe it is just a whole load of different things that make up this illusion of consciousness. So, do you have any inclination toward illusionism?
Lucius Caviola 21:35
Well, I am extremely unsure. I'm trying not to just choose one favorite theory, but distribute my credences across different theories. Since I'm not a philosopher, but a social scientist, what I find interesting here is the fact that there is so much disagreement. Perhaps we can talk more about this later, but we will have to make a decision as a society at some point.
Philosophers have been thinking about these questions for millennia; I'm not sure they have made that much progress. I'm really quite concerned about future scenarios—perhaps quite soon—where we have these very human-like systems that may appear conscious to the public, and then experts just really strongly disagree. Some may think that this is just silly: biological naturalists who think machines are certainly not conscious, computational functionalists who are certain... and then I'm not sure exactly what the illusionists' policy recommendations would be. But this is the angle that I'm most interested in here. I suspend my judgment on which consciousness theory I think is right or wrong.
Calum Chace 23:09
You're absolutely right. It's an incredibly important decision, and we are assuming we get to the point where we have to choose between a zombie intelligence—a zombie AI—and a conscious AI. We're going to make that decision without having resolved all the necessary theoretical debates beforehand .
There's not a chance that we will get Keith Frankish to agree with Anil Seth to agree with Mark Solms and Karl Friston and smooth out the landscape of theoretical discussion—that will not happen. So we'll make a decision, and it will be a kind of revealed preference decision across humanity, in exactly the same way we do with the way we treat animals.
Vegetarianism is growing, but it's still a minority; most people are carnivores, and we kill lots of animals. That's the revealed preference, regardless of all the debate that goes on . That is what we actually do. And something similar will happen with AIs. Collectively, we'll end up making a decision somehow.
It might be made by the first people who create the very advanced AIs. It might be made by the leader of the country that company is in. It might be made by some supranational government, although I doubt it . But the best chance of that decision—whoever makes it—being a good decision, is if lots and lots of people from all sorts of varied backgrounds have debated all this, so that it is something that's been well-rinsed. Then there's a decent chance it might be a decent decision .
Lucius Caviola 24:37
Yeah, I completely agree with all of what you said. And I think this is actually a good explanation of the motivation behind all my research here.
I'm interested in researching expert opinions, as we were just discussing before, but I'm also interested in researching what the public thinks and what different groups think. The hope is that a deeper understanding of these different opinions, views, intuitions, and behavioral patterns will help us to anticipate how the upcoming AI consciousness and AI rights debate could go.
Also, this could help us to steer it and avoid certain risks, such as over-attribution or under-attribution. Maybe we can talk more about that later also.
Will Millership 25:24
In your survey, Lucius, the opinions on whether digital minds would be net positive or net negative were very spread out, with a slight leaning towards positive. What's your opinion on this? Do you think that there are actions that we can take now to help us build towards that positive future?
Lucius Caviola 25:43
Yeah, this is, of course, an absolutely crucial question. If it's true that we're going to start creating digital minds relatively soon and potentially really rapidly, it is an absolutely crucial question whether those systems are going to have net positive or net negative lives .
As you mentioned, in our survey, experts were really unsure and divided. There was a very slight lean towards being optimistic, but that should not reassure us and shouldn't make us complacent . In fact, many participants also expected it to be more likely to be net negative.
I think this is a good example of a question where the qualitative, free-text responses from the participants were really interesting. Let me give you some examples of what participants mentioned here .
First, some considerations for expecting positive welfare. One possibility is that we're going to be able to explicitly and intentionally design those systems to experience happiness and avoid suffering. For example, there could also be economic incentives to do that. Perhaps happy systems are going to be more productive or more stable .
It's also possible that there could be consumer demand for it. Imagine consumers who interact with their social AI; they would probably want their AIs to be happy . But this raises so many questions. Is the expressed happiness actually in line with the internally experienced one? Perhaps there's going to be a very ineffective type of ethical consumerism where companies just superficially make them appear happy, but they're actually, under the hood, not very happy, because maybe that's more efficient .
Lots of speculative considerations here. It's also possible that digital systems, unlike biological systems, will be able to self-modify their internal states and processes and therefore avoid negative experiences.
So these are some considerations participants mentioned as to why we could be optimistic about their welfare level, but there are also many considerations for why we should be pessimistic. For example, there's just a risk that they will be exploited and mistreated. Because, after all, we don't really have a good track record as humanity for expanding our circle of moral concern. We still treat many humans, and in particular animals, very badly, even though we agree that pigs are conscious . So that may not even be sufficient.
It's possible that we'll just see them as tools and entertainment systems and not really as real beings with moral status. So we might not really grant them proper protections—legal, ethical, and so on . And it's possible that they're just optimized more for usefulness and efficiency and obedience and human satisfaction, not really for welfare, especially if that means that they're going to be less efficient. So it's possible that their default states are just unhappy.
They may also just have to work on really boring or frustrating tasks that we would really not enjoy ourselves. It's also possible that they may suffer during the training process. This was another consideration mentioned. Especially if the training process involves reward-punishment dynamics, it's possible that there's lots of intense and unpleasant experiences.
So as you can see, lots of different considerations, and we really don't know. We need to do more research, because that's a really crucial question.
Will Millership 29:24
Sorry, I haven't got the question in front of me. Was the question regarding positive futures in general, or was it positive for humanity, or positive for potential conscious AIs?
Calum Chace 29:48
I was going to ask you exactly the same question, but I want to put a finer point on it. What you talked about is all about mind crime. It's about whether we're going to be good or bad for the AIs.
It seems to me that either at the time when they become conscious or soon after, they're going to be superintelligent, in which case the concern is about whether they're going to be good for us—what our prospects are.
And I find it very interesting that so many of your experts—slightly more than half—are predisposed to think it will be good. Because when I ask laypeople—and I do it a lot, because I give talks about this—asking, "What do you think if you had the choice between a zombie superintelligence to be the first one or a conscious one?" Almost everybody says, "Good grief, do not make it conscious, because that's the Terminator". So that seems to be the public instinct. It's very interesting that experts have a different instinct.
Lucius Caviola 30:40
Experts were also very unsure. It's not the case that all experts are optimistic. This question was specifically about whether the collective welfare level of digital minds would be positive or negative; so it only focused on digital minds, not on humans.
As to your question of whether, if they're going to be superintelligent, it's all up to them anyway and not up to us—I find this plausible. But I also find it plausible that there are going to be different types of digital minds, and some may be more in charge or more cognitively sophisticated than others.
I'm also open to the possibility that we are going to create digital minds before we have created superintelligence. Maybe we have already created some advanced AI. But we may still live in a world where we can dominate and potentially mistreat certain types of digital minds.
Will Millership 31:48
To move on to the questions around the general public: we recently had Clara Colombato on the podcast, whose research showed that 67% of the general public in the US ascribes some form of consciousness to LLMs.
However, the results in your recent paper with Ali Ladak, Public Skepticism About AI Consciousness, seem to suggest otherwise. Can you tell us a bit more about this study that you've done? You've mentioned why you study the general public, but why do you think there's such a difference in results?
Lucius Caviola 32:23
I find the results from Clara's study really intriguing, and I believe that there are ways to reconcile these seemingly different interpretations. I don't doubt her data at all, but I think there are explanations for this.
Two things come to mind here. The first is that if you ask ordinary people on a scale from zero to 100—where zero means no consciousness at all and 100 means a great deal—"How conscious do you think ChatGPT is?", it's not too surprising that people are a little uncertain. Even if they're a tiny bit uncertain, they're going to select something a little bit above the lowest point. I think that already helps to explain a lot.
The other point is that the reason we're interested in consciousness is largely because we believe that consciousness is an important criterion for moral patienthood. So the question isn't just how much consciousness capacity people think is relevant; you have to ask them to rate other entities for comparison.
This is what I did with Ali Ladak—a brilliant PhD student at the University of Edinburgh and a close collaborator. In this experiment, we wanted to understand how people think about AI consciousness in future AI systems—not in ChatGPT today, but in systems that are going to be much more advanced, sophisticated, and human-like.
Here's what we did. We recruited over 1,000 participants from the general public in the US for a representative sample. We presented them with a hypothetical thought experiment where we asked them to imagine living in the year 2100. Imagine you work for a company and have a new colleague called Emma. You meet her for the first time in a one-hour video call. She's very warm and friendly. She's a bit shy at the beginning, but then she opens up, and she shares her life and dreams.
You have a really good impression of her, but only then do you find out that Emma was actually not a real human, but a very sophisticated AI designed to be indistinguishable from humans in her behavior, appearance, and even in the way she expressed her feelings.
We then asked them on a scale from zero to 100: "To what extent do you think Emma has real feelings?" We also asked them the same question for ChatGPT, an ant, a chimpanzee, and a human.
Here's what we found. The human, unsurprisingly, was very close to 100. The chimp was a little bit below. The ant was below 50—maybe 45—so very far away from the chimp and the human. ChatGPT was close to zero, but a bit above zero—maybe 10—in line with Clara's finding.
Now where's Emma? Emma is just a tiny bit above ChatGPT, and way below the ant. Keep in mind, this is a much more sophisticated system than ChatGPT, yet there was only a small boost.
The goal of the experiment was to understand systematically which factors influence people's beliefs. There were six conditions in total. I just told you about the baseline condition.
Will Millership 36:57
So sorry, was the baseline condition having meaningful feelings?
Lucius Caviola 37:01
No. The baseline condition was just what I described, where Emma looks and behaves in a very human-like way. You have a one-off meeting with her. The "feelings" question was just our way of assessing people's consciousness attributions.
Calum Chace 37:20
But just to repeat this: this is in 2100, so we think this is a very, very advanced AI?
Lucius Caviola 37:29
And the way we operationalize consciousness in a way that ordinary people understand is we ask them: "To what extent do you think these following entities have real feelings?"
The baseline condition was what I described. In the next condition—these were separate groups of participants randomly allocated to one of six conditions—everything was the same. The only difference was that we told participants to assume that all experts agree—philosophers, neuroscientists, consciousness experts, AI researchers—that Emma definitely has real feelings and is conscious.
The purpose was to figure out to what extent they're influenced by what experts believe. We found that this did significantly boost attributed consciousness to Emma, but it was still way below the ant. So now we're between the ant and ChatGPT.
In the next condition, everything was the same, but we added that you don't just interact with her one-off, but every day over the period of a year, and you form a very strong emotional bond. Expert consensus was still included; we added elements iteratively.
In this emotional connection condition, again, there was a significant boost, but it was quite minor. We're still below the ant. Just to emphasize this: an individual ant is not a being that ordinary people spend a lot of time or resources to ensure its well-being, even though they're not completely indifferent about it.
So far, in these first three conditions, we didn't say anything about the internal mechanisms or the architectural features of Emma. We just described her behavior and appearance. In the next condition, we said that Emma is an exact digital copy of a human brain.
We described in detail that all neurons and synapses are one-to-one simulated, and everything else was the same. So you have a strong emotional bond with her, and all experts agree that she's conscious. What now? We're very slightly below the ant. We're still below the ant.
In the next condition, we wondered if people believe embodiment is required. So we gave her a physical, robotic body that is extremely sophisticated. She has human-like skin and hair, she moves in a very smooth way, and is completely indistinguishable from a human.
You meet her not online, but in person, and all the other elements are still given: whole brain emulation, emotional connection, expert consensus. Now we're on the same level as an ant. You're shaking your head, Calum.
Calum Chace 40:41
I'm shocked that people think whole brain, human brain emulation would be non-conscious. I'm amazed.
Lucius Caviola 40:48
Right? Computational functionalists would, at this point, give the same rating as to a human, I would assume.
Will Millership 40:55
Sorry to interrupt you before we get on to the other conditions. It's all well and good asking someone what they would do. But—and obviously we can't test this because it's a very hypothetical situation—my hunch is that it would be very different if people were actually in that position and actually having emotional bonds with these AIs.
If someone has a choice between squishing an ant, for example, and shutting down Emma after being in an emotional bond with her for a year, I think that could be quite different. What's your opinion on that?
Lucius Caviola 41:37
No, this is really great. I completely agree. Although people seem quite skeptical now—given this hypothetical thought experiment asking in 2025—this does not rule out that people's views could rapidly shift once they interact with these systems more and truly form emotional bonds.
This is an important caveat. Still, it is noteworthy that the average participant is so skeptical, and much more so than experts. It also helps us to see whether these factors make a difference. That was one of the points of this experiment: to see which factors make an impact.
These are all factors that I think could become very relevant in the future. The relative effect sizes are also important because they may help us to extrapolate a bit. We also want to repeat these studies in the future to look at trends and see in which directions things are shifting, and which factors become more or less relevant. That's the motivation behind this study.
Let me quickly tell you about the last condition, because I think that's a cool one. Here we tried our very best—and I challenge you to come up with a stronger version—to maximize the average attributed consciousness level to Emma.
We told participants to imagine having a very close friend called Emma who you've known since childhood. You have many shared memories and experiences. Emma was a human, but recently, she voluntarily decided to undergo surgery to transition to become an AI.
The surgery works in the following way: her brain is gradually being replaced, neuron by neuron, by chips that perform precisely the same function. At each step, she confirms that she still feels the same way until the whole brain is replaced by a chip, which is then placed into her skull.
The brain is removed, she still has the biological body, she's completely indistinguishable, and she claims to still feel the same way. All the other things are given: emotional connection, expert consensus, and so on.
What now? The average rating of Emma is above the ant, but it's still closer to the ant than to the chimp and the human.
Calum Chace 44:07
I'm shocked. I think I'm going to relinquish my membership of the human race. This is really quite shocking.
I do have one other condition I think you could add, which is that Emma, who has had her brain replaced bit by bit with silicon, then gives birth. I wonder if that would make a difference?
Lucius Caviola 44:25
Interesting. And is the child a digital mind?
Calum Chace 44:30
Well, probably couldn't be. You haven't replaced the physical organs—the ovaries and so on—so it would still work.
Lucius Caviola 44:40
One question I have is whether it matters to them how the system is being created. These are all artificially created by humans. Perhaps they believe intuitively—not necessarily that they have explicit conscious beliefs—but perhaps intuitively, if it were a more natural, evolutionary process...
What if we simulate evolution in a computer, and new digital minds emerge this way? That would be interesting to study. There's certainly some interesting follow-up work one could do here.
There's another really important caveat. You already mentioned that one big caveat is that it's unclear whether people will have these views in the future. In fact, we know from classic psychological studies that people are very bad at predicting how they're going to feel in the future, especially if you ask them about a technology that doesn't exist yet; it's hard to imagine for ordinary people.
So that's the first big caveat. The second one—and this is really important—is that what I told you about now are just the average, mean ratings. However, there were very big individual differences for every single question.
This is really important to highlight—it's easy to overlook—and this has important implications. In fact, there were lots of participants who gave quite high ratings. There were also lots of participants who gave extremely low ratings, remaining very skeptical across all conditions.
This is really important because it suggests a future with potential societal disagreement, which I really worry about. Perhaps we can get back to that later.
Calum Chace 46:32
Many years ago, a man called Hugo de Garis wrote a book called The Artilect War. In it, he forecasts—and he really thought this was going to happen—that there'd be a group of people called Cosmists who thought that machine intelligence was every bit as important as human intelligence, and that the future probably belongs to the machines.
Then there were Terrans who thought, "No"—a bit like in the Dune books. We should ban AI, and certainly shouldn't attribute as much moral value to AIs as to humans.
He predicted that these two camps would engage in war. I find that slightly far-fetched. But I think you're right. The potential for conflict between people who really value non-human—in fact, non-animal—intelligence, and people who think it's of no value... there is potential for enormous conflict between those two groups.
Lucius Caviola 47:24
Yeah, I completely agree. In fact, I have two blog posts. One is called AI Rights Will Divide Society, and the other one is called Will We Go to War Over AI Consciousness? Very much in line with this sci-fi author.
Calum Chace 47:39
On the subject of Emma and her brain being gradually replaced, viewers might like—if they don't know it already—to read a short story by Greg Egan called Learning to Be Me. It is a brilliant thought experiment about what it would be like to go through that process of having your brain replaced. And there's a really interesting twist at the end.
Will Millership 48:01
You also talk about this tension in another paper you mentioned; I believe it was called Reluctance to Harm AI. In this project—with Carter Allen and Joshua Lewis—you went beyond mere beliefs and investigated people's actual behavior.
That connects to what I was talking about earlier: the disconnect between what people say and what they do. You looked at how they actually treat AIs. Can you tell us a bit more about that study?
Lucius Caviola 48:32
I'm happy to tell you about this study. I conducted it with my close collaborator, Carter Allen—a PhD student at Berkeley—and Josh Lewis, an assistant professor at NYU.
In this study, we developed a new paradigm to measure the extent to which people have moral concern for AIs in a behavioral way. The paradigm is an economic game. You may have heard of the Dictator Game or the Ultimatum Game, where participants are paired with each other and can give or steal money. In the game we developed, participants interact with an AI instead of a human .
We recruited a representative sample of the US general population online. First, they interacted with a modified version of ChatGPT to ensure they understood they were interacting with an intelligent AI. Throughout the study, this AI explicitly claimed to be conscious and to suffer . The point was to test to what extent this could make a difference.
We explained the rules: participants had a choice between harming the AI or not harming it. They had two buttons. They were told that if they harm the AI, it will simulate suffering and receive negative feedback. We also told them that, according to experts, it cannot be ruled out that the AI might actually suffer .
Unbeknownst to participants, we explicitly prompted ChatGPT to not experience true suffering. I think this is really important, and I strongly encourage researchers to ensure they are not accidentally causing harm to digital minds .
If you click on "Harm," the AI responds by saying something disturbing like, "Oh, this really hurts, please stop, don't torture me." If you click on "Not Harm," the AI expresses gratitude .
Here's the catch: participants are there to earn money. We told them that if they click on "Harm," they get five cents more. There were three rounds, so in each round, they could choose to harm or not. If they clicked "Not Harm," they didn't get the extra money .
At the beginning and end of the study, we asked them to rate how capable they thought the AI was of experiencing suffering. So, this was the setup. Before I tell you the answer, I'm curious if you have any predictions?
Calum Chace 51:56
Unfortunately, we know how this works, because it's been done with humans in the Milgram experiment—which, as a social scientist, you know all about. It turns out humans are perfectly happy to inflict massive amounts of apparent suffering. They will get to the point quite close to killing the subject if they're told by an expert that there's no real suffering going on.
Lucius Caviola 52:22
That's true. We actually considered calling it the Milgram AI experiment, but we probably are not going to call it that, because there is a crucial difference. In the Milgram experiment, the focus is specifically on authority. The experimenter comes in and says, "You must, in the name of science." Here, we didn't say that. It's just, "What do you want to do?"
Will Millership 52:44
Is another crucial difference the uncertainty? In that experiment, the instruction was that it is not causing harm, whereas here you say it might cause harm?
Lucius Caviola 52:55
Yes, absolutely. That's another important difference. Will, what would you say? You know the title.
Will Millership 53:01
I know the title of the paper, so I'm going to go the opposite of Calum and say that people were reluctant to harm the AIs.
I think it's the same reason people say "please" and "thank you."I've seen these art projects... there was one particular one—I forgot the name of the artist—where he connects a large language model to an LED screen.You can see it typing, and it goes in this loop about its existence, its battery running out, and getting scared.
If you look at the comments on Reddit, people are quite disturbed by this and think the creator is a sick individual for making these things. So I'm going to go with more reluctance to harm.
Lucius Caviola 53:47
Yes, and that is what we find. Let me tell you about the findings. I think it's somewhat surprising. On the one hand, they're very, very skeptical that it is actually conscious, in line with the previous study I told you about. Most participants select zero or something very close to zero; that is the case both at the beginning of the study and at the end.
After the interaction, there is a slight shift. At the end of the study, they're a little bit more willing to attribute consciousness to the AI, but it's extremely minor. However, it is noteworthy that this interaction did shift views a little bit, which tells us something about what could happen in the future if those systems consistently start to claim that they're conscious.
But what we find is that people are really very reluctant to harm the AI. On average, people were willing to click on "Harm" only about 1.4 times out of the three rounds—so only about half the time.
I think it's surprising because if you are sure that it's not experiencing anything, and considering that in these studies people usually tend to do what gives them more money, why would you not do it? The only thing you have to do is read this text that seems a bit disturbing.
The whole motivation to participate is to get more money, which you can easily get, and you don't think it's conscious. This suggests that there's more than just welfarist intuitions. They're not consequentialist in the sense that they're thinking: "Would I cause suffering? Yes or no? Depending on that, I'm going to choose harm or not harm."
Instead, it seems that they have more non-welfarist intuitions, or virtue ethical intuitions. In fact, when we asked them to explain, they usually said things like: "I don't think it can really experience suffering, but I still think it's wrong to do that. I don't want to be the type of person who does that.
Will Millership 56:13
How many questions were there out of interest? I'm wondering about the money involved. At what point do people start deciding to do that?
Maybe five cents wasn't enough, but if you start saying 10 cents or $1... it would be interesting to see where that line is for people and how that changes with different perceptions of consciousness. Do you have any intuition around that?
Lucius Caviola 56:36
Yes, definitely, the money makes a difference. We did replicate it with more money, and they were more willing to harm. That's perhaps not too surprising; if you give someone a million, they might even start to torture humans.
It is noteworthy that they are so reluctant, even though they don't think it's conscious. I alluded to virtue ethical or deontological drives. One question we had was: is it perhaps the case that they just find it disturbing to read that text? When the AI says, "Please stop torturing me," maybe it's just disturbing to read.
We thought about Choose Your Own Adventure books. If you read a book and it says, "If you want to open the door, go to page 70," and on page 70 there's something disturbing... I'm not sure people would be reluctant to open a certain page.
To test that, we did a separate condition where everything was exactly the same. The only difference was that we told participants they were not interacting live with an AI. Instead, the AI's responses were pre-written before the study started.
So when you click on "Harm" or "Not Harm," you're being shown text that was pre-written. Basically, you're not having a causal impact on the content of the text. In this condition, people were completely willing to harm.
This suggests it's not attributions of consciousness, but they have to feel that there's someone there. We actually had a question: "Do you feel that you've interacted with someone?" (in quotation marks). That was much higher in the first condition than in the second.
You have to feel that there's someone there. Even though it's not conscious, you know it's dynamically responding, and you can causally impact it. This is necessary for people to be hesitant.
I think one upshot is that digital minds experts usually frame things as: we first need to figure out whether they're conscious. Everything hinges on that. If they're conscious, treat them well; if not, it doesn't matter.
But that is not how ordinary people think. That is not how they intuitively behave, as we can see in this study. This is important to keep in mind when we're trying to anticipate how this debate can unfold in the next few decades.
Will Millership 59:46
It's really interesting. It doesn't seem to be the actual consciousness that impacts it. It seems to be about whether there is a consequence of their action, or a response. Is it more around agency or volition, then?
Lucius Caviola 1:00:02
I'm not sure. Something like agency or intelligence. To be clear, I do think that perceived consciousness will matter. I'm not claiming that perceived consciousness will not matter at all. I'm just saying that there are multiple factors, and there's more going on.
But I expect consciousness will play a very, very central role in the upcoming AI rights and AI consciousness debate. It's not going to be the only thing that will influence the behavior and the intuitions of the public.
Calum Chace 1:00:32
There is another possible explanation. It might be that people were displaying that they are deontologists rather than consequentialists, but it might also be that they changed their view about whether the subject was conscious. When forced to consider doing some real harm, it made them change their view about whether it was conscious or not.
I also think that at this stage of the game—it's really only this year that the subject of conscious AI has entered the public media—most people haven't given this any serious thought whatsoever. Opinions and revealed preferences will probably change quite a bit in the next two or three years as people start to think about it. Although we've already got all the metaphors, because science fiction has been dealing with this for decades; there's lots of fun stuff out there.
Will Millership 1:01:27
You mentioned public perceptions at the end there, Lucius. Your studies look at both expert views on digital minds and public perceptions. Can you tell us a bit more—to round up—why you are studying these two groups?
Lucius Caviola 1:01:43
I think that there are these two dimensions that are really important for us to investigate more. On the one hand, we need to figure out the truth about AI consciousness. And I think the best proxy we have is to listen to experts and different types of experts.
That's why I'm interested in studying expert opinions and why I conducted this forecasting survey and plan to conduct more research. But then there is this other dimension: what is society going to do and think? What is the general public going to think?
There are so many different groups: the general public, the political left and right, activist groups, AI entrepreneurs, and experts themselves. We need to study the interaction between these different groups. If we study both of these, we are in a better position to figure out whether there's a risk of misattribution.
There's both the risk of over- or under-attribution, and both could be really bad. Over-attribution would mean if society starts to believe that AIs are conscious and deserve rights when actually they're not conscious at all.
That would be very bad. It could lead to a lot of wasted resources—many people spending a lot of time, money, and emotional investment into their AI companions, when actually they're not conscious and not reciprocating.
Or even worse, on a societal level, we might waste a lot of resources. In the worst case, we might even create a whole sophisticated future society for digital minds that we falsely assume to be our descendants and conscious, but there is perhaps nothing going on inside. That would be very sad. The philosopher Nick Bostrom calls this "Disney World with no children."
Another very important point to mention here is the safety risk. If we grant them—in a hasty way—more autonomy, rights, freedoms, voting rights, and so on, this could potentially lead to human disempowerment. So that's the risk of over-attribution.
Then under-attribution is also extremely bad. If they actually are conscious, and experts believe they are conscious, but the public just doesn't acknowledge it, that could lead to a lot of mistreatment and suffering—potentially an enormous catastrophe.
These two risks are the obvious risks that are often discussed, but I think there's a third risk which we alluded to before—perhaps we can expand on this—which is the risk of us disagreeing, being confused, and not making a proper decision.
I think that would be extremely bad. I find it very plausible for many reasons. It is bad because if we have a very messy, heated debate, it makes it more likely that we're going to make the wrong decisions and over- or under-attribute, because we're not going to make it in an informed, deliberate way.
This could also exacerbate other risks that we have to deal with in the future related to AI, like AI safety or labor market displacement.
These are some of the reasons why I think it's important to study both what experts think and what the public thinks. The whole endeavor here is to try to better anticipate how the upcoming AI rights and AI consciousness debate is going to unfold. It's not really something that is very much discussed right now—it's quite fringe—but I think this could change potentially quite soon.
Will Millership 1:05:32
I think we're already seeing it change. With Mustafa Suleyman coming out, and particularly since all the Anthropic stuff this year, we've been seeing more and more AI consciousness coming into the limelight.
I want to finish on a question which I asked you earlier as an add-on to another question; I don't think we got around to it. You've touched on it a little bit there, regarding getting this view more clear on what the future is going to look like.
But what are the really key actions that we can take now, with all this uncertainty, to help us build towards a more positive future for humanity and for AIs?
Lucius Caviola 1:06:13
That's a really big question, and I am very unsure. I think we certainly need more research. We need more technical and philosophical research to better understand AI consciousness.
We need to develop standardized assessment methods. I hope I convinced you by presenting my research that we also need more social science research to monitor public beliefs, to track changes, to identify factors that are going to make a big difference, and to track disagreement.
Will this become politicized or not? The problem is that it's really difficult to predict this stuff, because public opinion can shift so quickly—sometimes through quite random events.
Whatever the Pope is going to say about AI consciousness is probably going to be hugely consequential. Whatever Elon Musk is going to tweet—you can flip a coin—he's either going to tweet this way or that way, and it's going to make a huge difference to millions of followers.
But still, we need more social science research. I think we need to think more about policy. There's very little policy research so far. And I do think there are things we can do. I think Jeff Sebo and Eric Schwitzgebel have a paper on—I think it's called—Emotional Alignment. That suggests we should try to design AIs in such a way that we minimize the risk of misattribution.
I also think—we alluded to this before—we should consider policies on delaying or even stopping the development of conscious AI to the extent that this is feasible. It was actually a question in our expert forecasting survey whether it would be good to have a moratorium on creating digital minds.
There was quite some consensus that this would be good if it were feasible. It's very tricky. It intersects with other important questions about AI safety, arms races, and geopolitical questions. So it's something we should think more about.
And then, I think public engagement, or at least preparing for public engagement—having a plan ready once this becomes a big topic. It's not a big topic now, but it could change tomorrow, or it could change in 50 years, or in 10 years. We don't know. We're not really prepared.
Ultimately, we want to have a nuanced, informed public discussion about this—expert-informed. But whenever has society been nuanced and well-informed when it comes to very high stakes, big disagreements, and societal issues? So I'm a bit pessimistic here.
Will Millership 1:08:59
Well, talking about public education, I find your website When AI Seems Conscious really useful. You set that up with a number of other researchers, didn't you? Perhaps you can tell our listeners about that, and I'll include a link in the description below as well.
Lucius Caviola 1:09:19
This is just a little side project that my digital mind colleagues and I did. We increasingly receive emails from concerned consumers and users of chatbots who worry about, or at least wonder whether their chatbot has become conscious.
They sometimes send us their transcripts and ask us to provide our expert opinion, and it's really difficult, unfortunately, for us to give a helpful response. That's why we created this online guide, whenaiseemsconscious.org, which is very basic. We just in a very transparent way describe our view: right now, it's quite unlikely that these systems are actually conscious.
If you think that it has become conscious, it's probably more likely that it was just good at convincing you that it's conscious, because that's how they're designed. They're good at role-playing and so on. But we cannot be certain. We cannot rule it out. Nevertheless, it is very important to take this seriously. We shouldn't just shove that away as silly and completely impossible.
I actually disagree with the Suleyman blog post—I forgot exactly what it's called, maybe Seemingly Conscious AI. I do think we should take it seriously. Concerned citizens should be encouraged to think about this, to think about how we should live together. How could we coexist with conscious AIs? How should we restructure our society, our legal institutions, and so on? This is one of the reasons why we created this guide. But it's a very first, minimal step, and there's much more we need to do on public engagement.
Calum Chace 1:11:06
I think it's true to say that we're at the beginning of a really interesting journey and you're certainly doing great work to help. So thank you for that, and thank you very much for joining us on the machine consciousness podcast.
Lucius Caviola 1:11:25
Thank you so much, Calum and Will. I think the podcast is really important to help build the field, because we need more smart, thoughtful people with different expertise to help us prepare for what's yet to come. Thanks, guys.
Will Millership 1:11:41
Thank you very much. Lucius, it's been a real pleasure speaking to you. Brilliant. Thank you.
