# Masters of Scale Summit '26 · Demis Hassabis — transcript（原文存档） | 项 | 值 | |---|---| | 形态 | 视频 / 峰会现场对谈 · 自动转录字幕（逐行带 `MM:SS` 时间码） | | 节目 | Masters of Scale · Summit '26 现场对谈 | | 单集标题 | Google DeepMind's Demis Hassabis: How AI could become dangerous (with Reid Hoffman) \| Summit '26 | | 主持 / 嘉宾 | Reid Hoffman / Demis Hassabis（Google DeepMind 联合创始人兼负责人、Alphabet 首席科学家） | | 录制 | 2026-10，美国旧金山 Presidio Theatre，Masters of Scale Summit 现场 | | 上线 | 2026-11-05 | | 时长 | 约 25 分钟（本份字幕末帧 `25:13`） | | 侃姐发来 | 2026-11-08 21:39，Downloads 里的 txt | | 转录性质 | **自动生成**。含口吃、重复词、专名误识（`Dennis Hades` / `Reed Hoffman` / `Alph Go` / `Asimoff` / `adne` 等）与漏标的说话人切换 | | 来源 | Masters of Scale 单集页 / YouTube；专名与活动信息核对参考 ceointerviews.ai、metapodcast.ai、geekynews.org 的同期报道 | | 状态 | 原文未加工、未校对。**不在本文件内修正任何字词**；译文校正与转录错误清单见 `../notes/2026-11-08-masters-of-scale-summit-26-demishassabis-zh.md` | | 定位方式 | 本份**有时间码**（`MM:SS`），译文用 ¶ 段落编号 + `tMM:SS` 锚点；不补写、不外推时间码 | --- 00:01 I think a lot of the cyber issues that 00:03 00:03 we've seen in the last six months or so 00:06 00:06 could have been addressed with better 00:07 00:07 engineering, more robust engineering. 00:09 00:09 The next phase which I think carries the 00:11 00:11 most risk is closed RSI, closed loop 00:14 00:14 without the human reviewer process and 00:16 00:16 that will obviously be faster but I 00:18 00:18 think is also a very dangerous way to 00:19 00:19 get to full AGI and our international 00:22 00:22 institutions are very very weak as we 00:23 00:23 all know just at the time when we need 00:25 00:25 them to be strongest in in a way. So 00:27 00:27 that's a bit unfortunate. But 00:28 00:28 >> by the way, British a bit unfortunate is 00:31 00:31 holy. Just for translate to translate 00:34 00:34 another translation. Yeah. It's like tea 00:36 00:36 time and dinner time. Exactly. A bit 00:38 00:38 unfortunate. 00:42 00:42 >> Please welcome Dennis Hades and Reed 00:44 00:44 Hoffman. 00:50 00:50 >> Dennis, thank you for joining us. 00:51 00:51 >> Great to be here. Right. 00:52 00:52 >> Thanks for having me. 00:53 00:53 >> We've known each other for I think well 00:55 00:55 over 10 years now. maybe approaching 12 00:57 00:57 or something like that. And uh it was an 00:60 00:60 honor when you asked me to be on the AI 01:03 01:03 ethics board for Google for you know for 01:05 01:05 our various funny 01:06 01:06 >> meetings. Uh yes and it's been great. So 01:10 01:10 one of the things we've been trying to 01:11 01:11 do uh at the summit is to be envisioning 01:16 01:16 the world that we as we'd want to make 01:17 01:17 it. The theme has been agency and you 01:20 01:20 know as you have been a leader in and 01:22 01:22 knew well before most people were 01:24 01:24 thinking about AI that one of the key 01:25 01:25 things for thinking about AI for human 01:27 01:27 benefit is medicine and medical. say a 01:30 01:30 little bit about how you envision this, 01:32 01:32 how this helps, 01:34 01:34 >> you know, people say, "Oh, if I could be 01:36 01:36 getting that, I understand why you AI 01:40 01:40 guys are actually trying to build 01:41 01:41 something that creates 01:43 01:43 >> a longer human experience, a more 01:46 01:46 quality of experience, something that's 01:47 01:47 not it's you call it anti-exrisk." 01:49 01:49 >> Yeah, for sure. I mean if we if we take 01:51 01:51 a step back for a moment is like um big 01:54 01:54 picture and and you know this having 01:56 01:56 been sort of um knowing us from quite 01:58 01:58 early in our deep mind journey that the 02:00 02:00 the whole point of me working on AI for 02:03 02:03 my whole career was to use it as a kind 02:05 02:05 of like the ultimate tool to to advance 02:07 02:07 science and and medicine and our 02:10 02:10 understanding of the world around us but 02:12 02:12 also improving human health. I've always 02:14 02:14 said that the number one thing we should 02:15 02:15 use these amazing tools that we've now 02:17 02:17 built and we were, you know, dreaming 02:19 02:19 about 20 years ago is to improve the 02:22 02:22 human condition and improve human 02:23 02:23 health. And I've tried to do that 02:25 02:25 certainly with my own uh kind of more 02:28 02:28 personal projects. uh obviously things 02:30 02:30 like Alphafold um but also uh the the 02:33 02:33 spinout drug discovery company 02:34 02:34 Isomorphic um I've tried to push it 02:37 02:37 really hard in that direction and as I 02:39 02:39 know you have too more recently and I 02:42 02:42 think it's exactly the sorts of things 02:44 02:44 we need to be showing the world that AI 02:46 02:46 can do. I think um you know what better 02:48 02:48 thing than to you know use AI to help us 02:50 02:50 cure terrible diseases. So some of the 02:52 02:52 protein folding was this kind of 02:55 02:55 intersection of the idea of like we have 02:57 02:57 really good data. We could use scale 02:59 02:59 learning because the whole learning like 03:01 03:01 the the cognitive revolution from 03:03 03:03 learning starting with even pong but the 03:07 03:07 you know and selfplay and all the rest 03:08 03:08 is like okay that's this would be a 03:09 03:09 really good way of doing it. How does um 03:12 03:12 like the kind of the alpha fold lead to 03:14 03:14 medicines? Like what's part of the arc 03:16 03:16 and and you know because one of the 03:17 03:17 things is a question I get frequently is 03:18 03:18 like yeah yeah you're talking about that 03:20 03:20 but is it true? Is it going to happen 03:22 03:22 anytime soon? You know D. So I'm you 03:24 03:24 know I'm I'm passing along the question 03:26 03:26 I get to you. 03:27 03:27 >> Yes. Sure. Well look biology and um and 03:30 03:30 is the most complex you know systems in 03:33 03:33 the universe right? So that's why it's 03:35 03:35 so hard because of the emergent behavior 03:38 03:38 um the interactions and proteins are the 03:42 03:42 building blocks of life. That's how um 03:44 03:44 I've always seen them and and the 03:46 03:46 protein folding problem which is what we 03:48 03:48 you know tried to solve with alphafold 03:50 03:50 was considered to be like a 50-year kind 03:53 03:53 of grand challenge in biology. It's 03:55 03:55 sometimes described to me as as like 03:56 03:56 Fermat's last theorem but for biology 03:59 03:59 and it's critical to understand the 04:01 04:01 structure of a protein from its genetic 04:03 04:03 sequence it amino acid sequence to 04:05 04:05 understand its function and that's 04:07 04:07 really important to understand basic 04:08 04:08 biological mechanisms but also to 04:11 04:11 understand um you know what goes wrong 04:13 04:13 in disease and what things to target you 04:16 04:16 know if you imagine you're designing a 04:17 04:17 drug then you need to know what the 3D 04:20 04:20 surface of the protein looks like so you 04:22 04:22 know which part of that protein uh to to 04:24 04:24 bind into. So it was always a critical 04:27 04:27 but not obviously only still a small 04:29 04:29 part of the entire drug discovery 04:31 04:31 process but it's absolutely critical 04:33 04:33 part of that and we were able to you 04:36 04:36 know with AlphaFold accelerate it you 04:38 04:38 know by by orders of magnitude um um by 04:42 04:42 building um AlphaFold and if you think 04:44 04:44 about it we finished AlphaFold 2 back in 04:46 04:46 2020 which is you know six plus years 04:49 04:49 ago now and it's still probably the best 04:51 04:51 example of concrete example of AI I 04:54 04:54 applied to a really hard scientific 04:56 04:56 problem uh that was thought to be sort 04:58 04:58 of uncrackable. Um and so that just 05:00 05:00 shows you how far ahead of its time it 05:02 05:02 was. And now you know we're really proud 05:03 05:03 of of the impact it's made. Like 4 05:05 05:05 million researchers around the world use 05:07 05:07 it. Pretty much every biology and 05:08 05:08 medical researcher in the world almost 05:10 05:10 every drug in future will have made use 05:13 05:13 of alpha fold all the structures. But 05:15 05:15 it's not enough. We want to expand it to 05:17 05:17 biochemistry and um also predicting 05:21 05:21 things like toxicity or adne properties, 05:23 05:23 all of these other things that you need 05:24 05:24 to know about before you can have a drug 05:27 05:27 that's safe to use and that's efficacus, 05:30 05:30 you know, efficient in the in in 05:32 05:32 whatever it is you're trying to tackle. 05:33 05:33 >> Y indeed. 05:38 05:38 Um, so speaking of accelerations, 05:41 05:41 >> uh, let's go to one of the hot topics of 05:43 05:43 the day, uh, recursive self-improvement. 05:46 05:46 >> Uh, what is your kind of theory of the 05:50 05:50 game? And I say this because you, you 05:52 05:52 know, are a master games person. So I 05:54 05:54 use that as kind of deliberate. And if 05:55 05:55 you don't know what I'm referring to, 05:57 05:57 either look at the documentary or look 05:58 05:58 it up on Wikipedia. Uh, so like RSI, 06:03 06:03 opportunities, concerns. 06:04 06:04 >> Yeah. Well, look, RSI uh it's obviously 06:08 06:08 an extremely uh useful technique to make 06:12 06:12 progress. It mostly works best in what I 06:15 06:15 call verifiable domains, what's called 06:17 06:17 verifiable viable domains. So, um math 06:20 06:20 is one, which is why we're seeing so 06:22 06:22 much advance there. Coding's another um 06:24 06:24 but originally it was games. Actually, 06:26 06:26 one of the first sort of famous 06:28 06:28 recursive self-improving systems was one 06:30 06:30 we invented quite a lot of years ago now 06:32 06:32 called Alpha Zero, which was off the 06:34 06:34 back built off the back of Alph Go, 06:36 06:36 which was the kind of, you know, famous 06:38 06:38 program that beat the world champion at 06:40 06:40 Go back in 20, you know, 10 years ago 06:42 06:42 now, 2016, and kind of was the 06:44 06:44 beginning, I think, of the modern AI 06:46 06:46 era. And then we built a second version 06:48 06:48 that was in some ways more 06:49 06:49 consequential, Alpha Zero, that didn't 06:52 06:52 need to use any human data anymore. And 06:55 06:55 um it just basically played against 06:56 06:56 itself. Started at random and then you 06:59 06:59 know around a dozen to two dozen 07:01 07:01 different epochs. You know it play like 07:03 07:03 a million games against itself. Uh it 07:05 07:05 would then train a new version of itself 07:08 07:08 uh uh that was slightly better than the 07:10 07:10 old version and then that would play a 07:12 07:12 million games against itself and so on 07:14 07:14 and and for example and then it could 07:15 07:15 play any game not just go it could play 07:18 07:18 any two-player perfect information game. 07:20 07:20 And for me it was especially visceral as 07:22 07:22 a chess player because I remember 07:24 07:24 watching it play chess and within um you 07:28 07:28 know 12 hours basically or less than 12 07:31 07:31 hours it would start from random play in 07:33 07:33 the morning to you know you'd have your 07:35 07:35 coffee break at 11:00 a.m. and I could 07:37 07:37 still just about beat it. Then by 07:39 07:39 lunchtime it was grandmaster level and 07:41 07:41 then by by tea time it was better than 07:44 07:44 um the any any chess playing entity. 07:46 07:46 time is dinner time for the US computer 07:48 07:48 time to you know or computer you know 07:51 07:51 kind of handcrafted chess computer is 07:53 07:53 better than all of them so we sort of 07:56 07:56 saw the power of that recursive 07:58 07:58 self-improvement and obviously the thing 07:60 07:60 that all these domains have in common is 08:01 08:01 you can verify the answer is it cor is 08:04 08:04 it a correct answer is it an optimal 08:05 08:05 answer is the equation correct it does 08:07 08:07 the code run it did you win the game and 08:10 08:10 the important thing about a verifiable 08:11 08:11 domain is you can verify the answer um 08:13 08:13 so you know when you've you've you 08:15 08:15 you've you've hit the by point. Um, but 08:17 08:17 also that means you can create almost 08:19 08:19 infinite synthetic data because you can 08:22 08:22 just use the current best versions you 08:24 08:24 have to set new problems or make new 08:26 08:26 moves and then um uh just sort of verify 08:30 08:30 uh uh whether the synthetic data was 08:32 08:32 correct. The problem with synthetic data 08:34 08:34 a lot of the time is you can't verify 08:35 08:35 that it's right um from the right 08:37 08:37 distribution but in these cases you can. 08:40 08:40 So we've seen the enormous progress with 08:42 08:42 maths and with um coding and coding's 08:45 08:45 obviously become enormously valuable and 08:47 08:47 everyone is having to do that now. But I 08:50 08:50 do worry about it because in the limit 08:52 08:52 so there's different forms of RSI right 08:54 08:54 now. We're in the the early days of RSI 08:57 08:57 I would say where what's happening is 08:59 08:59 and you're probably all experiencing 09:00 09:00 this as well with your teams is um you 09:03 09:03 know it's making a human engineer maybe 09:06 09:06 three, four, five times more productive. 09:09 09:09 Okay. So, uh but the but the human is 09:11 09:11 very much still in the loop and 09:13 09:13 reviewing code and architecting things 09:15 09:15 and um and and and writing code still. 09:18 09:18 Um but it's all just enhanced. Um the 09:21 09:21 next phase which I think carries the 09:23 09:23 most risk is closed RSI closed loop 09:26 09:26 where you basically manage to somehow 09:28 09:28 figure out how to do it without the 09:29 09:29 human reviewer process uh in this in the 09:32 09:32 in the whole procedure and that um will 09:35 09:35 obviously be faster but I think is also 09:37 09:37 a very dangerous way to get to full AGI 09:40 09:40 because we're likely for two reasons 09:42 09:42 it's it's sometimes used to be referred 09:44 09:44 to 20 years ago as like an intelligence 09:46 09:46 explosion because you don't quite know 09:48 09:48 uh how you know how do you control keep 09:51 09:51 control of such a system uh such a 09:53 09:53 process and then the end result will 09:56 09:56 likely be quite alien. It's going to be 09:57 09:57 like an alien intelligence. There's no 09:59 09:59 reason you to think it would be sort of 10:01 10:01 in the similar kind of mind space that 10:04 10:04 uh humans are in. So that will make it 10:06 10:06 very hard for it to understand for us to 10:08 10:08 understand it and maybe for it to 10:09 10:09 understand us. So I think it's you know 10:12 10:12 we should be very cautious as we get 10:14 10:14 closer towards that closed loop form of 10:16 10:16 RSI and I think all the AI researchers 10:19 10:19 sort of know that but there is a sort of 10:21 10:21 race to the there's this sort of intense 10:23 10:23 obviously uh uh uh competitive race to 10:27 10:27 improve your coding systems. So presume 10:29 10:29 that the the incentives to be in a race 10:33 10:33 continue and you know you and I some 10:35 10:35 people really know this you know have 10:37 10:37 tried to figure out if there's various 10:38 10:38 ways you can get global collaboration to 10:40 10:40 you've dedicated a huge amount of energy 10:42 10:42 to that so thank you um uh it's a hard 10:45 10:45 game as you know but presume that the 10:47 10:47 competitive pressure for continuing to 10:50 10:50 do RSI and coding continues 10:53 10:53 >> in that environment what are the best 10:54 10:54 ways that we can potentially navigate 10:58 10:58 that because we have to see kind of 10:59 10:59 where RSI is going, what it's pointed 11:01 11:01 at. What are the kinds of things that 11:03 11:03 the, you know, the leading labs such as 11:05 11:05 your own and the others, 11:07 11:07 >> what should they be doing to try to 11:09 11:09 navigate this environment? Well, what 11:11 11:11 we're doing and I think other leading 11:12 11:12 labs are trying to do is you got to uh 11:15 11:15 so there's there's more prosaic things 11:17 11:17 like make sure uh uh cyber defense is 11:21 11:21 accelerating faster than cyber attack 11:22 11:22 and making sure you give access to 11:24 11:24 trusted cyber defenders first on on 11:27 11:27 maybe slightly fine-tuned versions of 11:29 11:29 your general models that are fine-tuned 11:31 11:31 for cyber defense. Um and so you keep 11:34 11:34 the defenders ahead of the attackers. 11:36 11:36 It's a pretty tricky problem to do in 11:37 11:37 general. um also building much more 11:40 11:40 hardened sandboxes. I think a lot of the 11:43 11:43 uh cyber kind of issues that we've seen 11:46 11:46 in the last six months or so could have 11:48 11:48 been addressed with better engineering, 11:50 11:50 more robust engineering. I think better 11:52 11:52 you need really amazing monitoring 11:53 11:53 systems. You know, they shouldn't be 11:55 11:55 possible for agents to be running for 11:56 11:56 for weeks without anyone realizing. You 11:59 11:59 know, these kinds of things that we've 12:00 12:00 seen some of our uh some of our 12:02 12:02 competitors and others maybe, you know, 12:04 12:04 it's just the beginning of this 12:05 12:05 happening. So I think everyone will get 12:07 12:07 rapidly better but they have to as well 12:09 12:09 because otherwise something bad's going 12:11 12:11 to happen inadvertently. Some kind of 12:13 12:13 incident you know I sometimes call it 12:14 12:14 like an oil spill is going to happen on 12:17 12:17 the current trajectory if we're not if 12:19 12:19 we don't get uh collectively if if 12:21 12:21 everyone doesn't pay a lot more 12:22 12:22 attention and put a lot more resources 12:24 12:24 on these types of systems eventually you 12:27 12:27 kind of want formal guarantees maybe 12:29 12:29 some bounds around what these systems 12:31 12:31 can do uh test how robust like the 12:33 12:33 things like the guardrails are. This is 12:35 12:35 actually something I encourage a lot of 12:36 12:36 top mathematicians that I know well who 12:38 12:38 are maybe, you know, pretty sad about 12:40 12:40 the state of pure maths, but they their 12:43 12:43 skills would be incredible uh if they 12:45 12:45 pivoted to these types of domains, more 12:48 12:48 like complexity theory, game theory. How 12:50 12:50 do you uh and of course when they're 12:52 12:52 going to have to start dealing with 12:53 12:53 swarms of these agents, potentially 12:55 12:55 collaborating in some way. Um how does 12:57 12:57 one keep track of all of that? And 12:59 12:59 should there be ways that the labs are 13:02 13:02 encouraged, required, permitted to 13:05 13:05 collaborate on this? And if so, what 13:07 13:07 should those what should some of those 13:09 13:09 ways be? Because it obviously gets 13:11 13:11 stronger if there is sharing of 13:13 13:13 intelligence, methodology. Yeah. 13:15 13:15 >> You know, mutual accountability. 13:16 13:16 >> Yeah. I mean, there's there's been lots 13:17 13:17 of proposals and I've been working on 13:19 13:19 these things a lot. And you you you know 13:20 13:20 you saw my essay over the summer, maybe 13:22 13:22 some of you read it about potential for 13:25 13:25 like an SRO industry run standards body 13:29 13:29 with some kind of government uh 13:31 13:31 oversight over it. Um I think something 13:33 13:33 like that is going to be needed and then 13:35 13:35 ideally it becomes international. You 13:37 13:37 know it's USled and then obviously the 13:39 13:39 state of geopolitics today makes it and 13:41 13:41 and our international institutions are 13:42 13:42 very very weak as we all know uh just at 13:45 13:45 the time when we need them to be 13:46 13:46 strongest in in a way. So that's a bit 13:49 13:49 unfortunate, but we got to start 13:50 13:50 somewhere. 13:51 13:51 >> By the way, British a bit unfortunate is 13:53 13:53 holy. Just for translation. 13:57 13:57 Yeah. It's like tea time and dinner 13:58 13:58 time. Yes. Exactly. A bit unfortunate. 14:01 14:01 and um 14:04 14:04 and and so we're trying to you know 14:05 14:05 various of us are trying to do things 14:07 14:07 about that and and that essay did have 14:09 14:09 an impact and uh you know in the white 14:11 14:11 house and in administration and and with 14:13 14:13 our colleagues and eventually you know I 14:16 14:16 think some of that energy became the 14:18 14:18 accord that you the famous letter 14:19 14:19 everyone signed which is which is better 14:21 14:21 than nothing. I don't mean goes far 14:23 14:23 enough but it's a start and at least it 14:25 14:25 acknowledges from all of the main 14:26 14:26 players uh uh the issues that are there 14:29 14:29 and and that they need to be addressed. 14:31 14:31 doesn't say too much about what to do 14:32 14:32 about them but you know that has to come 14:34 14:34 next and at least it gives air cover I 14:36 14:36 think for the teams to work together on 14:39 14:39 you know what sorts of benchmarks what 14:40 14:40 are safe best practices on safety what 14:42 14:42 are you seeing you know uh uh in terms 14:44 14:44 of analysis of what things could happen 14:47 14:47 with these types of systems and I do 14:49 14:49 think there are some things that 14:49 14:49 generally all the leaning labs do agree 14:51 14:51 on like um the idea of third party 14:54 14:54 auditors or evaluators coming in 14:56 14:56 obviously there's a question mark which 14:58 14:58 third parties and and I think we need a 15:00 15:00 healthy kind of ecosystem of that that 15:02 15:02 actually you know you could have good 15:05 15:05 startups and companies doing that that 15:07 15:07 that that there's enough revenue there 15:08 15:08 for them and there should be given the 15:10 15:10 amount of investment going into AI as a 15:12 15:12 whole bit like there are auditors you 15:13 15:13 know ini in the finance industry or any 15:15 15:15 of these industries so I could see that 15:17 15:17 happening um and some of the big players 15:19 15:19 are moving into this like Accenture 15:21 15:21 we're working in other leaders are 15:23 15:23 working with I think they're a little 15:24 15:24 bit ahead of the game because they've 15:25 15:25 acquired you know a good startup good 15:26 15:26 friend of ours Mark Warner startup so 15:28 15:28 they have some expertise internally but 15:31 15:31 I think we need more of that and then 15:32 15:32 other things like you know the leading 15:34 15:34 labs maybe red teaming each other uh 15:36 15:36 before uh general release of these 15:39 15:39 systems 15:39 15:39 >> so let's move from that to a little bit 15:41 15:41 more of the fantastical you know it 15:43 15:43 started actually a few years back I 15:45 15:45 think there was a Google engineer who 15:46 15:46 said I asked Gemini if it was conscious 15:48 15:48 and it said yes so therefore it's 15:49 15:49 conscious which seemed like a little bit 15:51 15:51 of a a weak inference that's now gaining 15:55 15:55 some motion with some of the you know 15:57 15:57 things especially anthropic is doing and 15:60 15:60 and in and the way that it u kind of 16:02 16:02 strategizes thinking that well at some 16:03 16:03 point it's almost like a Pascal's wager 16:05 16:05 at some point there will be 16:07 16:07 consciousness in these systems and we 16:08 16:08 want want the those conscious systems to 16:10 16:10 look back saying that we were treating 16:12 16:12 them the way the same way that we like 16:14 16:14 treat a 2-year-old um which is a hey 16:16 16:16 look we were good to you as a 2-year-old 16:18 16:18 and so you know it kind of continues 16:20 16:20 through 16:21 16:21 >> obviously to many people that looks kind 16:23 16:23 of wacky right and um and so what do you 16:28 16:28 think I mean cuz Part of what I think 16:29 16:29 maybe many people don't realize is you 16:31 16:31 also come to this from a depth of 16:32 16:32 expertise in brain science 16:34 16:34 >> and you know one of the things is not 16:36 16:36 just a oh did the chatbot say it was 16:38 16:38 conscious or not is not the way to make 16:40 16:40 a decision on this kind of thing. 16:41 16:41 >> What do you think some of the ways to 16:42 16:42 think about this is and also how we 16:44 16:44 should be deliberately trying to shape 16:46 16:46 it. 16:46 16:46 >> Yeah. Well look I think there's quite a 16:48 16:48 few things to say about this. First of 16:49 16:49 all um we haven't despite thousands of 16:53 16:53 years of thinking about this there isn't 16:54 16:54 a agreed upon definition of 16:56 16:56 consciousness. So that's one one issue I 16:58 16:58 think and that's one issue with labeling 17:00 17:00 things conscious um especially the hard 17:02 17:02 problem of consciousness you know is it 17:04 17:04 really feeling this internally like the 17:06 17:06 way we we seem to you know behaviorally 17:08 17:08 these things I don't know about you but 17:10 17:10 yes exactly but the behaviorally well 17:11 17:11 but I used to argue with Daniel Dennett 17:13 17:13 about this that there's an additional 17:14 17:14 thing is we're not just behaving in ways 17:16 17:16 like we'd imagine conscious beings 17:18 17:18 behave but we also share the same 17:20 17:20 substrate we have the same squishy you 17:22 17:22 know brains and so then it's most 17:24 17:24 parsimmonious to think you're feeling 17:25 17:25 what I'm feeling But of course with a 17:28 17:28 with an AI that's not the case. You 17:29 17:29 don't have the second the substrate 17:31 17:31 equivalent. So I think it's going to be 17:33 17:33 very hard to conclude that. I think 17:35 17:35 there are dangers. Look there's there's 17:36 17:36 two reasons why I think we should be 17:38 17:38 focusing on building intelligent tools 17:39 17:39 today not uh conscious entities. Um I 17:43 17:43 think for one thing is a safety argument 17:45 17:45 and uh technical uh safety and risk 17:48 17:48 which is that you know there's a direct 17:50 17:50 trade-off between creating something 17:53 17:53 let's say uh with selfition right or 17:56 17:56 self-awareness um you know even prior to 17:59 17:59 consciousness these are the 17:60 17:60 prerequisites right to a conscious 18:02 18:02 entity um that it will and that and 18:05 18:05 instruction following or sometimes 18:07 18:07 called courageability. So this idea of a 18:10 18:10 system actually following instructions 18:12 18:12 and if it's going wrong you telling it 18:14 18:14 you know you as the user or the the 18:16 18:16 human user telling it like stop doing 18:17 18:17 that right or switch yourself off or 18:20 18:20 whatever it is that you're going to you 18:21 18:21 know change change course. So if you are 18:23 18:23 training your system to be able to 18:26 18:26 reject commands based on its own 18:28 18:28 internal uh workings, then you're 18:31 18:31 setting up a situation where if you've 18:34 18:34 got your constitution wrong or your 18:36 18:36 rules wrong, and this is what Asimoff 18:37 18:37 was always warning about with these 18:38 18:38 three laws, you know, that was a very 18:40 18:40 simple form of a constitution. And his 18:43 18:43 books are all warnings that that's 18:45 18:45 you're never going to get that right in 18:47 18:47 every corner case, in every edge case. 18:49 18:49 And and and and I think his books show 18:51 18:51 that very well. And yet we're sort of I 18:54 18:54 think a bit hubristically thinking that 18:55 18:55 that some people thinking that you know 18:57 18:57 sometimes called the misalignment 18:59 18:59 problem is sort of under control just 19:01 19:01 because today's systems are you know the 19:03 19:03 good news is they seem more malleable 19:05 19:05 than we might have imagined 15 20 years 19:07 19:07 ago which is great news in terms of 19:09 19:09 their steerability but that doesn't mean 19:11 19:11 we've solved um the misalignment problem 19:13 19:13 in total as we get closer to you know a 19:16 19:16 GI and super intelligence and so on. And 19:18 19:18 I just think um it's it's it's dangerous 19:21 19:21 to think one has. So that's the first 19:23 19:23 thing. There's a technical problem 19:24 19:24 especially if you combine it with 19:25 19:25 something like an RSI. 19:26 19:26 >> Yes. Exactly. 19:29 19:29 Right. So RSI with self self valition is 19:32 19:32 you know sort of doubly dangerous. So 19:35 19:35 that's one thing. Second thing is a 19:37 19:37 societal more of a societal issue which 19:39 19:39 is that I don't think we are ready yet 19:41 19:41 as a society. We haven't had the debate 19:43 19:43 with a wider society, philosophers, 19:45 19:45 theologians, um, artists, humanities, 19:48 19:48 not just technologists as to, you know, 19:51 19:51 and from many different cultures. Like, 19:53 19:53 do we want to cross that second rubicon? 19:55 19:55 Are we ready to have the discussion 19:57 19:57 about do these systems have moral 19:59 19:59 standing? Um, do we need to worry about, 20:02 20:02 you know, what are we doing with them in 20:03 20:03 this box? You know, are they slaves of 20:05 20:05 some sort? you know, all these questions 20:07 20:07 come up that are complex questions and I 20:10 20:10 don't think we're equipped to even begin 20:12 20:12 to start dealing with those questions. 20:14 20:14 So, my view is we should um cross the 20:17 20:17 first Rubicon uh on its own which is 20:19 20:19 build these intelligent tools and then 20:21 20:21 actually use these intelligent tools to 20:23 20:23 do more neuroscience to go back I'd go 20:26 20:26 back and do more neuroscience to 20:28 20:28 actually understand what the neural 20:29 20:29 correlates of consciousness really are. 20:31 20:31 finally understand that and then you 20:34 20:34 know that'll probably take 5 10 years 20:35 20:35 and then cross the second Rubicon well 20:38 20:38 to have a discussion about should we 20:40 20:40 cross the second Rubicon and then if so 20:42 20:42 how right of of the of the conscious 20:44 20:44 entity type 20:45 20:45 >> and how would you cuz you know you 20:47 20:47 engaged in hours of these discussions 20:49 20:49 how would you answer the folks who say 20:50 20:50 look the only way we get the intelligent 20:52 20:52 coralate to be powerful enough is it it 20:55 20:55 has as it were the ephemeral accidental 20:57 20:57 it actually has this you know cognitive 20:59 20:59 state and so 20:59 20:59 >> right so we should talk about this a We 21:01 21:01 discussed this you know even 10 plus 21:03 21:03 years ago and I discussed it in my PhD 21:05 21:05 back 20 plus years ago of look it I felt 21:08 21:08 and I still feel intelligence and 21:10 21:10 consciousness are double dissociable 21:12 21:12 which means that you can have an 21:13 21:13 intelligence system that's very smart on 21:15 21:15 IQ and so on kind of like the systems we 21:17 21:17 have today which I don't I personally 21:19 21:19 don't feel are at all conscious yet 21:21 21:21 right from my own you know everyone 21:23 21:23 probably has their own view but I don't 21:24 21:24 feel yet and then you can have conscious 21:27 21:27 entities uh that are not that smart for 21:30 21:30 example my pet dog. He's lovely, but 21:32 21:32 he's and he's pretty conscious. I think 21:34 21:34 he dreams and everything, but he's not 21:35 21:35 that smart, 21:36 21:36 >> right? 21:36 21:36 >> And that's provable. 21:37 21:37 >> So, I think you can have double 21:39 21:39 dissociation here. You know, many of the 21:41 21:41 higher animals, dolphins, monkeys, and 21:43 21:43 so on, elephants, I think most of us 21:45 21:45 would view their conscious beings, but 21:48 21:48 and they exhibit all of that that sort 21:49 21:49 of behavior. And of of course, they're 21:51 21:51 also made of carbons and and the same 21:53 21:53 neurons and types of things we're made 21:55 21:55 of. So, I feel like it's pretty obvious 21:57 21:57 they have some of those qualities. um 21:59 21:59 but they're not necessarily that smart. 22:01 22:01 So it feels like you can dissociate 22:02 22:02 these things and and I think to the 22:05 22:05 extent that you can I feel like we 22:06 22:06 should 22:07 22:07 >> you know we're coming up close to the 22:08 22:08 end of our session and the end of you 22:11 22:11 know the summit sad um what should we 22:15 22:15 think about in terms of the whole notion 22:17 22:17 of AI sovereignty 22:18 22:18 >> right because obviously one of the one 22:20 22:20 of the things that is kind of going 22:22 22:22 around in a lot of countries right now 22:24 22:24 is oh is this really important and you 22:26 22:26 know one of our earliest conversations 22:28 22:28 was with you trying to solve this 22:30 22:30 problem so I wanted 22:31 22:31 have share some thinking and obviously 22:33 22:33 it's updated for whatever that was 13 14 22:36 22:36 years ago but 22:36 22:36 >> well look I think first of all first 22:38 22:38 thing to say is it looks to me like 22:40 22:40 there's going to be you know half a 22:42 22:42 dozen you know six to eight if you 22:43 22:43 include Chinese companies company you 22:46 22:46 know offerings at the frontier near the 22:48 22:48 frontier and it feels like there's going 22:50 22:50 to be leaprogging and maybe something's 22:52 22:52 like three to six months ahead but it's 22:54 22:54 going to be uh relatively close so 22:55 22:55 there's a lot of options if depends if 22:57 22:57 you're talking about an enterprise or a 22:59 22:59 country right So I think as an 23:01 23:01 enterprise you have a lot of options to 23:03 23:03 arbitrage between these different 23:05 23:05 things. Um some of them are better at 23:07 23:07 certain things you know multimodal 23:08 23:08 understanding versus coding or it's very 23:11 23:11 hard to be better on all vectors of 23:12 23:12 importance at once. So I think that's 23:14 23:14 going to happen and then open source um 23:17 23:17 which also has its risks because how do 23:19 23:19 you as we've discussed for many many a 23:20 23:20 year is how do you restrict um bad actor 23:23 23:23 using that whilst enabling of course as 23:25 23:25 many good actors scientists and others 23:27 23:27 uh to use uh these fantastic tools. So 23:30 23:30 that's an unresolved problem but open 23:32 23:32 source is you know maybe only lagging 6 23:34 23:34 months to a year behind. Uh obviously we 23:36 23:36 need kind of western open source 23:38 23:38 solutions. Um but I think at this point 23:40 23:40 if we're talking about countries as well 23:42 23:42 all companies that need their data to be 23:44 23:44 private or they want to fine-tune 23:46 23:46 further on their own you know sensitive 23:48 23:48 data again be that enterprise data or or 23:50 23:50 sovereign data. I think that's that's 23:52 23:52 going to be one solution. There's also 23:55 23:55 you know looking at what are the 23:56 23:56 downstream things that can be done with 23:58 23:58 these tools you know big companies and 24:00 24:00 big ve sectors that can benefit from 24:03 24:03 these tools like healthcare that you can 24:05 24:05 just use this you know almost like 24:07 24:07 commodities and then build the value on 24:10 24:10 top of it rather than you know I 24:12 24:12 wouldn't recommend every country for 24:13 24:13 example trying to build a a frontier 24:15 24:15 model that just doesn't make sense to me 24:17 24:17 so we're going to they're going to have 24:18 24:18 to find some other way um of 24:19 24:19 shortcutting that 24:20 24:20 >> well and also collaborating and uh 24:23 24:23 working about arrangements with 24:24 24:24 multinational companies like your own. I 24:26 24:26 think one of the reasons I wanted to 24:28 24:28 kind of end on that question is that 24:30 24:30 part of the theme of of the summit is 24:32 24:32 agency and agency is not just a question 24:35 24:35 of individual agency like how do you 24:37 24:37 take agency but it's also a collective 24:39 24:39 how do we take agency 24:41 24:41 >> and I think part of the thing that's 24:42 24:42 actually important in the AI age is that 24:45 24:45 we take agency not just as any one 24:47 24:47 country but as countries together it's 24:49 24:49 part of how we have a we are in this 24:51 24:51 together and we are we are building for 24:54 24:54 the benefit of humanity and I think 24:55 24:55 that's the kind arc of we started with 24:57 24:57 Sarah Frier and thank you for joining us 24:59 24:59 because Demis has been at this with such 25:02 25:02 intensity that I was so glad grateful 25:04 25:04 that you set up the agenda for me 25:06 25:06 thinking about how to help in this from 25:08 25:08 our very first you know discussion at 25:10 25:10 King's Cross. So uh let's give Demis a 25:13 25:13 hand. Thank you. Thank you.