Masters of Scale Summit '26 · Demis Hassabis — transcript（原文存档）
| 项 | 值 |
|---|---|
| 形态 | 视频 / 峰会现场对谈 · 自动转录字幕（逐行带 MM:SS 时间码） |
| 节目 | Masters of Scale · Summit '26 现场对谈 |
| 单集标题 | Google DeepMind's Demis Hassabis: How AI could become dangerous (with Reid Hoffman) | Summit '26 |
| 主持 / 嘉宾 | Reid Hoffman / Demis Hassabis（Google DeepMind 联合创始人兼负责人、Alphabet 首席科学家） |
| 录制 | 2026-10，美国旧金山 Presidio Theatre，Masters of Scale Summit 现场 |
| 上线 | 2026-11-05 |
| 时长 | 约 25 分钟（本份字幕末帧 25:13） |
| 侃姐发来 | 2026-11-08 21:39，Downloads 里的 txt |
| 转录性质 | 自动生成。含口吃、重复词、专名误识（Dennis Hades / Reed Hoffman / Alph Go / Asimoff / adne 等）与漏标的说话人切换 |
| 来源 | Masters of Scale 单集页 / YouTube；专名与活动信息核对参考 ceointerviews.ai、metapodcast.ai、geekynews.org 的同期报道 |
| 状态 | 原文未加工、未校对。不在本文件内修正任何字词；译文校正与转录错误清单见 ../notes/2026-11-08-masters-of-scale-summit-26-demishassabis-zh.md |
| 定位方式 | 本份有时间码（MM:SS），译文用 ¶ 段落编号 + tMM:SS 锚点；不补写、不外推时间码 |
00:01 I think a lot of the cyber issues that 00:03
00:03 we've seen in the last six months or so 00:06
00:06 could have been addressed with better 00:07
00:07 engineering, more robust engineering. 00:09
00:09 The next phase which I think carries the 00:11
00:11 most risk is closed RSI, closed loop 00:14
00:14 without the human reviewer process and 00:16
00:16 that will obviously be faster but I 00:18
00:18 think is also a very dangerous way to 00:19
00:19 get to full AGI and our international 00:22
00:22 institutions are very very weak as we 00:23
00:23 all know just at the time when we need 00:25
00:25 them to be strongest in in a way. So 00:27
00:27 that's a bit unfortunate. But 00:28
00:28 >> by the way, British a bit unfortunate is 00:31
00:31 holy. Just for translate to translate 00:34
00:34 another translation. Yeah. It's like tea 00:36
00:36 time and dinner time. Exactly. A bit 00:38
00:38 unfortunate. 00:42
00:42 >> Please welcome Dennis Hades and Reed 00:44
00:44 Hoffman. 00:50
00:50 >> Dennis, thank you for joining us. 00:51
00:51 >> Great to be here. Right. 00:52
00:52 >> Thanks for having me. 00:53
00:53 >> We've known each other for I think well 00:55
00:55 over 10 years now. maybe approaching 12 00:57
00:57 or something like that. And uh it was an 00:60
00:60 honor when you asked me to be on the AI 01:03
01:03 ethics board for Google for you know for 01:05
01:05 our various funny 01:06
01:06 >> meetings. Uh yes and it's been great. So 01:10
01:10 one of the things we've been trying to 01:11
01:11 do uh at the summit is to be envisioning 01:16
01:16 the world that we as we'd want to make 01:17
01:17 it. The theme has been agency and you 01:20
01:20 know as you have been a leader in and 01:22
01:22 knew well before most people were 01:24
01:24 thinking about AI that one of the key 01:25
01:25 things for thinking about AI for human 01:27
01:27 benefit is medicine and medical. say a 01:30
01:30 little bit about how you envision this, 01:32
01:32 how this helps, 01:34
01:34 >> you know, people say, "Oh, if I could be 01:36
01:36 getting that, I understand why you AI 01:40
01:40 guys are actually trying to build 01:41
01:41 something that creates 01:43
01:43 >> a longer human experience, a more 01:46
01:46 quality of experience, something that's 01:47
01:47 not it's you call it anti-exrisk." 01:49
01:49 >> Yeah, for sure. I mean if we if we take 01:51
01:51 a step back for a moment is like um big 01:54
01:54 picture and and you know this having 01:56
01:56 been sort of um knowing us from quite 01:58
01:58 early in our deep mind journey that the 02:00
02:00 the whole point of me working on AI for 02:03
02:03 my whole career was to use it as a kind 02:05
02:05 of like the ultimate tool to to advance 02:07
02:07 science and and medicine and our 02:10
02:10 understanding of the world around us but 02:12
02:12 also improving human health. I've always 02:14
02:14 said that the number one thing we should 02:15
02:15 use these amazing tools that we've now 02:17
02:17 built and we were, you know, dreaming 02:19
02:19 about 20 years ago is to improve the 02:22
02:22 human condition and improve human 02:23
02:23 health. And I've tried to do that 02:25
02:25 certainly with my own uh kind of more 02:28
02:28 personal projects. uh obviously things 02:30
02:30 like Alphafold um but also uh the the 02:33
02:33 spinout drug discovery company 02:34
02:34 Isomorphic um I've tried to push it 02:37
02:37 really hard in that direction and as I 02:39
02:39 know you have too more recently and I 02:42
02:42 think it's exactly the sorts of things 02:44
02:44 we need to be showing the world that AI 02:46
02:46 can do. I think um you know what better 02:48
02:48 thing than to you know use AI to help us 02:50
02:50 cure terrible diseases. So some of the 02:52
02:52 protein folding was this kind of 02:55
02:55 intersection of the idea of like we have 02:57
02:57 really good data. We could use scale 02:59
02:59 learning because the whole learning like 03:01
03:01 the the cognitive revolution from 03:03
03:03 learning starting with even pong but the 03:07
03:07 you know and selfplay and all the rest 03:08
03:08 is like okay that's this would be a 03:09
03:09 really good way of doing it. How does um 03:12
03:12 like the kind of the alpha fold lead to 03:14
03:14 medicines? Like what's part of the arc 03:16
03:16 and and you know because one of the 03:17
03:17 things is a question I get frequently is 03:18
03:18 like yeah yeah you're talking about that 03:20
03:20 but is it true? Is it going to happen 03:22
03:22 anytime soon? You know D. So I'm you 03:24
03:24 know I'm I'm passing along the question 03:26
03:26 I get to you. 03:27
03:27 >> Yes. Sure. Well look biology and um and 03:30
03:30 is the most complex you know systems in 03:33
03:33 the universe right? So that's why it's 03:35
03:35 so hard because of the emergent behavior 03:38
03:38 um the interactions and proteins are the 03:42
03:42 building blocks of life. That's how um 03:44
03:44 I've always seen them and and the 03:46
03:46 protein folding problem which is what we 03:48
03:48 you know tried to solve with alphafold 03:50
03:50 was considered to be like a 50-year kind 03:53
03:53 of grand challenge in biology. It's 03:55
03:55 sometimes described to me as as like 03:56
03:56 Fermat's last theorem but for biology 03:59
03:59 and it's critical to understand the 04:01
04:01 structure of a protein from its genetic 04:03
04:03 sequence it amino acid sequence to 04:05
04:05 understand its function and that's 04:07
04:07 really important to understand basic 04:08
04:08 biological mechanisms but also to 04:11
04:11 understand um you know what goes wrong 04:13
04:13 in disease and what things to target you 04:16
04:16 know if you imagine you're designing a 04:17
04:17 drug then you need to know what the 3D 04:20
04:20 surface of the protein looks like so you 04:22
04:22 know which part of that protein uh to to 04:24
04:24 bind into. So it was always a critical 04:27
04:27 but not obviously only still a small 04:29
04:29 part of the entire drug discovery 04:31
04:31 process but it's absolutely critical 04:33
04:33 part of that and we were able to you 04:36
04:36 know with AlphaFold accelerate it you 04:38
04:38 know by by orders of magnitude um um by 04:42
04:42 building um AlphaFold and if you think 04:44
04:44 about it we finished AlphaFold 2 back in 04:46
04:46 2020 which is you know six plus years 04:49
04:49 ago now and it's still probably the best 04:51
04:51 example of concrete example of AI I 04:54
04:54 applied to a really hard scientific 04:56
04:56 problem uh that was thought to be sort 04:58
04:58 of uncrackable. Um and so that just 05:00
05:00 shows you how far ahead of its time it 05:02
05:02 was. And now you know we're really proud 05:03
05:03 of of the impact it's made. Like 4 05:05
05:05 million researchers around the world use 05:07
05:07 it. Pretty much every biology and 05:08
05:08 medical researcher in the world almost 05:10
05:10 every drug in future will have made use 05:13
05:13 of alpha fold all the structures. But 05:15
05:15 it's not enough. We want to expand it to 05:17
05:17 biochemistry and um also predicting 05:21
05:21 things like toxicity or adne properties, 05:23
05:23 all of these other things that you need 05:24
05:24 to know about before you can have a drug 05:27
05:27 that's safe to use and that's efficacus, 05:30
05:30 you know, efficient in the in in 05:32
05:32 whatever it is you're trying to tackle. 05:33
05:33 >> Y indeed. 05:38
05:38 Um, so speaking of accelerations, 05:41
05:41 >> uh, let's go to one of the hot topics of 05:43
05:43 the day, uh, recursive self-improvement. 05:46
05:46 >> Uh, what is your kind of theory of the 05:50
05:50 game? And I say this because you, you 05:52
05:52 know, are a master games person. So I 05:54
05:54 use that as kind of deliberate. And if 05:55
05:55 you don't know what I'm referring to, 05:57
05:57 either look at the documentary or look 05:58
05:58 it up on Wikipedia. Uh, so like RSI, 06:03
06:03 opportunities, concerns. 06:04
06:04 >> Yeah. Well, look, RSI uh it's obviously 06:08
06:08 an extremely uh useful technique to make 06:12
06:12 progress. It mostly works best in what I 06:15
06:15 call verifiable domains, what's called 06:17
06:17 verifiable viable domains. So, um math 06:20
06:20 is one, which is why we're seeing so 06:22
06:22 much advance there. Coding's another um 06:24
06:24 but originally it was games. Actually, 06:26
06:26 one of the first sort of famous 06:28
06:28 recursive self-improving systems was one 06:30
06:30 we invented quite a lot of years ago now 06:32
06:32 called Alpha Zero, which was off the 06:34
06:34 back built off the back of Alph Go, 06:36
06:36 which was the kind of, you know, famous 06:38
06:38 program that beat the world champion at 06:40
06:40 Go back in 20, you know, 10 years ago 06:42
06:42 now, 2016, and kind of was the 06:44
06:44 beginning, I think, of the modern AI 06:46
06:46 era. And then we built a second version 06:48
06:48 that was in some ways more 06:49
06:49 consequential, Alpha Zero, that didn't 06:52
06:52 need to use any human data anymore. And 06:55
06:55 um it just basically played against 06:56
06:56 itself. Started at random and then you 06:59
06:59 know around a dozen to two dozen 07:01
07:01 different epochs. You know it play like 07:03
07:03 a million games against itself. Uh it 07:05
07:05 would then train a new version of itself 07:08
07:08 uh uh that was slightly better than the 07:10
07:10 old version and then that would play a 07:12
07:12 million games against itself and so on 07:14
07:14 and and for example and then it could 07:15
07:15 play any game not just go it could play 07:18
07:18 any two-player perfect information game. 07:20
07:20 And for me it was especially visceral as 07:22
07:22 a chess player because I remember 07:24
07:24 watching it play chess and within um you 07:28
07:28 know 12 hours basically or less than 12 07:31
07:31 hours it would start from random play in 07:33
07:33 the morning to you know you'd have your 07:35
07:35 coffee break at 11:00 a.m. and I could 07:37
07:37 still just about beat it. Then by 07:39
07:39 lunchtime it was grandmaster level and 07:41
07:41 then by by tea time it was better than 07:44
07:44 um the any any chess playing entity. 07:46
07:46 time is dinner time for the US computer 07:48
07:48 time to you know or computer you know 07:51
07:51 kind of handcrafted chess computer is 07:53
07:53 better than all of them so we sort of 07:56
07:56 saw the power of that recursive 07:58
07:58 self-improvement and obviously the thing 07:60
07:60 that all these domains have in common is 08:01
08:01 you can verify the answer is it cor is 08:04
08:04 it a correct answer is it an optimal 08:05
08:05 answer is the equation correct it does 08:07
08:07 the code run it did you win the game and 08:10
08:10 the important thing about a verifiable 08:11
08:11 domain is you can verify the answer um 08:13
08:13 so you know when you've you've you 08:15
08:15 you've you've hit the by point. Um, but 08:17
08:17 also that means you can create almost 08:19
08:19 infinite synthetic data because you can 08:22
08:22 just use the current best versions you 08:24
08:24 have to set new problems or make new 08:26
08:26 moves and then um uh just sort of verify 08:30
08:30 uh uh whether the synthetic data was 08:32
08:32 correct. The problem with synthetic data 08:34
08:34 a lot of the time is you can't verify 08:35
08:35 that it's right um from the right 08:37
08:37 distribution but in these cases you can. 08:40
08:40 So we've seen the enormous progress with 08:42
08:42 maths and with um coding and coding's 08:45
08:45 obviously become enormously valuable and 08:47
08:47 everyone is having to do that now. But I 08:50
08:50 do worry about it because in the limit 08:52
08:52 so there's different forms of RSI right 08:54
08:54 now. We're in the the early days of RSI 08:57
08:57 I would say where what's happening is 08:59
08:59 and you're probably all experiencing 09:00
09:00 this as well with your teams is um you 09:03
09:03 know it's making a human engineer maybe 09:06
09:06 three, four, five times more productive. 09:09
09:09 Okay. So, uh but the but the human is 09:11
09:11 very much still in the loop and 09:13
09:13 reviewing code and architecting things 09:15
09:15 and um and and and writing code still. 09:18
09:18 Um but it's all just enhanced. Um the 09:21
09:21 next phase which I think carries the 09:23
09:23 most risk is closed RSI closed loop 09:26
09:26 where you basically manage to somehow 09:28
09:28 figure out how to do it without the 09:29
09:29 human reviewer process uh in this in the 09:32
09:32 in the whole procedure and that um will 09:35
09:35 obviously be faster but I think is also 09:37
09:37 a very dangerous way to get to full AGI 09:40
09:40 because we're likely for two reasons 09:42
09:42 it's it's sometimes used to be referred 09:44
09:44 to 20 years ago as like an intelligence 09:46
09:46 explosion because you don't quite know 09:48
09:48 uh how you know how do you control keep 09:51
09:51 control of such a system uh such a 09:53
09:53 process and then the end result will 09:56
09:56 likely be quite alien. It's going to be 09:57
09:57 like an alien intelligence. There's no 09:59
09:59 reason you to think it would be sort of 10:01
10:01 in the similar kind of mind space that 10:04
10:04 uh humans are in. So that will make it 10:06
10:06 very hard for it to understand for us to 10:08
10:08 understand it and maybe for it to 10:09
10:09 understand us. So I think it's you know 10:12
10:12 we should be very cautious as we get 10:14
10:14 closer towards that closed loop form of 10:16
10:16 RSI and I think all the AI researchers 10:19
10:19 sort of know that but there is a sort of 10:21
10:21 race to the there's this sort of intense 10:23
10:23 obviously uh uh uh competitive race to 10:27
10:27 improve your coding systems. So presume 10:29
10:29 that the the incentives to be in a race 10:33
10:33 continue and you know you and I some 10:35
10:35 people really know this you know have 10:37
10:37 tried to figure out if there's various 10:38
10:38 ways you can get global collaboration to 10:40
10:40 you've dedicated a huge amount of energy 10:42
10:42 to that so thank you um uh it's a hard 10:45
10:45 game as you know but presume that the 10:47
10:47 competitive pressure for continuing to 10:50
10:50 do RSI and coding continues 10:53
10:53 >> in that environment what are the best 10:54
10:54 ways that we can potentially navigate 10:58
10:58 that because we have to see kind of 10:59
10:59 where RSI is going, what it's pointed 11:01
11:01 at. What are the kinds of things that 11:03
11:03 the, you know, the leading labs such as 11:05
11:05 your own and the others, 11:07
11:07 >> what should they be doing to try to 11:09
11:09 navigate this environment? Well, what 11:11
11:11 we're doing and I think other leading 11:12
11:12 labs are trying to do is you got to uh 11:15
11:15 so there's there's more prosaic things 11:17
11:17 like make sure uh uh cyber defense is 11:21
11:21 accelerating faster than cyber attack 11:22
11:22 and making sure you give access to 11:24
11:24 trusted cyber defenders first on on 11:27
11:27 maybe slightly fine-tuned versions of 11:29
11:29 your general models that are fine-tuned 11:31
11:31 for cyber defense. Um and so you keep 11:34
11:34 the defenders ahead of the attackers. 11:36
11:36 It's a pretty tricky problem to do in 11:37
11:37 general. um also building much more 11:40
11:40 hardened sandboxes. I think a lot of the 11:43
11:43 uh cyber kind of issues that we've seen 11:46
11:46 in the last six months or so could have 11:48
11:48 been addressed with better engineering, 11:50
11:50 more robust engineering. I think better 11:52
11:52 you need really amazing monitoring 11:53
11:53 systems. You know, they shouldn't be 11:55
11:55 possible for agents to be running for 11:56
11:56 for weeks without anyone realizing. You 11:59
11:59 know, these kinds of things that we've 12:00
12:00 seen some of our uh some of our 12:02
12:02 competitors and others maybe, you know, 12:04
12:04 it's just the beginning of this 12:05
12:05 happening. So I think everyone will get 12:07
12:07 rapidly better but they have to as well 12:09
12:09 because otherwise something bad's going 12:11
12:11 to happen inadvertently. Some kind of 12:13
12:13 incident you know I sometimes call it 12:14
12:14 like an oil spill is going to happen on 12:17
12:17 the current trajectory if we're not if 12:19
12:19 we don't get uh collectively if if 12:21
12:21 everyone doesn't pay a lot more 12:22
12:22 attention and put a lot more resources 12:24
12:24 on these types of systems eventually you 12:27
12:27 kind of want formal guarantees maybe 12:29
12:29 some bounds around what these systems 12:31
12:31 can do uh test how robust like the 12:33
12:33 things like the guardrails are. This is 12:35
12:35 actually something I encourage a lot of 12:36
12:36 top mathematicians that I know well who 12:38
12:38 are maybe, you know, pretty sad about 12:40
12:40 the state of pure maths, but they their 12:43
12:43 skills would be incredible uh if they 12:45
12:45 pivoted to these types of domains, more 12:48
12:48 like complexity theory, game theory. How 12:50
12:50 do you uh and of course when they're 12:52
12:52 going to have to start dealing with 12:53
12:53 swarms of these agents, potentially 12:55
12:55 collaborating in some way. Um how does 12:57
12:57 one keep track of all of that? And 12:59
12:59 should there be ways that the labs are 13:02
13:02 encouraged, required, permitted to 13:05
13:05 collaborate on this? And if so, what 13:07
13:07 should those what should some of those 13:09
13:09 ways be? Because it obviously gets 13:11
13:11 stronger if there is sharing of 13:13
13:13 intelligence, methodology. Yeah. 13:15
13:15 >> You know, mutual accountability. 13:16
13:16 >> Yeah. I mean, there's there's been lots 13:17
13:17 of proposals and I've been working on 13:19
13:19 these things a lot. And you you you know 13:20
13:20 you saw my essay over the summer, maybe 13:22
13:22 some of you read it about potential for 13:25
13:25 like an SRO industry run standards body 13:29
13:29 with some kind of government uh 13:31
13:31 oversight over it. Um I think something 13:33
13:33 like that is going to be needed and then 13:35
13:35 ideally it becomes international. You 13:37
13:37 know it's USled and then obviously the 13:39
13:39 state of geopolitics today makes it and 13:41
13:41 and our international institutions are 13:42
13:42 very very weak as we all know uh just at 13:45
13:45 the time when we need them to be 13:46
13:46 strongest in in a way. So that's a bit 13:49
13:49 unfortunate, but we got to start 13:50
13:50 somewhere. 13:51
13:51 >> By the way, British a bit unfortunate is 13:53
13:53 holy. Just for translation. 13:57
13:57 Yeah. It's like tea time and dinner 13:58
13:58 time. Yes. Exactly. A bit unfortunate. 14:01
14:01 and um 14:04
14:04 and and so we're trying to you know 14:05
14:05 various of us are trying to do things 14:07
14:07 about that and and that essay did have 14:09
14:09 an impact and uh you know in the white 14:11
14:11 house and in administration and and with 14:13
14:13 our colleagues and eventually you know I 14:16
14:16 think some of that energy became the 14:18
14:18 accord that you the famous letter 14:19
14:19 everyone signed which is which is better 14:21
14:21 than nothing. I don't mean goes far 14:23
14:23 enough but it's a start and at least it 14:25
14:25 acknowledges from all of the main 14:26
14:26 players uh uh the issues that are there 14:29
14:29 and and that they need to be addressed. 14:31
14:31 doesn't say too much about what to do 14:32
14:32 about them but you know that has to come 14:34
14:34 next and at least it gives air cover I 14:36
14:36 think for the teams to work together on 14:39
14:39 you know what sorts of benchmarks what 14:40
14:40 are safe best practices on safety what 14:42
14:42 are you seeing you know uh uh in terms 14:44
14:44 of analysis of what things could happen 14:47
14:47 with these types of systems and I do 14:49
14:49 think there are some things that 14:49
14:49 generally all the leaning labs do agree 14:51
14:51 on like um the idea of third party 14:54
14:54 auditors or evaluators coming in 14:56
14:56 obviously there's a question mark which 14:58
14:58 third parties and and I think we need a 15:00
15:00 healthy kind of ecosystem of that that 15:02
15:02 actually you know you could have good 15:05
15:05 startups and companies doing that that 15:07
15:07 that that there's enough revenue there 15:08
15:08 for them and there should be given the 15:10
15:10 amount of investment going into AI as a 15:12
15:12 whole bit like there are auditors you 15:13
15:13 know ini in the finance industry or any 15:15
15:15 of these industries so I could see that 15:17
15:17 happening um and some of the big players 15:19
15:19 are moving into this like Accenture 15:21
15:21 we're working in other leaders are 15:23
15:23 working with I think they're a little 15:24
15:24 bit ahead of the game because they've 15:25
15:25 acquired you know a good startup good 15:26
15:26 friend of ours Mark Warner startup so 15:28
15:28 they have some expertise internally but 15:31
15:31 I think we need more of that and then 15:32
15:32 other things like you know the leading 15:34
15:34 labs maybe red teaming each other uh 15:36
15:36 before uh general release of these 15:39
15:39 systems 15:39
15:39 >> so let's move from that to a little bit 15:41
15:41 more of the fantastical you know it 15:43
15:43 started actually a few years back I 15:45
15:45 think there was a Google engineer who 15:46
15:46 said I asked Gemini if it was conscious 15:48
15:48 and it said yes so therefore it's 15:49
15:49 conscious which seemed like a little bit 15:51
15:51 of a a weak inference that's now gaining 15:55
15:55 some motion with some of the you know 15:57
15:57 things especially anthropic is doing and 15:60
15:60 and in and the way that it u kind of 16:02
16:02 strategizes thinking that well at some 16:03
16:03 point it's almost like a Pascal's wager 16:05
16:05 at some point there will be 16:07
16:07 consciousness in these systems and we 16:08
16:08 want want the those conscious systems to 16:10
16:10 look back saying that we were treating 16:12
16:12 them the way the same way that we like 16:14
16:14 treat a 2-year-old um which is a hey 16:16
16:16 look we were good to you as a 2-year-old 16:18
16:18 and so you know it kind of continues 16:20
16:20 through 16:21
16:21 >> obviously to many people that looks kind 16:23
16:23 of wacky right and um and so what do you 16:28
16:28 think I mean cuz Part of what I think 16:29
16:29 maybe many people don't realize is you 16:31
16:31 also come to this from a depth of 16:32
16:32 expertise in brain science 16:34
16:34 >> and you know one of the things is not 16:36
16:36 just a oh did the chatbot say it was 16:38
16:38 conscious or not is not the way to make 16:40
16:40 a decision on this kind of thing. 16:41
16:41 >> What do you think some of the ways to 16:42
16:42 think about this is and also how we 16:44
16:44 should be deliberately trying to shape 16:46
16:46 it. 16:46
16:46 >> Yeah. Well look I think there's quite a 16:48
16:48 few things to say about this. First of 16:49
16:49 all um we haven't despite thousands of 16:53
16:53 years of thinking about this there isn't 16:54
16:54 a agreed upon definition of 16:56
16:56 consciousness. So that's one one issue I 16:58
16:58 think and that's one issue with labeling 17:00
17:00 things conscious um especially the hard 17:02
17:02 problem of consciousness you know is it 17:04
17:04 really feeling this internally like the 17:06
17:06 way we we seem to you know behaviorally 17:08
17:08 these things I don't know about you but 17:10
17:10 yes exactly but the behaviorally well 17:11
17:11 but I used to argue with Daniel Dennett 17:13
17:13 about this that there's an additional 17:14
17:14 thing is we're not just behaving in ways 17:16
17:16 like we'd imagine conscious beings 17:18
17:18 behave but we also share the same 17:20
17:20 substrate we have the same squishy you 17:22
17:22 know brains and so then it's most 17:24
17:24 parsimmonious to think you're feeling 17:25
17:25 what I'm feeling But of course with a 17:28
17:28 with an AI that's not the case. You 17:29
17:29 don't have the second the substrate 17:31
17:31 equivalent. So I think it's going to be 17:33
17:33 very hard to conclude that. I think 17:35
17:35 there are dangers. Look there's there's 17:36
17:36 two reasons why I think we should be 17:38
17:38 focusing on building intelligent tools 17:39
17:39 today not uh conscious entities. Um I 17:43
17:43 think for one thing is a safety argument 17:45
17:45 and uh technical uh safety and risk 17:48
17:48 which is that you know there's a direct 17:50
17:50 trade-off between creating something 17:53
17:53 let's say uh with selfition right or 17:56
17:56 self-awareness um you know even prior to 17:59
17:59 consciousness these are the 17:60
17:60 prerequisites right to a conscious 18:02
18:02 entity um that it will and that and 18:05
18:05 instruction following or sometimes 18:07
18:07 called courageability. So this idea of a 18:10
18:10 system actually following instructions 18:12
18:12 and if it's going wrong you telling it 18:14
18:14 you know you as the user or the the 18:16
18:16 human user telling it like stop doing 18:17
18:17 that right or switch yourself off or 18:20
18:20 whatever it is that you're going to you 18:21
18:21 know change change course. So if you are 18:23
18:23 training your system to be able to 18:26
18:26 reject commands based on its own 18:28
18:28 internal uh workings, then you're 18:31
18:31 setting up a situation where if you've 18:34
18:34 got your constitution wrong or your 18:36
18:36 rules wrong, and this is what Asimoff 18:37
18:37 was always warning about with these 18:38
18:38 three laws, you know, that was a very 18:40
18:40 simple form of a constitution. And his 18:43
18:43 books are all warnings that that's 18:45
18:45 you're never going to get that right in 18:47
18:47 every corner case, in every edge case. 18:49
18:49 And and and and I think his books show 18:51
18:51 that very well. And yet we're sort of I 18:54
18:54 think a bit hubristically thinking that 18:55
18:55 that some people thinking that you know 18:57
18:57 sometimes called the misalignment 18:59
18:59 problem is sort of under control just 19:01
19:01 because today's systems are you know the 19:03
19:03 good news is they seem more malleable 19:05
19:05 than we might have imagined 15 20 years 19:07
19:07 ago which is great news in terms of 19:09
19:09 their steerability but that doesn't mean 19:11
19:11 we've solved um the misalignment problem 19:13
19:13 in total as we get closer to you know a 19:16
19:16 GI and super intelligence and so on. And 19:18
19:18 I just think um it's it's it's dangerous 19:21
19:21 to think one has. So that's the first 19:23
19:23 thing. There's a technical problem 19:24
19:24 especially if you combine it with 19:25
19:25 something like an RSI. 19:26
19:26 >> Yes. Exactly. 19:29
19:29 Right. So RSI with self self valition is 19:32
19:32 you know sort of doubly dangerous. So 19:35
19:35 that's one thing. Second thing is a 19:37
19:37 societal more of a societal issue which 19:39
19:39 is that I don't think we are ready yet 19:41
19:41 as a society. We haven't had the debate 19:43
19:43 with a wider society, philosophers, 19:45
19:45 theologians, um, artists, humanities, 19:48
19:48 not just technologists as to, you know, 19:51
19:51 and from many different cultures. Like, 19:53
19:53 do we want to cross that second rubicon? 19:55
19:55 Are we ready to have the discussion 19:57
19:57 about do these systems have moral 19:59
19:59 standing? Um, do we need to worry about, 20:02
20:02 you know, what are we doing with them in 20:03
20:03 this box? You know, are they slaves of 20:05
20:05 some sort? you know, all these questions 20:07
20:07 come up that are complex questions and I 20:10
20:10 don't think we're equipped to even begin 20:12
20:12 to start dealing with those questions. 20:14
20:14 So, my view is we should um cross the 20:17
20:17 first Rubicon uh on its own which is 20:19
20:19 build these intelligent tools and then 20:21
20:21 actually use these intelligent tools to 20:23
20:23 do more neuroscience to go back I'd go 20:26
20:26 back and do more neuroscience to 20:28
20:28 actually understand what the neural 20:29
20:29 correlates of consciousness really are. 20:31
20:31 finally understand that and then you 20:34
20:34 know that'll probably take 5 10 years 20:35
20:35 and then cross the second Rubicon well 20:38
20:38 to have a discussion about should we 20:40
20:40 cross the second Rubicon and then if so 20:42
20:42 how right of of the of the conscious 20:44
20:44 entity type 20:45
20:45 >> and how would you cuz you know you 20:47
20:47 engaged in hours of these discussions 20:49
20:49 how would you answer the folks who say 20:50
20:50 look the only way we get the intelligent 20:52
20:52 coralate to be powerful enough is it it 20:55
20:55 has as it were the ephemeral accidental 20:57
20:57 it actually has this you know cognitive 20:59
20:59 state and so 20:59
20:59 >> right so we should talk about this a We 21:01
21:01 discussed this you know even 10 plus 21:03
21:03 years ago and I discussed it in my PhD 21:05
21:05 back 20 plus years ago of look it I felt 21:08
21:08 and I still feel intelligence and 21:10
21:10 consciousness are double dissociable 21:12
21:12 which means that you can have an 21:13
21:13 intelligence system that's very smart on 21:15
21:15 IQ and so on kind of like the systems we 21:17
21:17 have today which I don't I personally 21:19
21:19 don't feel are at all conscious yet 21:21
21:21 right from my own you know everyone 21:23
21:23 probably has their own view but I don't 21:24
21:24 feel yet and then you can have conscious 21:27
21:27 entities uh that are not that smart for 21:30
21:30 example my pet dog. He's lovely, but 21:32
21:32 he's and he's pretty conscious. I think 21:34
21:34 he dreams and everything, but he's not 21:35
21:35 that smart, 21:36
21:36 >> right? 21:36
21:36 >> And that's provable. 21:37
21:37 >> So, I think you can have double 21:39
21:39 dissociation here. You know, many of the 21:41
21:41 higher animals, dolphins, monkeys, and 21:43
21:43 so on, elephants, I think most of us 21:45
21:45 would view their conscious beings, but 21:48
21:48 and they exhibit all of that that sort 21:49
21:49 of behavior. And of of course, they're 21:51
21:51 also made of carbons and and the same 21:53
21:53 neurons and types of things we're made 21:55
21:55 of. So, I feel like it's pretty obvious 21:57
21:57 they have some of those qualities. um 21:59
21:59 but they're not necessarily that smart. 22:01
22:01 So it feels like you can dissociate 22:02
22:02 these things and and I think to the 22:05
22:05 extent that you can I feel like we 22:06
22:06 should 22:07
22:07 >> you know we're coming up close to the 22:08
22:08 end of our session and the end of you 22:11
22:11 know the summit sad um what should we 22:15
22:15 think about in terms of the whole notion 22:17
22:17 of AI sovereignty 22:18
22:18 >> right because obviously one of the one 22:20
22:20 of the things that is kind of going 22:22
22:22 around in a lot of countries right now 22:24
22:24 is oh is this really important and you 22:26
22:26 know one of our earliest conversations 22:28
22:28 was with you trying to solve this 22:30
22:30 problem so I wanted 22:31
22:31 have share some thinking and obviously 22:33
22:33 it's updated for whatever that was 13 14 22:36
22:36 years ago but 22:36
22:36 >> well look I think first of all first 22:38
22:38 thing to say is it looks to me like 22:40
22:40 there's going to be you know half a 22:42
22:42 dozen you know six to eight if you 22:43
22:43 include Chinese companies company you 22:46
22:46 know offerings at the frontier near the 22:48
22:48 frontier and it feels like there's going 22:50
22:50 to be leaprogging and maybe something's 22:52
22:52 like three to six months ahead but it's 22:54
22:54 going to be uh relatively close so 22:55
22:55 there's a lot of options if depends if 22:57
22:57 you're talking about an enterprise or a 22:59
22:59 country right So I think as an 23:01
23:01 enterprise you have a lot of options to 23:03
23:03 arbitrage between these different 23:05
23:05 things. Um some of them are better at 23:07
23:07 certain things you know multimodal 23:08
23:08 understanding versus coding or it's very 23:11
23:11 hard to be better on all vectors of 23:12
23:12 importance at once. So I think that's 23:14
23:14 going to happen and then open source um 23:17
23:17 which also has its risks because how do 23:19
23:19 you as we've discussed for many many a 23:20
23:20 year is how do you restrict um bad actor 23:23
23:23 using that whilst enabling of course as 23:25
23:25 many good actors scientists and others 23:27
23:27 uh to use uh these fantastic tools. So 23:30
23:30 that's an unresolved problem but open 23:32
23:32 source is you know maybe only lagging 6 23:34
23:34 months to a year behind. Uh obviously we 23:36
23:36 need kind of western open source 23:38
23:38 solutions. Um but I think at this point 23:40
23:40 if we're talking about countries as well 23:42
23:42 all companies that need their data to be 23:44
23:44 private or they want to fine-tune 23:46
23:46 further on their own you know sensitive 23:48
23:48 data again be that enterprise data or or 23:50
23:50 sovereign data. I think that's that's 23:52
23:52 going to be one solution. There's also 23:55
23:55 you know looking at what are the 23:56
23:56 downstream things that can be done with 23:58
23:58 these tools you know big companies and 24:00
24:00 big ve sectors that can benefit from 24:03
24:03 these tools like healthcare that you can 24:05
24:05 just use this you know almost like 24:07
24:07 commodities and then build the value on 24:10
24:10 top of it rather than you know I 24:12
24:12 wouldn't recommend every country for 24:13
24:13 example trying to build a a frontier 24:15
24:15 model that just doesn't make sense to me 24:17
24:17 so we're going to they're going to have 24:18
24:18 to find some other way um of 24:19
24:19 shortcutting that 24:20
24:20 >> well and also collaborating and uh 24:23
24:23 working about arrangements with 24:24
24:24 multinational companies like your own. I 24:26
24:26 think one of the reasons I wanted to 24:28
24:28 kind of end on that question is that 24:30
24:30 part of the theme of of the summit is 24:32
24:32 agency and agency is not just a question 24:35
24:35 of individual agency like how do you 24:37
24:37 take agency but it's also a collective 24:39
24:39 how do we take agency 24:41
24:41 >> and I think part of the thing that's 24:42
24:42 actually important in the AI age is that 24:45
24:45 we take agency not just as any one 24:47
24:47 country but as countries together it's 24:49
24:49 part of how we have a we are in this 24:51
24:51 together and we are we are building for 24:54
24:54 the benefit of humanity and I think 24:55
24:55 that's the kind arc of we started with 24:57
24:57 Sarah Frier and thank you for joining us 24:59
24:59 because Demis has been at this with such 25:02
25:02 intensity that I was so glad grateful 25:04
25:04 that you set up the agenda for me 25:06
25:06 thinking about how to help in this from 25:08
25:08 our very first you know discussion at 25:10
25:10 King's Cross. So uh let's give Demis a 25:13
25:13 hand. Thank you. Thank you.