Masters of Scale Summit 26 · Demis Hassabis

00:01 I think a lot of the cyber issues that
00:03 

00:03 we've seen in the last six months or so
00:06 

00:06 could have been addressed with better
00:07 

00:07 engineering, more robust engineering.
00:09 

00:09 The next phase which I think carries the
00:11 

00:11 most risk is closed RSI, closed loop
00:14 

00:14 without the human reviewer process and
00:16 

00:16 that will obviously be faster but I
00:18 

00:18 think is also a very dangerous way to
00:19 

00:19 get to full AGI and our international
00:22 

00:22 institutions are very very weak as we
00:23 

00:23 all know just at the time when we need
00:25 

00:25 them to be strongest in in a way. So
00:27 

00:27 that's a bit unfortunate. But
00:28 

00:28 >> by the way, British a bit unfortunate is
00:31 

00:31 holy. Just for translate to translate
00:34 

00:34 another translation. Yeah. It's like tea
00:36 

00:36 time and dinner time. Exactly. A bit
00:38 

00:38 unfortunate.
00:42 

00:42 >> Please welcome Dennis Hades and Reed
00:44 

00:44 Hoffman.
00:50 

00:50 >> Dennis, thank you for joining us.
00:51 

00:51 >> Great to be here. Right.
00:52 

00:52 >> Thanks for having me.
00:53 

00:53 >> We've known each other for I think well
00:55 

00:55 over 10 years now. maybe approaching 12
00:57 

00:57 or something like that. And uh it was an
00:60 

00:60 honor when you asked me to be on the AI
01:03 

01:03 ethics board for Google for you know for
01:05 

01:05 our various funny
01:06 

01:06 >> meetings. Uh yes and it's been great. So
01:10 

01:10 one of the things we've been trying to
01:11 

01:11 do uh at the summit is to be envisioning
01:16 

01:16 the world that we as we'd want to make
01:17 

01:17 it. The theme has been agency and you
01:20 

01:20 know as you have been a leader in and
01:22 

01:22 knew well before most people were
01:24 

01:24 thinking about AI that one of the key
01:25 

01:25 things for thinking about AI for human
01:27 

01:27 benefit is medicine and medical. say a
01:30 

01:30 little bit about how you envision this,
01:32 

01:32 how this helps,
01:34 

01:34 >> you know, people say, "Oh, if I could be
01:36 

01:36 getting that, I understand why you AI
01:40 

01:40 guys are actually trying to build
01:41 

01:41 something that creates
01:43 

01:43 >> a longer human experience, a more
01:46 

01:46 quality of experience, something that's
01:47 

01:47 not it's you call it anti-exrisk."
01:49 

01:49 >> Yeah, for sure. I mean if we if we take
01:51 

01:51 a step back for a moment is like um big
01:54 

01:54 picture and and you know this having
01:56 

01:56 been sort of um knowing us from quite
01:58 

01:58 early in our deep mind journey that the
02:00 

02:00 the whole point of me working on AI for
02:03 

02:03 my whole career was to use it as a kind
02:05 

02:05 of like the ultimate tool to to advance
02:07 

02:07 science and and medicine and our
02:10 

02:10 understanding of the world around us but
02:12 

02:12 also improving human health. I've always
02:14 

02:14 said that the number one thing we should
02:15 

02:15 use these amazing tools that we've now
02:17 

02:17 built and we were, you know, dreaming
02:19 

02:19 about 20 years ago is to improve the
02:22 

02:22 human condition and improve human
02:23 

02:23 health. And I've tried to do that
02:25 

02:25 certainly with my own uh kind of more
02:28 

02:28 personal projects. uh obviously things
02:30 

02:30 like Alphafold um but also uh the the
02:33 

02:33 spinout drug discovery company
02:34 

02:34 Isomorphic um I've tried to push it
02:37 

02:37 really hard in that direction and as I
02:39 

02:39 know you have too more recently and I
02:42 

02:42 think it's exactly the sorts of things
02:44 

02:44 we need to be showing the world that AI
02:46 

02:46 can do. I think um you know what better
02:48 

02:48 thing than to you know use AI to help us
02:50 

02:50 cure terrible diseases. So some of the
02:52 

02:52 protein folding was this kind of
02:55 

02:55 intersection of the idea of like we have
02:57 

02:57 really good data. We could use scale
02:59 

02:59 learning because the whole learning like
03:01 

03:01 the the cognitive revolution from
03:03 

03:03 learning starting with even pong but the
03:07 

03:07 you know and selfplay and all the rest
03:08 

03:08 is like okay that's this would be a
03:09 

03:09 really good way of doing it. How does um
03:12 

03:12 like the kind of the alpha fold lead to
03:14 

03:14 medicines? Like what's part of the arc
03:16 

03:16 and and you know because one of the
03:17 

03:17 things is a question I get frequently is
03:18 

03:18 like yeah yeah you're talking about that
03:20 

03:20 but is it true? Is it going to happen
03:22 

03:22 anytime soon? You know D. So I'm you
03:24 

03:24 know I'm I'm passing along the question
03:26 

03:26 I get to you.
03:27 

03:27 >> Yes. Sure. Well look biology and um and
03:30 

03:30 is the most complex you know systems in
03:33 

03:33 the universe right? So that's why it's
03:35 

03:35 so hard because of the emergent behavior
03:38 

03:38 um the interactions and proteins are the
03:42 

03:42 building blocks of life. That's how um
03:44 

03:44 I've always seen them and and the
03:46 

03:46 protein folding problem which is what we
03:48 

03:48 you know tried to solve with alphafold
03:50 

03:50 was considered to be like a 50-year kind
03:53 

03:53 of grand challenge in biology. It's
03:55 

03:55 sometimes described to me as as like
03:56 

03:56 Fermat's last theorem but for biology
03:59 

03:59 and it's critical to understand the
04:01 

04:01 structure of a protein from its genetic
04:03 

04:03 sequence it amino acid sequence to
04:05 

04:05 understand its function and that's
04:07 

04:07 really important to understand basic
04:08 

04:08 biological mechanisms but also to
04:11 

04:11 understand um you know what goes wrong
04:13 

04:13 in disease and what things to target you
04:16 

04:16 know if you imagine you're designing a
04:17 

04:17 drug then you need to know what the 3D
04:20 

04:20 surface of the protein looks like so you
04:22 

04:22 know which part of that protein uh to to
04:24 

04:24 bind into. So it was always a critical
04:27 

04:27 but not obviously only still a small
04:29 

04:29 part of the entire drug discovery
04:31 

04:31 process but it's absolutely critical
04:33 

04:33 part of that and we were able to you
04:36 

04:36 know with AlphaFold accelerate it you
04:38 

04:38 know by by orders of magnitude um um by
04:42 

04:42 building um AlphaFold and if you think
04:44 

04:44 about it we finished AlphaFold 2 back in
04:46 

04:46 2020 which is you know six plus years
04:49 

04:49 ago now and it's still probably the best
04:51 

04:51 example of concrete example of AI I
04:54 

04:54 applied to a really hard scientific
04:56 

04:56 problem uh that was thought to be sort
04:58 

04:58 of uncrackable. Um and so that just
05:00 

05:00 shows you how far ahead of its time it
05:02 

05:02 was. And now you know we're really proud
05:03 

05:03 of of the impact it's made. Like 4
05:05 

05:05 million researchers around the world use
05:07 

05:07 it. Pretty much every biology and
05:08 

05:08 medical researcher in the world almost
05:10 

05:10 every drug in future will have made use
05:13 

05:13 of alpha fold all the structures. But
05:15 

05:15 it's not enough. We want to expand it to
05:17 

05:17 biochemistry and um also predicting
05:21 

05:21 things like toxicity or adne properties,
05:23 

05:23 all of these other things that you need
05:24 

05:24 to know about before you can have a drug
05:27 

05:27 that's safe to use and that's efficacus,
05:30 

05:30 you know, efficient in the in in
05:32 

05:32 whatever it is you're trying to tackle.
05:33 

05:33 >> Y indeed.
05:38 

05:38 Um, so speaking of accelerations,
05:41 

05:41 >> uh, let's go to one of the hot topics of
05:43 

05:43 the day, uh, recursive self-improvement.
05:46 

05:46 >> Uh, what is your kind of theory of the
05:50 

05:50 game? And I say this because you, you
05:52 

05:52 know, are a master games person. So I
05:54 

05:54 use that as kind of deliberate. And if
05:55 

05:55 you don't know what I'm referring to,
05:57 

05:57 either look at the documentary or look
05:58 

05:58 it up on Wikipedia. Uh, so like RSI,
06:03 

06:03 opportunities, concerns.
06:04 

06:04 >> Yeah. Well, look, RSI uh it's obviously
06:08 

06:08 an extremely uh useful technique to make
06:12 

06:12 progress. It mostly works best in what I
06:15 

06:15 call verifiable domains, what's called
06:17 

06:17 verifiable viable domains. So, um math
06:20 

06:20 is one, which is why we're seeing so
06:22 

06:22 much advance there. Coding's another um
06:24 

06:24 but originally it was games. Actually,
06:26 

06:26 one of the first sort of famous
06:28 

06:28 recursive self-improving systems was one
06:30 

06:30 we invented quite a lot of years ago now
06:32 

06:32 called Alpha Zero, which was off the
06:34 

06:34 back built off the back of Alph Go,
06:36 

06:36 which was the kind of, you know, famous
06:38 

06:38 program that beat the world champion at
06:40 

06:40 Go back in 20, you know, 10 years ago
06:42 

06:42 now, 2016, and kind of was the
06:44 

06:44 beginning, I think, of the modern AI
06:46 

06:46 era. And then we built a second version
06:48 

06:48 that was in some ways more
06:49 

06:49 consequential, Alpha Zero, that didn't
06:52 

06:52 need to use any human data anymore. And
06:55 

06:55 um it just basically played against
06:56 

06:56 itself. Started at random and then you
06:59 

06:59 know around a dozen to two dozen
07:01 

07:01 different epochs. You know it play like
07:03 

07:03 a million games against itself. Uh it
07:05 

07:05 would then train a new version of itself
07:08 

07:08 uh uh that was slightly better than the
07:10 

07:10 old version and then that would play a
07:12 

07:12 million games against itself and so on
07:14 

07:14 and and for example and then it could
07:15 

07:15 play any game not just go it could play
07:18 

07:18 any two-player perfect information game.
07:20 

07:20 And for me it was especially visceral as
07:22 

07:22 a chess player because I remember
07:24 

07:24 watching it play chess and within um you
07:28 

07:28 know 12 hours basically or less than 12
07:31 

07:31 hours it would start from random play in
07:33 

07:33 the morning to you know you'd have your
07:35 

07:35 coffee break at 11:00 a.m. and I could
07:37 

07:37 still just about beat it. Then by
07:39 

07:39 lunchtime it was grandmaster level and
07:41 

07:41 then by by tea time it was better than
07:44 

07:44 um the any any chess playing entity.
07:46 

07:46 time is dinner time for the US computer
07:48 

07:48 time to you know or computer you know
07:51 

07:51 kind of handcrafted chess computer is
07:53 

07:53 better than all of them so we sort of
07:56 

07:56 saw the power of that recursive
07:58 

07:58 self-improvement and obviously the thing
07:60 

07:60 that all these domains have in common is
08:01 

08:01 you can verify the answer is it cor is
08:04 

08:04 it a correct answer is it an optimal
08:05 

08:05 answer is the equation correct it does
08:07 

08:07 the code run it did you win the game and
08:10 

08:10 the important thing about a verifiable
08:11 

08:11 domain is you can verify the answer um
08:13 

08:13 so you know when you've you've you
08:15 

08:15 you've you've hit the by point. Um, but
08:17 

08:17 also that means you can create almost
08:19 

08:19 infinite synthetic data because you can
08:22 

08:22 just use the current best versions you
08:24 

08:24 have to set new problems or make new
08:26 

08:26 moves and then um uh just sort of verify
08:30 

08:30 uh uh whether the synthetic data was
08:32 

08:32 correct. The problem with synthetic data
08:34 

08:34 a lot of the time is you can't verify
08:35 

08:35 that it's right um from the right
08:37 

08:37 distribution but in these cases you can.
08:40 

08:40 So we've seen the enormous progress with
08:42 

08:42 maths and with um coding and coding's
08:45 

08:45 obviously become enormously valuable and
08:47 

08:47 everyone is having to do that now. But I
08:50 

08:50 do worry about it because in the limit
08:52 

08:52 so there's different forms of RSI right
08:54 

08:54 now. We're in the the early days of RSI
08:57 

08:57 I would say where what's happening is
08:59 

08:59 and you're probably all experiencing
09:00 

09:00 this as well with your teams is um you
09:03 

09:03 know it's making a human engineer maybe
09:06 

09:06 three, four, five times more productive.
09:09 

09:09 Okay. So, uh but the but the human is
09:11 

09:11 very much still in the loop and
09:13 

09:13 reviewing code and architecting things
09:15 

09:15 and um and and and writing code still.
09:18 

09:18 Um but it's all just enhanced. Um the
09:21 

09:21 next phase which I think carries the
09:23 

09:23 most risk is closed RSI closed loop
09:26 

09:26 where you basically manage to somehow
09:28 

09:28 figure out how to do it without the
09:29 

09:29 human reviewer process uh in this in the
09:32 

09:32 in the whole procedure and that um will
09:35 

09:35 obviously be faster but I think is also
09:37 

09:37 a very dangerous way to get to full AGI
09:40 

09:40 because we're likely for two reasons
09:42 

09:42 it's it's sometimes used to be referred
09:44 

09:44 to 20 years ago as like an intelligence
09:46 

09:46 explosion because you don't quite know
09:48 

09:48 uh how you know how do you control keep
09:51 

09:51 control of such a system uh such a
09:53 

09:53 process and then the end result will
09:56 

09:56 likely be quite alien. It's going to be
09:57 

09:57 like an alien intelligence. There's no
09:59 

09:59 reason you to think it would be sort of
10:01 

10:01 in the similar kind of mind space that
10:04 

10:04 uh humans are in. So that will make it
10:06 

10:06 very hard for it to understand for us to
10:08 

10:08 understand it and maybe for it to
10:09 

10:09 understand us. So I think it's you know
10:12 

10:12 we should be very cautious as we get
10:14 

10:14 closer towards that closed loop form of
10:16 

10:16 RSI and I think all the AI researchers
10:19 

10:19 sort of know that but there is a sort of
10:21 

10:21 race to the there's this sort of intense
10:23 

10:23 obviously uh uh uh competitive race to
10:27 

10:27 improve your coding systems. So presume
10:29 

10:29 that the the incentives to be in a race
10:33 

10:33 continue and you know you and I some
10:35 

10:35 people really know this you know have
10:37 

10:37 tried to figure out if there's various
10:38 

10:38 ways you can get global collaboration to
10:40 

10:40 you've dedicated a huge amount of energy
10:42 

10:42 to that so thank you um uh it's a hard
10:45 

10:45 game as you know but presume that the
10:47 

10:47 competitive pressure for continuing to
10:50 

10:50 do RSI and coding continues
10:53 

10:53 >> in that environment what are the best
10:54 

10:54 ways that we can potentially navigate
10:58 

10:58 that because we have to see kind of
10:59 

10:59 where RSI is going, what it's pointed
11:01 

11:01 at. What are the kinds of things that
11:03 

11:03 the, you know, the leading labs such as
11:05 

11:05 your own and the others,
11:07 

11:07 >> what should they be doing to try to
11:09 

11:09 navigate this environment? Well, what
11:11 

11:11 we're doing and I think other leading
11:12 

11:12 labs are trying to do is you got to uh
11:15 

11:15 so there's there's more prosaic things
11:17 

11:17 like make sure uh uh cyber defense is
11:21 

11:21 accelerating faster than cyber attack
11:22 

11:22 and making sure you give access to
11:24 

11:24 trusted cyber defenders first on on
11:27 

11:27 maybe slightly fine-tuned versions of
11:29 

11:29 your general models that are fine-tuned
11:31 

11:31 for cyber defense. Um and so you keep
11:34 

11:34 the defenders ahead of the attackers.
11:36 

11:36 It's a pretty tricky problem to do in
11:37 

11:37 general. um also building much more
11:40 

11:40 hardened sandboxes. I think a lot of the
11:43 

11:43 uh cyber kind of issues that we've seen
11:46 

11:46 in the last six months or so could have
11:48 

11:48 been addressed with better engineering,
11:50 

11:50 more robust engineering. I think better
11:52 

11:52 you need really amazing monitoring
11:53 

11:53 systems. You know, they shouldn't be
11:55 

11:55 possible for agents to be running for
11:56 

11:56 for weeks without anyone realizing. You
11:59 

11:59 know, these kinds of things that we've
12:00 

12:00 seen some of our uh some of our
12:02 

12:02 competitors and others maybe, you know,
12:04 

12:04 it's just the beginning of this
12:05 

12:05 happening. So I think everyone will get
12:07 

12:07 rapidly better but they have to as well
12:09 

12:09 because otherwise something bad's going
12:11 

12:11 to happen inadvertently. Some kind of
12:13 

12:13 incident you know I sometimes call it
12:14 

12:14 like an oil spill is going to happen on
12:17 

12:17 the current trajectory if we're not if
12:19 

12:19 we don't get uh collectively if if
12:21 

12:21 everyone doesn't pay a lot more
12:22 

12:22 attention and put a lot more resources
12:24 

12:24 on these types of systems eventually you
12:27 

12:27 kind of want formal guarantees maybe
12:29 

12:29 some bounds around what these systems
12:31 

12:31 can do uh test how robust like the
12:33 

12:33 things like the guardrails are. This is
12:35 

12:35 actually something I encourage a lot of
12:36 

12:36 top mathematicians that I know well who
12:38 

12:38 are maybe, you know, pretty sad about
12:40 

12:40 the state of pure maths, but they their
12:43 

12:43 skills would be incredible uh if they
12:45 

12:45 pivoted to these types of domains, more
12:48 

12:48 like complexity theory, game theory. How
12:50 

12:50 do you uh and of course when they're
12:52 

12:52 going to have to start dealing with
12:53 

12:53 swarms of these agents, potentially
12:55 

12:55 collaborating in some way. Um how does
12:57 

12:57 one keep track of all of that? And
12:59 

12:59 should there be ways that the labs are
13:02 

13:02 encouraged, required, permitted to
13:05 

13:05 collaborate on this? And if so, what
13:07 

13:07 should those what should some of those
13:09 

13:09 ways be? Because it obviously gets
13:11 

13:11 stronger if there is sharing of
13:13 

13:13 intelligence, methodology. Yeah.
13:15 

13:15 >> You know, mutual accountability.
13:16 

13:16 >> Yeah. I mean, there's there's been lots
13:17 

13:17 of proposals and I've been working on
13:19 

13:19 these things a lot. And you you you know
13:20 

13:20 you saw my essay over the summer, maybe
13:22 

13:22 some of you read it about potential for
13:25 

13:25 like an SRO industry run standards body
13:29 

13:29 with some kind of government uh
13:31 

13:31 oversight over it. Um I think something
13:33 

13:33 like that is going to be needed and then
13:35 

13:35 ideally it becomes international. You
13:37 

13:37 know it's USled and then obviously the
13:39 

13:39 state of geopolitics today makes it and
13:41 

13:41 and our international institutions are
13:42 

13:42 very very weak as we all know uh just at
13:45 

13:45 the time when we need them to be
13:46 

13:46 strongest in in a way. So that's a bit
13:49 

13:49 unfortunate, but we got to start
13:50 

13:50 somewhere.
13:51 

13:51 >> By the way, British a bit unfortunate is
13:53 

13:53 holy. Just for translation.
13:57 

13:57 Yeah. It's like tea time and dinner
13:58 

13:58 time. Yes. Exactly. A bit unfortunate.
14:01 

14:01 and um
14:04 

14:04 and and so we're trying to you know
14:05 

14:05 various of us are trying to do things
14:07 

14:07 about that and and that essay did have
14:09 

14:09 an impact and uh you know in the white
14:11 

14:11 house and in administration and and with
14:13 

14:13 our colleagues and eventually you know I
14:16 

14:16 think some of that energy became the
14:18 

14:18 accord that you the famous letter
14:19 

14:19 everyone signed which is which is better
14:21 

14:21 than nothing. I don't mean goes far
14:23 

14:23 enough but it's a start and at least it
14:25 

14:25 acknowledges from all of the main
14:26 

14:26 players uh uh the issues that are there
14:29 

14:29 and and that they need to be addressed.
14:31 

14:31 doesn't say too much about what to do
14:32 

14:32 about them but you know that has to come
14:34 

14:34 next and at least it gives air cover I
14:36 

14:36 think for the teams to work together on
14:39 

14:39 you know what sorts of benchmarks what
14:40 

14:40 are safe best practices on safety what
14:42 

14:42 are you seeing you know uh uh in terms
14:44 

14:44 of analysis of what things could happen
14:47 

14:47 with these types of systems and I do
14:49 

14:49 think there are some things that
14:49 

14:49 generally all the leaning labs do agree
14:51 

14:51 on like um the idea of third party
14:54 

14:54 auditors or evaluators coming in
14:56 

14:56 obviously there's a question mark which
14:58 

14:58 third parties and and I think we need a
15:00 

15:00 healthy kind of ecosystem of that that
15:02 

15:02 actually you know you could have good
15:05 

15:05 startups and companies doing that that
15:07 

15:07 that that there's enough revenue there
15:08 

15:08 for them and there should be given the
15:10 

15:10 amount of investment going into AI as a
15:12 

15:12 whole bit like there are auditors you
15:13 

15:13 know ini in the finance industry or any
15:15 

15:15 of these industries so I could see that
15:17 

15:17 happening um and some of the big players
15:19 

15:19 are moving into this like Accenture
15:21 

15:21 we're working in other leaders are
15:23 

15:23 working with I think they're a little
15:24 

15:24 bit ahead of the game because they've
15:25 

15:25 acquired you know a good startup good
15:26 

15:26 friend of ours Mark Warner startup so
15:28 

15:28 they have some expertise internally but
15:31 

15:31 I think we need more of that and then
15:32 

15:32 other things like you know the leading
15:34 

15:34 labs maybe red teaming each other uh
15:36 

15:36 before uh general release of these
15:39 

15:39 systems
15:39 

15:39 >> so let's move from that to a little bit
15:41 

15:41 more of the fantastical you know it
15:43 

15:43 started actually a few years back I
15:45 

15:45 think there was a Google engineer who
15:46 

15:46 said I asked Gemini if it was conscious
15:48 

15:48 and it said yes so therefore it's
15:49 

15:49 conscious which seemed like a little bit
15:51 

15:51 of a a weak inference that's now gaining
15:55 

15:55 some motion with some of the you know
15:57 

15:57 things especially anthropic is doing and
15:60 

15:60 and in and the way that it u kind of
16:02 

16:02 strategizes thinking that well at some
16:03 

16:03 point it's almost like a Pascal's wager
16:05 

16:05 at some point there will be
16:07 

16:07 consciousness in these systems and we
16:08 

16:08 want want the those conscious systems to
16:10 

16:10 look back saying that we were treating
16:12 

16:12 them the way the same way that we like
16:14 

16:14 treat a 2-year-old um which is a hey
16:16 

16:16 look we were good to you as a 2-year-old
16:18 

16:18 and so you know it kind of continues
16:20 

16:20 through
16:21 

16:21 >> obviously to many people that looks kind
16:23 

16:23 of wacky right and um and so what do you
16:28 

16:28 think I mean cuz Part of what I think
16:29 

16:29 maybe many people don't realize is you
16:31 

16:31 also come to this from a depth of
16:32 

16:32 expertise in brain science
16:34 

16:34 >> and you know one of the things is not
16:36 

16:36 just a oh did the chatbot say it was
16:38 

16:38 conscious or not is not the way to make
16:40 

16:40 a decision on this kind of thing.
16:41 

16:41 >> What do you think some of the ways to
16:42 

16:42 think about this is and also how we
16:44 

16:44 should be deliberately trying to shape
16:46 

16:46 it.
16:46 

16:46 >> Yeah. Well look I think there's quite a
16:48 

16:48 few things to say about this. First of
16:49 

16:49 all um we haven't despite thousands of
16:53 

16:53 years of thinking about this there isn't
16:54 

16:54 a agreed upon definition of
16:56 

16:56 consciousness. So that's one one issue I
16:58 

16:58 think and that's one issue with labeling
17:00 

17:00 things conscious um especially the hard
17:02 

17:02 problem of consciousness you know is it
17:04 

17:04 really feeling this internally like the
17:06 

17:06 way we we seem to you know behaviorally
17:08 

17:08 these things I don't know about you but
17:10 

17:10 yes exactly but the behaviorally well
17:11 

17:11 but I used to argue with Daniel Dennett
17:13 

17:13 about this that there's an additional
17:14 

17:14 thing is we're not just behaving in ways
17:16 

17:16 like we'd imagine conscious beings
17:18 

17:18 behave but we also share the same
17:20 

17:20 substrate we have the same squishy you
17:22 

17:22 know brains and so then it's most
17:24 

17:24 parsimmonious to think you're feeling
17:25 

17:25 what I'm feeling But of course with a
17:28 

17:28 with an AI that's not the case. You
17:29 

17:29 don't have the second the substrate
17:31 

17:31 equivalent. So I think it's going to be
17:33 

17:33 very hard to conclude that. I think
17:35 

17:35 there are dangers. Look there's there's
17:36 

17:36 two reasons why I think we should be
17:38 

17:38 focusing on building intelligent tools
17:39 

17:39 today not uh conscious entities. Um I
17:43 

17:43 think for one thing is a safety argument
17:45 

17:45 and uh technical uh safety and risk
17:48 

17:48 which is that you know there's a direct
17:50 

17:50 trade-off between creating something
17:53 

17:53 let's say uh with selfition right or
17:56 

17:56 self-awareness um you know even prior to
17:59 

17:59 consciousness these are the
17:60 

17:60 prerequisites right to a conscious
18:02 

18:02 entity um that it will and that and
18:05 

18:05 instruction following or sometimes
18:07 

18:07 called courageability. So this idea of a
18:10 

18:10 system actually following instructions
18:12 

18:12 and if it's going wrong you telling it
18:14 

18:14 you know you as the user or the the
18:16 

18:16 human user telling it like stop doing
18:17 

18:17 that right or switch yourself off or
18:20 

18:20 whatever it is that you're going to you
18:21 

18:21 know change change course. So if you are
18:23 

18:23 training your system to be able to
18:26 

18:26 reject commands based on its own
18:28 

18:28 internal uh workings, then you're
18:31 

18:31 setting up a situation where if you've
18:34 

18:34 got your constitution wrong or your
18:36 

18:36 rules wrong, and this is what Asimoff
18:37 

18:37 was always warning about with these
18:38 

18:38 three laws, you know, that was a very
18:40 

18:40 simple form of a constitution. And his
18:43 

18:43 books are all warnings that that's
18:45 

18:45 you're never going to get that right in
18:47 

18:47 every corner case, in every edge case.
18:49 

18:49 And and and and I think his books show
18:51 

18:51 that very well. And yet we're sort of I
18:54 

18:54 think a bit hubristically thinking that
18:55 

18:55 that some people thinking that you know
18:57 

18:57 sometimes called the misalignment
18:59 

18:59 problem is sort of under control just
19:01 

19:01 because today's systems are you know the
19:03 

19:03 good news is they seem more malleable
19:05 

19:05 than we might have imagined 15 20 years
19:07 

19:07 ago which is great news in terms of
19:09 

19:09 their steerability but that doesn't mean
19:11 

19:11 we've solved um the misalignment problem
19:13 

19:13 in total as we get closer to you know a
19:16 

19:16 GI and super intelligence and so on. And
19:18 

19:18 I just think um it's it's it's dangerous
19:21 

19:21 to think one has. So that's the first
19:23 

19:23 thing. There's a technical problem
19:24 

19:24 especially if you combine it with
19:25 

19:25 something like an RSI.
19:26 

19:26 >> Yes. Exactly.
19:29 

19:29 Right. So RSI with self self valition is
19:32 

19:32 you know sort of doubly dangerous. So
19:35 

19:35 that's one thing. Second thing is a
19:37 

19:37 societal more of a societal issue which
19:39 

19:39 is that I don't think we are ready yet
19:41 

19:41 as a society. We haven't had the debate
19:43 

19:43 with a wider society, philosophers,
19:45 

19:45 theologians, um, artists, humanities,
19:48 

19:48 not just technologists as to, you know,
19:51 

19:51 and from many different cultures. Like,
19:53 

19:53 do we want to cross that second rubicon?
19:55 

19:55 Are we ready to have the discussion
19:57 

19:57 about do these systems have moral
19:59 

19:59 standing? Um, do we need to worry about,
20:02 

20:02 you know, what are we doing with them in
20:03 

20:03 this box? You know, are they slaves of
20:05 

20:05 some sort? you know, all these questions
20:07 

20:07 come up that are complex questions and I
20:10 

20:10 don't think we're equipped to even begin
20:12 

20:12 to start dealing with those questions.
20:14 

20:14 So, my view is we should um cross the
20:17 

20:17 first Rubicon uh on its own which is
20:19 

20:19 build these intelligent tools and then
20:21 

20:21 actually use these intelligent tools to
20:23 

20:23 do more neuroscience to go back I'd go
20:26 

20:26 back and do more neuroscience to
20:28 

20:28 actually understand what the neural
20:29 

20:29 correlates of consciousness really are.
20:31 

20:31 finally understand that and then you
20:34 

20:34 know that'll probably take 5 10 years
20:35 

20:35 and then cross the second Rubicon well
20:38 

20:38 to have a discussion about should we
20:40 

20:40 cross the second Rubicon and then if so
20:42 

20:42 how right of of the of the conscious
20:44 

20:44 entity type
20:45 

20:45 >> and how would you cuz you know you
20:47 

20:47 engaged in hours of these discussions
20:49 

20:49 how would you answer the folks who say
20:50 

20:50 look the only way we get the intelligent
20:52 

20:52 coralate to be powerful enough is it it
20:55 

20:55 has as it were the ephemeral accidental
20:57 

20:57 it actually has this you know cognitive
20:59 

20:59 state and so
20:59 

20:59 >> right so we should talk about this a We
21:01 

21:01 discussed this you know even 10 plus
21:03 

21:03 years ago and I discussed it in my PhD
21:05 

21:05 back 20 plus years ago of look it I felt
21:08 

21:08 and I still feel intelligence and
21:10 

21:10 consciousness are double dissociable
21:12 

21:12 which means that you can have an
21:13 

21:13 intelligence system that's very smart on
21:15 

21:15 IQ and so on kind of like the systems we
21:17 

21:17 have today which I don't I personally
21:19 

21:19 don't feel are at all conscious yet
21:21 

21:21 right from my own you know everyone
21:23 

21:23 probably has their own view but I don't
21:24 

21:24 feel yet and then you can have conscious
21:27 

21:27 entities uh that are not that smart for
21:30 

21:30 example my pet dog. He's lovely, but
21:32 

21:32 he's and he's pretty conscious. I think
21:34 

21:34 he dreams and everything, but he's not
21:35 

21:35 that smart,
21:36 

21:36 >> right?
21:36 

21:36 >> And that's provable.
21:37 

21:37 >> So, I think you can have double
21:39 

21:39 dissociation here. You know, many of the
21:41 

21:41 higher animals, dolphins, monkeys, and
21:43 

21:43 so on, elephants, I think most of us
21:45 

21:45 would view their conscious beings, but
21:48 

21:48 and they exhibit all of that that sort
21:49 

21:49 of behavior. And of of course, they're
21:51 

21:51 also made of carbons and and the same
21:53 

21:53 neurons and types of things we're made
21:55 

21:55 of. So, I feel like it's pretty obvious
21:57 

21:57 they have some of those qualities. um
21:59 

21:59 but they're not necessarily that smart.
22:01 

22:01 So it feels like you can dissociate
22:02 

22:02 these things and and I think to the
22:05 

22:05 extent that you can I feel like we
22:06 

22:06 should
22:07 

22:07 >> you know we're coming up close to the
22:08 

22:08 end of our session and the end of you
22:11 

22:11 know the summit sad um what should we
22:15 

22:15 think about in terms of the whole notion
22:17 

22:17 of AI sovereignty
22:18 

22:18 >> right because obviously one of the one
22:20 

22:20 of the things that is kind of going
22:22 

22:22 around in a lot of countries right now
22:24 

22:24 is oh is this really important and you
22:26 

22:26 know one of our earliest conversations
22:28 

22:28 was with you trying to solve this
22:30 

22:30 problem so I wanted
22:31 

22:31 have share some thinking and obviously
22:33 

22:33 it's updated for whatever that was 13 14
22:36 

22:36 years ago but
22:36 

22:36 >> well look I think first of all first
22:38 

22:38 thing to say is it looks to me like
22:40 

22:40 there's going to be you know half a
22:42 

22:42 dozen you know six to eight if you
22:43 

22:43 include Chinese companies company you
22:46 

22:46 know offerings at the frontier near the
22:48 

22:48 frontier and it feels like there's going
22:50 

22:50 to be leaprogging and maybe something's
22:52 

22:52 like three to six months ahead but it's
22:54 

22:54 going to be uh relatively close so
22:55 

22:55 there's a lot of options if depends if
22:57 

22:57 you're talking about an enterprise or a
22:59 

22:59 country right So I think as an
23:01 

23:01 enterprise you have a lot of options to
23:03 

23:03 arbitrage between these different
23:05 

23:05 things. Um some of them are better at
23:07 

23:07 certain things you know multimodal
23:08 

23:08 understanding versus coding or it's very
23:11 

23:11 hard to be better on all vectors of
23:12 

23:12 importance at once. So I think that's
23:14 

23:14 going to happen and then open source um
23:17 

23:17 which also has its risks because how do
23:19 

23:19 you as we've discussed for many many a
23:20 

23:20 year is how do you restrict um bad actor
23:23 

23:23 using that whilst enabling of course as
23:25 

23:25 many good actors scientists and others
23:27 

23:27 uh to use uh these fantastic tools. So
23:30 

23:30 that's an unresolved problem but open
23:32 

23:32 source is you know maybe only lagging 6
23:34 

23:34 months to a year behind. Uh obviously we
23:36 

23:36 need kind of western open source
23:38 

23:38 solutions. Um but I think at this point
23:40 

23:40 if we're talking about countries as well
23:42 

23:42 all companies that need their data to be
23:44 

23:44 private or they want to fine-tune
23:46 

23:46 further on their own you know sensitive
23:48 

23:48 data again be that enterprise data or or
23:50 

23:50 sovereign data. I think that's that's
23:52 

23:52 going to be one solution. There's also
23:55 

23:55 you know looking at what are the
23:56 

23:56 downstream things that can be done with
23:58 

23:58 these tools you know big companies and
24:00 

24:00 big ve sectors that can benefit from
24:03 

24:03 these tools like healthcare that you can
24:05 

24:05 just use this you know almost like
24:07 

24:07 commodities and then build the value on
24:10 

24:10 top of it rather than you know I
24:12 

24:12 wouldn't recommend every country for
24:13 

24:13 example trying to build a a frontier
24:15 

24:15 model that just doesn't make sense to me
24:17 

24:17 so we're going to they're going to have
24:18 

24:18 to find some other way um of
24:19 

24:19 shortcutting that
24:20 

24:20 >> well and also collaborating and uh
24:23 

24:23 working about arrangements with
24:24 

24:24 multinational companies like your own. I
24:26 

24:26 think one of the reasons I wanted to
24:28 

24:28 kind of end on that question is that
24:30 

24:30 part of the theme of of the summit is
24:32 

24:32 agency and agency is not just a question
24:35 

24:35 of individual agency like how do you
24:37 

24:37 take agency but it's also a collective
24:39 

24:39 how do we take agency
24:41 

24:41 >> and I think part of the thing that's
24:42 

24:42 actually important in the AI age is that
24:45 

24:45 we take agency not just as any one
24:47 

24:47 country but as countries together it's
24:49 

24:49 part of how we have a we are in this
24:51 

24:51 together and we are we are building for
24:54 

24:54 the benefit of humanity and I think
24:55 

24:55 that's the kind arc of we started with
24:57 

24:57 Sarah Frier and thank you for joining us
24:59 

24:59 because Demis has been at this with such
25:02 

25:02 intensity that I was so glad grateful
25:04 

25:04 that you set up the agenda for me
25:06 

25:06 thinking about how to help in this from
25:08 

25:08 our very first you know discussion at
25:10 

25:10 King's Cross. So uh let's give Demis a
25:13 

25:13 hand. Thank you. Thank you.