WEBVTT

1
00:00:00.300 --> 00:00:07.860
So yeah, this is a document which ca- which I first saw at 3:00 this morning, I think. It claims to be leaked from Google.

2
00:00:08.160 --> 00:00:15.980
There's good reasons to believe it is leaked from Google, and to be honest, if it's not, it doesn't actually matter because the quality of the analysis I think stands alone.

3
00:00:16.079 --> 00:00:26.420
If this was just a document by some anonymous person, I'd still think it was interesting and worth discussing. And the title of the document is "We Have No Moat and Neither Does OpenAI."

4
00:00:26.960 --> 00:00:41.520
And the argument it makes is that while Google and OpenAI have been competing on training bigger and bigger language models, the open source community's already starting to outrun them, given only a couple of months of really, like, h- really, really serious activity.

5
00:00:41.800 --> 00:00:49.620
You know, Facebook LLaMA was the thing that really kicked this off. There were open source language models like Bloom before that and GPTJ, and they weren't very impressive.

6
00:00:49.660 --> 00:01:04.900
Like, nobody was really thinking that they were ChatGPT equivalent. Facebook LLaMA came out in March, I think March 15th, and was the first one that really sort of showed signs of being as capable maybe as ChatGPT. My...

7
00:01:05.480 --> 00:01:09.960
I don't-- I think all of these models, they've been... The analysis of them has tended to be a bit hyped.

8
00:01:10.020 --> 00:01:17.440
Like, I don't think any of them are even quite up to GPT 3.5 standards yet, but they're within spitting distance in some respects.

9
00:01:17.780 --> 00:01:26.840
So anyway, LLaMA came out, and then two weeks later, Stanford Alpaca came out, which was fine-tuned on top of LLaMA and was a massive leap forward in terms of quality.

10
00:01:27.200 --> 00:01:35.480
And then a week after that, Vicuna came out, which is, to this date, the, the best model I've been able to run on my own hardware. And I've run it on my mobile phone now.

11
00:01:35.520 --> 00:01:46.620
Like, it's astonishing how little resources you need to run these things. But anyway, the, the argument this paper makes, which I found very convincing, is it only took open source two months to get this far.

12
00:01:47.320 --> 00:01:52.200
It's now every researcher in the world is kicking in on new, new things. But it feels like they're being...

13
00:01:52.240 --> 00:01:59.780
There, there are problems that Google have been trying to solve that the open source models are already addressing. And really, how do you compete with that? Like, with your...

14
00:01:59.900 --> 00:02:04.700
If closed ecosystem, how are you going to beat these open models with all of this innovation going on?

15
00:02:05.020 --> 00:02:11.680
But then the most interesting argument in there is it talks about the size of models and says that maybe large isn't a competitive advantage.

16
00:02:12.080 --> 00:02:33.540
Maybe actually a smaller model with lots of, like, different people fine-tuning it and having these sort of, these LoRA, L-O-R-A, stackable fine-tuning innovations on top of it, maybe those can move faster, and actually having to retrain your giant model every few months from scratch is, is way less useful than having small models that you can tr- you can fine-tune in a couple of hours on a laptop.

17
00:02:34.020 --> 00:02:42.200
So it's, it's fascinating. I tr- I... Basically, if you haven't read this thing, you should read every word of it. It's not very long. It's beautifully written. Like, it, it's...

18
00:02:42.360 --> 00:02:51.000
I mean, if you try and find the quotable lines in it, almost every line of it's quotable. Yeah. So yeah, that's, that, that, that's the status of this thing. That's a wonderful summary, Simon.

19
00:02:51.200 --> 00:02:53.680
Yeah, there, there's so many angles we can take to this.

20
00:02:53.740 --> 00:03:10.900
I'll, I'll just observe one more, one thing, which if you think about the open versus closed narrative, Emad Mostaque, who is CEO of Stability, has always been that open will trail behind closed because the closed alternatives can always take le- learnings and lessons from open source.

21
00:03:11.400 --> 00:03:19.860
And this is the first highly credible statement that is basically saying the exact opposite, that open source is moving than, than, than closed source, and they are scared. They seem to be scared,

22
00:03:21.160 --> 00:03:31.920
um, which is [chuckles] interesting. Travis, over to you. Yeah. The, the, the, a, a few things that, that I'll, I'll, I'll say. The only thing which can keep up with the pace of AI these days is open source.

23
00:03:32.360 --> 00:03:45.020
I think we're, we're seeing that unfold in real time before our eyes. And y- y- you know, I, I think the other interesting angle of this is, to some degree, LLMs are... They, they don't really have switching costs.

24
00:03:45.060 --> 00:03:50.200
They are going to be- become commoditized. At least that's, that's what a lot of, a lot of people kind of think.

25
00:03:50.520 --> 00:04:00.940
To, to what extent is it, i- i- is it a, a, a race in terms of, of pricing of these things and, you know, they all kind of become roughly the s- the, the same i- in terms of their, their underlying abilities.

26
00:04:01.240 --> 00:04:04.900
And, and open source is gonna, gonna be actively pushing, pushing that forward.

27
00:04:05.340 --> 00:04:14.180
And, and then this is kind of coming from, if it is to be believed, you know, the, the kind of Google or an insider type, type mentality around, you know, where is the actual competitive advantage?

28
00:04:14.280 --> 00:04:15.660
What should they be focusing on?

29
00:04:15.960 --> 00:04:28.760
How can they get back in, into the game, uh, when, you know, when, when, when, when currently the, the, the external view of, of Google is that they're kind of spinning their wheels and they have this code red and, you know, it's like they're, they're playing catch up already.

30
00:04:28.820 --> 00:04:38.820
Like, how, you know, could they use the open source community and work with them, which is gonna be really, really hard, you know, from a structural perspective given Google's place in the ecosystem.

31
00:04:38.900 --> 00:04:49.520
But a, a lot, lot of, lot of jumping off points there. I was gonna say, I think the post is really focused on how do we get the best model, but it's not focused on, like, how do you build the best product around it.

32
00:04:50.260 --> 00:04:54.200
A lot of these models are limited by how many GPUs you can get to run them, you know.

33
00:04:54.260 --> 00:05:05.560
And we've seen on traditional open source, like, everybody can use some of these projects like Kafka and like Elastic for free, but the reality is that not everybody can afford to run the infrastructure needed for it.

34
00:05:06.000 --> 00:05:17.160
So I, I think, like, the main takeaway that I have from this is, like, a lot of the moats are probably around just getting the, the sand, so to speak, and having the GPUs to actually serve these models.

35
00:05:17.500 --> 00:05:27.580
Because even if the best model is open source, like, running it at large scale for a net is not easy and, like, it's not super convenient to get a lot of the, a lot of the infrastructure.

36
00:05:28.280 --> 00:05:34.940
And we've seen that model work in open source where you have the open source project and then you have a enterprise cloud hosted version for it.

37
00:05:35.520 --> 00:05:47.600
I think that's gonna look really different in open source models because just hosting a model doesn't have a lot of value. So I'm curious to hear how people end up getting rewarded to do open source. You know? It's...

38
00:05:48.240 --> 00:05:51.980
We figured that out in infrastructure, but we haven't figured that out in, in LLMs yet.

39
00:05:53.280 --> 00:06:01.480
I mean, one thing I'll say is that the, the models that you can run on your own devices are so far ahead of what I ever dreamed they would be at this point.

40
00:06:01.780 --> 00:06:12.428
Like, Vicuna 13B I, I, I think is the- Current best available open mode model that I've played with, it's derived from Facebook LLaMA, so you can't use it for commercial purposes yet.

41
00:06:12.528 --> 00:06:18.028
But the point about Vicuna 13B is it runs in the browser directly on WebGPU.

42
00:06:18.088 --> 00:06:27.408
There's this amazing web LLM project where you literally, your browser downloads a two gigabyte file, and it fires up a ChatGPT style interface, and it's quite good.

43
00:06:27.528 --> 00:06:35.968
It can do rap battles between different animals, and all of the kinds of fun stuff that you'd expect to be able to do with a language model, running entirely in Chrome Canary.

44
00:06:36.288 --> 00:06:38.448
It's shocking to me that that's even possible.

45
00:06:38.788 --> 00:06:51.588
But that kind of shows that once, once you get to inference, if you can shrink the model down, and the techniques for shrinking these models, the, the first one was the, the quantization, which the LLaMA.CPP project really sort of popularized.

46
00:06:51.628 --> 00:06:57.448
And that can, you know, by using four-bit instead of 16-bit floating point numbers, you can shrink it down quite a lot.

47
00:06:57.748 --> 00:07:05.048
And then there was a paper that came out days ago suggesting that you can prune the models and ditch half the model and maintain the same level of quality.

48
00:07:05.708 --> 00:07:15.128
So with, with things like that, with all of these tricks coming together, it's really astonishing how much you can get done on hardware that people actually have in their pockets even.

49
00:07:15.408 --> 00:07:25.448
Just for completion, I've been following all of your posts. Oh, sorry. Yes. Huh? I just wanna follow up. S- Simon, you're, you said you're running a model on your phone. Which model is it?

50
00:07:25.488 --> 00:07:31.128
And I don't think you've written it up. Yeah, that one's Vicuna. I did... Did I write it up? I did.

51
00:07:31.228 --> 00:07:43.968
I've got a blog post about how it, it, it, it knows who I am, sort of, but it said that I invented a des- a, a pattern for living called the bear or bunny pattern, which I definitely didn't, but I love that my phone decided that I did.

52
00:07:44.368 --> 00:07:53.508
I will hunt for that because I'm not yet running Vicuna on my phone, and I feel like I should, and, and as like a very base thing. But I'll... Okay. I'll, I'll follow up two things, right?

53
00:07:53.568 --> 00:08:00.688
Like one, I'm very interested in, and I won't... Let's, let's talk about that a little bit more, because this concept of stackable improvements to models I think is extremely interesting.

54
00:08:01.028 --> 00:08:11.068
Like, I would love to NPM install a- a- abilities onto my models, right, which is really awesome. But the, the first thing I think is under-discussed is I don't get the panic.

55
00:08:11.268 --> 00:08:21.188
Like, honestly, like Google has the most moats. I, I, I was arguing maybe like three months ago on my blog, like Google has the most moat out of a lot of people because, hey, we have your calendar.

56
00:08:21.568 --> 00:08:29.768
Hey, we have your email. Hey, we have your, you know, Google Docs. Like, isn't that a, a sufficient moat? Like, why are these guys panicking so much? I don't, I still don't get it.

57
00:08:29.848 --> 00:08:38.428
Like sure, you know, open source is running ahead and like it's, it's on device and what have, what have you, but they have so much more moat. Like w- what are we talking about here?

58
00:08:39.048 --> 00:08:42.528
There's many dimensions to compete on. Yeah. There's...

59
00:08:42.648 --> 00:09:00.088
Like one of, one of the, the things that, that the author, you know, mentions in, in here is when, when you start to, to, to have the feeling of what we're, we're trailing behind, then your, your, your, your brightest researchers, you know, jump ship and go to OpenAI or go to work at, at, at academia or whatever.

60
00:09:00.128 --> 00:09:12.328
And, and like the talent drain at the, the level of the, the senior AI researchers that are pushing these things ahead within Google, I think is a serious, serious concern. And my, my take on... It's a good point, right?

61
00:09:12.388 --> 00:09:16.888
Like, like, like, like what Google has moats. They, they, they're not running out of money anytime soon. You know?

62
00:09:17.228 --> 00:09:26.688
I think they, they do see the level of the, the defensibility and, and the fact that they want to be a, a, a tri- the, the, the, the leader around pretty much anything tech first. Sure.

63
00:09:26.688 --> 00:09:31.308
Around pretty much anything tech first. Um- They've definitely ha- ha- have lost that, that, that feeling, right?

64
00:09:31.468 --> 00:09:41.428
And, you know, to what degree they can, they can with the, the open source community to, to get that back and, and help drive that. You know, all of the LLaMA, the subset of models- Yeah. Yeah...

65
00:09:41.668 --> 00:09:46.448
with, with Alpaca and Vicuna, et cetera, that all came from, from Meta, right? Like that... Yeah.

66
00:09:46.548 --> 00:09:51.888
Like it's not licensed in an open way where you can build a company on top of it, but is now kind of driving this family of, of models.

67
00:09:51.928 --> 00:09:56.888
Like there's a tree of models that, that they're, they're leading and, and where is Google in that, in that playbook?

68
00:09:56.948 --> 00:10:17.568
Like for a long time they were the one releasing those models, being super open and, and now it's just they, they seem to be trailing and there's, there's people jumping ship, and to what degree can they, can they, can they close off those wounds and, and focus on, on where, where they, they have unique ability to, to, to gain momentum, I think is a core part of my takeaway from this.

69
00:10:19.528 --> 00:10:26.568
Yeah. And I think another big thing in the post is, oh, as long as you have high quality data, like you don't need that much data, you can just use that.

70
00:10:27.288 --> 00:10:32.548
The first party data loops are probably gonna be the most important going forward if we do believe that this is true.

71
00:10:32.628 --> 00:10:45.708
So Databricks, we had Mike Conway from Databricks on the podcast and they talked about how they came up with the training set for DALL-E, which they basically had Databricks employees write down very good questions and very good answers for it.

72
00:10:46.248 --> 00:10:56.708
Not every company has the skill to do that. And I think, you know, products like Google, they have millions of people writing Google Docs. They have millions of people using Google Sheets.

73
00:10:56.728 --> 00:11:00.108
They have millions of people writing stuff, creating content on YouTube.

74
00:11:01.068 --> 00:11:12.988
The question is, if you wanna compete against these companies, maybe the model is not what you're gonna do it with because the open source kind of commoditizes it, but how do you build even better data first party loops?

75
00:11:13.068 --> 00:11:15.988
And that's kind of the hardest thing for startups, right?

76
00:11:16.288 --> 00:11:27.488
Like even if we open up the, the models to everybody and everybody can just go on GitHub and, or Hugging Face and get the weights to the best model, but get enough people to generate data for me so that I can still make it good.

77
00:11:27.828 --> 00:11:39.428
That's, that's what I would be worried about if I was a, a new company. How do I make that happen really quickly? I'm not convinced that the data is that big a challenge. So there's this p- project.

78
00:11:39.488 --> 00:11:48.208
So the problem with Facebook LLaMA is that it's not available for, for commercial use, so people are now trying to train a alternative to LLaMA that's entirely on openly licensed data.

79
00:11:48.548 --> 00:11:58.288
And the, the biggest project around that is this Red Pajama project. They released their training data a few weeks ago, and it was 2.7 terabytes, right? So actually tiny, right?

80
00:11:58.348 --> 00:12:06.688
You can buy a laptop that you can fit 2.7 terabytes on. That's it. But it was the same exact data that Facebook... It was the same thing that Facebook LLaMA had been trained on.

81
00:12:06.988 --> 00:12:14.748
'Cause for your base model, you're not really trying to teach it facts about the world. You're just trying to teach it how English and other languages work, how they fit together.

82
00:12:15.048 --> 00:12:19.828
And then the real magic's when you fine-tune on top of that. That's what Alpaca did on top of LLaMA and so on.

83
00:12:20.068 --> 00:12:26.168
And the fine-tuning sets, it looks like tens of thousands of examples to kick one of these raw models into shape.

84
00:12:26.488 --> 00:12:37.676
And tens of thousands of examples, like Databricks spent a month and got the 2,000 employees of their company to help kick in and it worked. You've got the Open Assistant project and crowdsourcing this stuff now as well.

85
00:12:38.456 --> 00:12:43.696
So it's achievable Sorry to interrupt. I agree. I think it's a fas- fascinating point.

86
00:12:43.856 --> 00:12:57.956
Actually, so I've heard through the grapevine that Red Pajamas model trained on the, the data that they released is gonna be releasing tomorrow, and, uh, it's a, it's a very exciting time because the, the, there, there's a, there's a couple more models that are coming down the pipe which are independently produced.

87
00:12:58.396 --> 00:13:03.116
And so, yeah, the, we, everyone is challenging all these assumptions from, from first principles, which is, uh, fascinating.

88
00:13:03.316 --> 00:13:13.356
[laughs] I, I did, I did want it to, to, like, try to get a little bit more technical in terms of, like, the, the, the, the specific points raised because this doc, this doc was just amazing. Can we talk about LoRA?

89
00:13:13.936 --> 00:13:21.596
I, I, I'll open it up to Simon again if he's back. I'd rather someone else take on LoRA. I've, I, I know as much as I've read in that paper, but not much more than that.

90
00:13:21.976 --> 00:13:34.136
So I thought this was kind of like an optimization technique, so LoRA stands for Low-Rank Adaptation. But this is the first mention of LoRA as a form of stackable improvements where he... I forget what...

91
00:13:35.076 --> 00:13:43.076
Let me just, let me just kind of Google this. But obviously anyone who's more knowledgeable, please, uh, come on in. Yeah. I, I, all I've learned is through ChatGPT.

92
00:13:43.116 --> 00:13:50.796
I spent about 20 minutes on GPT-4 trying to figure out [laughs] what LoRA was. Yeah. I studied computer science, but this is not, this is not my area of expertise.

93
00:13:51.616 --> 00:14:06.976
What I got from it is that basically instead of having to retrain the whole model, you can just pick one of the ranks and you take one of, like, the, the weight matrices and, like, make two smaller matrices from it, and then just two to be retrained and training the whole model.

94
00:14:07.416 --> 00:14:15.716
So it saves a lot of- All these techniques, yeah, you freeze part of the thing and then you just train the smaller part. Like, that- Exactly... that seems to be a, a area of a lot of fru- fru research. Uh- Yeah...

95
00:14:15.856 --> 00:14:23.236
I think, uh, Mini GPT-4 recently did something similar as well, and then there's, there's, there's a, there's a sparse model people out today that- Mm-hmm... also did the same thing.

96
00:14:23.736 --> 00:14:30.636
So I've seen a lot of LoRA, but Stable, the, the Stable Diffusion community have been using LoRA a lot, so they, in that case, they had a...

97
00:14:30.996 --> 00:14:48.476
I, the thing I've seen is people releasing LoRAs that are like you, you train a concept, like a, a particular person's face or something, you release, and the, the LoRA version of this ends up being megabytes of data, like, which is, is, you know, is small enough that you can just trade those around, and you can effectively load multiple of those into the model.

98
00:14:48.836 --> 00:14:57.136
But what I hadn't realized is that you can use the same trick on, on language models. That was one of the big, new things for me in reading the, the leaked Google paper today. Yeah.

99
00:14:57.196 --> 00:15:13.896
And I think the point to make around owning the infrastructure, so what ChatGPT has told me is that when you're figuring out what rank you actually wanna do this fine-tuning at, you can have it either go too low and, like, the model doesn't actually learn it, or you can go too high and the model overfits those learnings.

100
00:15:14.336 --> 00:15:23.536
So if you have a base model that everybody agrees on, then all the subsequent, like, LoRA work is done around the same rank, which gives you an advantage.

101
00:15:23.676 --> 00:15:35.716
And the point they made in the, is that since Llama has, has been the base for a lot of this LoRA work, like, they own the, the mind share of the community, so everything that they're building is compatible with their architecture.

102
00:15:35.796 --> 00:15:45.896
But if Google open sources their own model, you know, the rank that they chose for LoRA on Llama might not work on the Google model, so all of the existing work is not portable.

103
00:15:46.156 --> 00:15:55.576
So the impression I got is that one of the challenges with LoRA is that you train all these LoRAs on top of your model, but then if you retrain that base model, those LoRAs become invalid, right?

104
00:15:55.636 --> 00:15:58.156
They're essentially, they're, they're, they're built for an exact model version.

105
00:15:58.416 --> 00:16:20.156
So this means that being the big company with all of the GPUs that can afford to retrain a model every three months, that's suddenly not nearly as valuable as it used to be because now maybe there's an open source model that's five years old at this point and has, like, multiple, multiple stacks of LoRAs trained all over the world on top of it which can outperform your brand new model just because there's been so much more iteration on that base.

106
00:16:20.976 --> 00:16:25.776
I, I think it's, I think it's fascinating. It's, uh, Jin Fan from, uh, NVIDIA was recently making this argument for transformers.

107
00:16:25.836 --> 00:16:41.816
Like, even if we do come up with a better architecture than transformers, there, the sheer hundreds and millions of dollars that have been invested on top of transformers make it actually a, there is some switching costs and it's not exactly obvious that better architecture equals [laughs]

108
00:16:41.896 --> 00:16:48.576
equals we should all switch immediately tomorrow. Um [laughs] It's, it's, it's kind of like the, the difficulty of launching a new programming language today- Yes...

109
00:16:48.616 --> 00:16:57.396
is that Python and JavaScript have a million packages, so no matter how good your new language is, if it can't tap into those existing package libraries, it's, it's not gonna be useful for- Yes...

110
00:16:57.416 --> 00:17:04.576
which is why Mojo is so clever because they did build on top of Python, so they get all of that existing infrastruc- all of that existing code working already.

111
00:17:05.236 --> 00:17:10.736
I mean, what, what we thought you, since you're, you know, co-creator of Django and all that, do, do we wanna take a diversion into Mojo? Yeah.

112
00:17:10.776 --> 00:17:14.636
Oh, no, I would, I, I'd be happy to, to, to jump in and get Simon's take on, on Mojo.

113
00:17:14.816 --> 00:17:32.196
One, one, one small, small point on LoRA is, you know, I, I just think i- if you think about at a high level what the, the major down- downsides are of these, these large language models, it's the fact that they, well, they, they, they're difficult to, to, to train, right?

114
00:17:32.316 --> 00:17:38.356
They, they tend to hallucinate and they, uh, are, have, have a static, like, like they were trained at a certain date, right?

115
00:17:38.856 --> 00:18:21.456
And with, with LoRA, I think it makes it a lot more amenable to training ne- new updates on top of that, that, like, base model on the fly where you can, you know, incorporate new, new data and i- i- i- in a way that is, is, is an interesting and potentially more optimal alternative than in the kind of in context generation, you know, 'cause, 'cause most of, like, if you have perplexity.ai or, or any of these, these approaches currently it's, like, all based off of doing real time searches and then injecting as much into the, the, the local context window as possible so that you, you try to ground your, your, your, your language model both in terms of the, the information it has access to, that, that, that helps to reduce hallucinations.

116
00:18:21.616 --> 00:18:29.276
It can't produce it, but you know, helps to reduce it and then also gives it access to up-to-date information that wasn't around for that, that massive, like, like, pre-training stuff.

117
00:18:29.316 --> 00:18:43.460
And, you know, I think LoRA i- i- in my opinion really, uh, makes it more, more amenable to having Having constantly shifting lightweight pre-training on top of it that scales better than, than normal pre- uh, I'm sorry, n- fine-tune, fine-tuning.

118
00:18:43.940 --> 00:18:52.780
Yeah, that, that was just kind of my one takeaway there. I mean, for me, I've never been... I want to run models on my own hardware. I don't actually care about their factual content.

119
00:18:52.819 --> 00:18:59.420
Like, I don't need a model that's been, that's trained on the most up-to-date things. What I need is a model that can do the Bing and Bard trick, right?

120
00:18:59.480 --> 00:19:05.220
That can tell when it needs to run a search, and then go and run a search to get extra information and, and bring that context in.

121
00:19:05.400 --> 00:19:13.940
And similarly, I want it to be able to operate tools where it can access my email or look at my notes or all of those kinds of things, and I don't think you need a very powerful model for that.

122
00:19:14.280 --> 00:19:26.100
Like, that's one of the things where I feel like, yeah, Vicuna running on my, on my laptop is probably powerful enough to drive a sort of personal research assistant which can look things up for me and it can summarize things from my own notes and it can do all of that.

123
00:19:26.360 --> 00:19:35.160
And I don't care that it doesn't know about the Ukraine war because the Ukraine war had a training cut off. That doesn't matter if it's got those additional capabilities, which are quite easy to build.

124
00:19:35.460 --> 00:19:43.460
You know, the reason everyone's going crazy building agents and tools right now is that it's a few lines of Python code and a sort of couple of paragraphs of prompt to get it to work.

125
00:19:44.980 --> 00:19:49.920
Well, let's, let's, let's maybe dig in on that a little bit, and this, this also is, is very related to Mojo.

126
00:19:50.100 --> 00:20:06.720
'Cause I, I do think there are use cases and domains where having, uh, the, the hyper-optimized, like, uh, version of these models running on device is, is very relevant, where you can't necessarily make API calls out, you know, on the fly and aug- do context au- augmented, you know, generation.

127
00:20:06.900 --> 00:20:18.780
And I was, I was f- talking with, with a, a researcher at Lockheed Martin yesterday literally about, like, like, the, the version of this that's running of, of language models running on, on fighter jets, right?

128
00:20:18.820 --> 00:20:33.680
And you, you talk about, like, the, the, the amount of engineering precision and optimization that has to go into, to those type of models, and the fact that, that you spend so much money, like, like, training a super distilled, you know, ver- version where milliseconds matter, you know, and it's a life or death situation there.

129
00:20:34.060 --> 00:20:40.580
You know, and you couldn't even, even remotely ha- have, have a use case there where you could, like, call out and, and have, have API calls or something.

130
00:20:40.620 --> 00:20:56.200
So I, I do think there's, like, keeping in mind the, the use cases where, where th- there'll be use cases that I'm more excited about, you know, at, at the application level where, where, yeah, I, I want to, to just have it be super flexible and be able to call out to APIs and have this agentic type, type thing.

131
00:20:56.600 --> 00:21:02.160
And then there's also industries and, and use cases where, where you really need everything baked into the model. Yep, agreed.

132
00:21:02.160 --> 00:21:12.500
[laughs] My, my favorite piece take on this is I think GPT-4 as a reasoning engine, which I think came from the, from Nathan at Every 2.0, which I think, uh, yeah, y- I see the 100 score over there.

133
00:21:12.820 --> 00:21:20.300
Simon, do you, do you have a, a few seconds on Mojo? Sure. So Mojo is a brand new programming language. It was just announced a few days ago.

134
00:21:20.620 --> 00:21:29.700
It's not actually available yet, so I think there's an online demo, but presuming it becomes an open source language we can use, it's got really some very interesting characteristics.

135
00:21:29.800 --> 00:21:43.420
It's a superset of Python, so anything written in Py- Python will just work, but it adds additional features on top that let you basically do very highly optimized code w- written in Python syntax but that compiles down.

136
00:21:43.740 --> 00:21:54.040
The, the main thing that's exciting about it is the pedigree that it comes from. It's a team led by Chris Lattner. Lattner built LLVM and Clang, and then he designed Swift at Apple.

137
00:21:54.180 --> 00:22:00.120
So he's got, like, three, three for three on, on extraordinarily impactful high performance computing product.

138
00:22:00.260 --> 00:22:14.380
And he put together this team and they've basically, they're trying to go after the problem of how do you build a language which you can do really high performance optimized work in, but where you don't have to do everything again from scratch, and that's where building on top of Python is so clever.

139
00:22:14.860 --> 00:22:26.760
So I wasn't... Like, if this thing came along, I, I didn't really pay attention to it until J- Jeremy Howard, who built Fast.ai, put up a very detailed blog post about why he was excited about Mojo, which included a...

140
00:22:26.920 --> 00:22:40.460
There's a video demo in there which everyone should watch because in that video he takes matrix multiplication implemented in Python, and then he uses the Mojo extras to 2,000x the performance of that matrix multiplication.

141
00:22:40.740 --> 00:22:51.040
Like, he adds a few static type functions, sort of structs instead of a class, and he gets 2,000 times the performance out of it, which is phenomenal. Like, absolutely extraordinary. Yeah.

142
00:22:51.080 --> 00:23:04.680
So yeah, and that, that got me really excited, like the idea that we can still use Python and all of the stuff we've got in Python, but we can just very slightly tweak some things and get literally, like, 1,000 times upwards performance out of the things that matter.

143
00:23:05.000 --> 00:23:17.920
That's really exciting. Yeah, I, I, I'm curious, like, how come this wasn't thought of before? It's not like the, the, the concept of a language superset hasn't, you know, hasn't, has, has, isn't, is completely new.

144
00:23:18.460 --> 00:23:26.680
But all, as far as I know, all the previous Python interpreter approaches, like the alternate runtime approaches are, you know, like they, they, they're more, they're more sort of

145
00:23:28.080 --> 00:23:38.840
fit conforming to standard Python, but never really tried this additional approach of augmenting the language. I, I, I'm wondering if you have any insights there on, like, why, like why is this a, a, a breakthrough?

146
00:23:39.180 --> 00:23:48.760
Yeah, that's a really interesting question. So Jeremy Howard's piece talks about this thing called MLIR, which I hadn't heard of before, but this was another Chris Lattner project.

147
00:23:48.780 --> 00:23:54.700
You know, he built LLVM as a low level virtual machine that you could build compilers on top of.

148
00:23:54.740 --> 00:24:06.700
And then MLIR was this one that he initially kicked off at Google, and I think it's part of TensorFlow and things like that. But it was very much optimized for multiple cores and GPU access and all of that kind of thing.

149
00:24:07.020 --> 00:24:13.460
And so my reading of Jeremy Howard's article is that they basically built Mojo on top of MLIR.

150
00:24:13.580 --> 00:24:20.540
So they had a huge, huge, like, a starting point where they'd al- And they, they, they knew this technology better than anyone else.

151
00:24:21.020 --> 00:24:34.240
And because they had this very, very robust high performant basis that they could build things on, I think maybe they're just the first people to try and build a high, try and combine a high level language with MLIR with some extra things.

152
00:24:34.300 --> 00:24:46.420
So it feels like they're basically taking a whole bunch of ideas people have been sort of experimenting with over the last decade and bundled them all together with exactly the right team, the right level of expertise, and it looks like they've got the thing to work.

153
00:24:46.900 --> 00:24:52.380
But yeah, I mean, I've, I've... I'm very intrigued to see, especially once this is actually available and we can start using it.

154
00:24:52.420 --> 00:25:02.939
It- bit- Jeremy Howard is someone I respect very deeply, and he's v- he's hyping this thing like crazy, right? His headline, [laughs] his... And he's not the kind of person who hypes things if they're not worth hyping.

155
00:25:03.000 --> 00:25:11.580
He said, "Mojo may be the biggest programming language advance in decades." And from anyone else, I'd kind of ignore that headline, but from him, it, it really means something.

156
00:25:12.060 --> 00:25:19.940
Yes, because he doesn't hype things up randomly. Uh, yeah. And, and, and he's a noted skeptic of Julia, which is, which is also another data science hot topic.

157
00:25:20.440 --> 00:25:33.380
But from the TypeScript and web, web development worlds, there has been a dialect of TypeScript that was specifically optimized to compile to WebAssembly, uh, which I thought was, like, promising and then, uh, and, and eventually it never really took off.

158
00:25:33.480 --> 00:25:43.900
But I, I like this approach because I think more frameworks should, should essentially be languages and recognize that they are language supersets and maybe work on compilers that, that work on them.

159
00:25:43.940 --> 00:25:49.400
And then that is the, the, by the way, that's the direction that React is going right now. So fun times. [laughs] Huh.

160
00:25:49.900 --> 00:25:59.280
T- TypeScript's an interesting comparison actually, because TypeScript is effectively a superset of JavaScript, right? It is. Yeah. And s- But there's not a performance angle. It is purely types, right? Gotcha. Right.

161
00:25:59.380 --> 00:26:08.420
So, so I guess Mojo is the superset of Python, but the emphasis is absolutely on tapping into the performance stuff. Right. Well, the, the- Just things people actually care about. [laughs] Yeah.

162
00:26:08.460 --> 00:26:12.800
The, the one thing I found is, is, is very similar to the early days of TypeScript.

163
00:26:12.920 --> 00:26:26.360
There was the, the, the, the, the most important thing was that it's incrementally adoptable, you know, because people had JavaScript code bases and, and they wanted to incrementally, like, add the, the, the main value prop for TypeScript was reliability and the, the, the, the static typing.

164
00:26:26.760 --> 00:26:28.400
And with Mojo, we see it being...

165
00:26:28.740 --> 00:26:40.420
A- basically anyone who's a target, a large enterprise user of, of Mojo or even researchers, like, they're all going to be coming from a, a hardcore back- background in, in Python and, and have large existing libraries.

166
00:26:40.460 --> 00:26:59.200
And the, the question will be: For what use cases will Mojo be like a, a, a really good fit for that incremental adoption where you can still tap into your, your, your massive, like, Python exi- existing infrastructure, workflows, data tooling, et cetera, and, and, you know, what does, what does that path to adoption look like?

167
00:26:59.420 --> 00:27:03.760
Uh, yeah. We, we, we don't know 'cause it's a wait-listed language, which people were complaining about.

168
00:27:04.280 --> 00:27:09.920
They, they, the, the Mojo creators were, like, saying something about they had to scale up their servers and I'm like, "What language requires a central server?"

169
00:27:10.320 --> 00:27:17.760
So it's a little bit sus, a little bit like there's a, there's a cloud product already in place and, you know, they're waiting for it, but we'll see. We'll see.

170
00:27:17.820 --> 00:27:22.760
I, I mean, Mojo's, like, extremely promising and I, I, I actually want more programming language innovation this way.

171
00:27:22.880 --> 00:27:33.160
You know, I was complaining years ago that programming language innovation is all about stronger types, all fu- all about, like, more functional, more strong types everywhere, and, and this is the first one that's actually much more practical, which I, which I really enjoy.

172
00:27:33.560 --> 00:27:39.940
This is why I wrote about self-provisioning runtimes. And, I mean, this is kind of related to the post, right?

173
00:27:40.160 --> 00:27:46.920
Like, if you stop all of a sudden where, like, the models are all the same and we can improve them, like, where can we get the improvements, you know?

174
00:27:47.140 --> 00:27:52.140
It's like better runtimes, better languages, better tooling, better data collection. Yeah. So

175
00:27:53.340 --> 00:28:00.400
if I were a founder today, I wouldn't worry as much about the model maybe, but I would say, "Okay, what can I build into my product?" And like...

176
00:28:00.720 --> 00:28:09.520
Or what can I do at the engineering level that maybe it's not model optimization because everybody's working on it. But like you said, it's like why haven't people thought of this before?

177
00:28:09.740 --> 00:28:19.020
It's like it's, it- it's definitely super hard, but I'm sure that if you're like Google or you're like OpenAI or you're like Databricks, you got smart enough people that can think about these problems.

178
00:28:19.040 --> 00:28:28.360
So hopefully we see more of this. You need a Zook learner. Okay, I promised to, to keep this relatively tight. I know Simon on a beautiful day. It is a very nice day in California.

179
00:28:28.760 --> 00:28:36.760
I wanted to go through a few more points that you have pulled out, Simon, and, and just give you the opportunity to, to rant and riff and, and what have you.

180
00:28:37.260 --> 00:28:43.480
I- is, are there any other points from going back to the sort of Google, OpenAI mode documents that, that you felt like we, we should dive in on?

181
00:28:44.520 --> 00:28:47.640
I mean, the really interesting stuff there is the strategy component, right?

182
00:28:47.780 --> 00:28:55.840
The, this idea that, that Facebook accidentally stumbled into leading this because they put out this model that everyone else is innovating on top of.

183
00:28:56.100 --> 00:29:07.820
And there's a very open question for me as to would Facebook relicense Llama to allow for commercial usage. So because- Yeah. Is there, is there some rumor? Is that, is that today? Is there a rumor about that?

184
00:29:07.860 --> 00:29:17.020
That would be interesting. Yeah. I saw, I saw something about Zook saying that he would release the, the Llama weights officially. Oh my goodness, no. That I admit, that is, that's huge.

185
00:29:17.020 --> 00:29:19.860
'Cause Mo- Let me, let me confirm the tweet. Let me find the tweet and then, uh, yeah. Okay.

186
00:29:20.380 --> 00:29:36.140
'Cause actually I met somebody from Facebook Machine Learning Research a couple of weeks ago, and, and I, I pressed them on this and they said basically they don't think it'll ever happen because if it happens and then somebody does horrible fascist stuff with this model, all of the headlines will be Berg releases a monster into the world.

187
00:29:36.700 --> 00:29:43.760
So, so hi- his, the, the, the... A couple of weeks ago, his feeling was that it's just too risky for them to, to allow it to be used like that.

188
00:29:44.060 --> 00:29:57.800
But, you know, a couple of weeks is, is, is a couple of months in the AI world, so yeah, it wouldn't be s- It feels to me like strategically Facebook should be jumping right on this because this puts them at the very, the very lead of, of open source innovation around this stuff.

189
00:29:58.520 --> 00:30:02.740
So I've pinned the tweet talking about Zook and Zook saying that Meta will open up Llama.

190
00:30:03.200 --> 00:30:14.560
It's from the founder of Obsidian, which gives it a slight bit more credibility, but it is the only tweet that I can find about it, so completely unsourced. [laughs] We shall see.

191
00:30:15.240 --> 00:30:17.780
I, I, I mean, I have friends within Meta. I should just go ask them.

192
00:30:18.220 --> 00:30:28.540
But yeah, I, I mean, one interesting angle on, on the memo actually is, is that, and, and, and they were linking to this in a, in a doc, which is apparently, like, Facebook got a bunch of people to do...

193
00:30:28.800 --> 00:30:38.560
'Cause they, they never released it for commercial use, but a lot of people went ahead anyway and, and optimize and, and build extensions and stuff. They, they got a bunch of free work out of open source, [laughs]

194
00:30:38.620 --> 00:30:43.500
which is an interesting strategy. There's... Okay. No, I, I don't know if I have to, like- Oh, I've got an exciting piece of news.

195
00:30:43.640 --> 00:30:56.020
I've just heard from somebody with contacts at Google that they've heard people in Google confirm the leak, that that document was indeed a Git Google document, which I don't find surprising at all, but I'm now up to a 10 out of 10 on, on whether that's, that's, that's real.

196
00:30:56.020 --> 00:31:07.384
[laughs] Excellent. Excellent. Yeah, it is fascinating. Yeah, I mean, the, the strategy is, is, is really i- interesting. I think Google has been kind- Definitely sleeping on monetizing.

197
00:31:07.644 --> 00:31:15.364
You know, I, I, I heard someone call w- when Google Brain and DeepMind merged that they were, it was like goodbye to the Xerox PARC of our era.

198
00:31:15.744 --> 00:31:25.004
And it definitely feels like Google X and Google Brain were definitely Xerox PARCs of our, of our era, and I, I guess we all benefit from that. So one thing I'd say about the, the, the Google side of things, like the...

199
00:31:25.124 --> 00:31:30.304
There was a question earlier, why are Google so worried about this stuff? And I think it's just, it's just all about the money.

200
00:31:30.444 --> 00:31:49.464
You know, the, the, the engine of money at Google is Google Search and Google Search Ads, and who uses ChatGPT on a daily basis like me will have noticed that their usage of Google has dropped like a stone because there are many, many questions that, that ChatGPT, which shows you no ads at all, is, is, is a better source of information for than Google now.

201
00:31:49.924 --> 00:31:50.744
And so, yeah, I'm not...

202
00:31:50.804 --> 00:32:06.464
It doesn't surprise me that Google would see this as an existential threat because whether or not they can b- e- you know, Bard, it's actually, it's not great, but it, it exists, but it isn't it yet either, and if I've got a chatbot, chatbot that's not showing me ads and chatbot that is showing me ads, I'm gonna pick the one that's not showing me ads.

203
00:32:08.144 --> 00:32:19.304
Yeah. Yeah. I, I agree. I, I did see a prototype of Bing with ads, Bing Chat with ads. Um- I haven't seen the prototype yet, no. Yeah. Yeah. Anyway, I, I... It, it will come obviously, and then we will choose.

204
00:32:19.344 --> 00:32:24.404
We'll, we'll go out of our way to avoid ads just like we always do. [laughs] We'll need ad blockers in chat. Excellent.

205
00:32:24.704 --> 00:32:30.724
So I feel like on the safety side, the, the safety side, there are basically two areas of safety that I, I, I sort of split it into.

206
00:32:30.804 --> 00:32:46.624
There's the science fiction scenarios, the AI breaking out and killing all humans and creating viruses and all of that kind of thing, the sort of the Terminator stuff, and then there's the, the people doing e- bad things with AI, and that latter one is the one that I think is much more interesting.

207
00:32:47.044 --> 00:32:56.924
And, you know, 'cause you could u- like things like romance scams, right? Romance scams already take billions of dollars from, from vulnerable people every year. Those are very easy to automate using existing tools.

208
00:32:56.984 --> 00:33:05.264
I'm pretty sure Vicuna 13B running on my laptop could spin up a pretty decent romance scam if I was evil and wanted to use it for that.

209
00:33:05.484 --> 00:33:14.704
So that's the kind of thing where I get really nervous about it, like the fact that these models are out there and bad people can use these, do bad things, most importantly at scale.

210
00:33:14.744 --> 00:33:23.624
Like romance scamming, you don't need a language model to pull off one romance scam, but if you want to pull off 1,000 at once, a language model might be the, the thing that, that helps you scale to that point.

211
00:33:24.004 --> 00:33:32.804
And yeah, in terms of the science fiction stuff and also a model on my laptop that can guess what comes next in a sentence, I'm not worried that that's going to break out of my laptop and destroy the world.

212
00:33:33.304 --> 00:33:43.164
There, there... I get slightly nervous about the huge number of people who are trying to build AGIs on top of these models, the baby AGI stuff and so forth, but I don't think they're gonna get anywhere.

213
00:33:43.224 --> 00:33:51.524
I feel like if you actually wanted a model that was, was a threat to humanity, a language model would be a tiny corner of what that thing was actually built on top of.

214
00:33:51.564 --> 00:33:54.344
You'd need goal setting and all sorts of other bits and pieces.

215
00:33:54.744 --> 00:34:07.024
So yeah, for the moment, the science fiction stuff doesn't really interest me, although it is a little bit alarming seeing more and more of the very senior figures in this industry sort of tip the hat and say, "You know, we're getting a little bit nervous about this stuff now."

216
00:34:08.924 --> 00:34:20.484
Yeah, so that would be Jeff Hinton and Yoshua Bengio. And I, I saw this meme this morning that Yann LeCun was like happily saying, "This is fine." [laughs] Being the third Turing Award winner. [laughs] Yeah.

217
00:34:20.524 --> 00:34:36.144
But you'll see a lot of the AI sa- the people who've been talking about AI safety for the longest are getting really angry about the science fiction scenarios, 'cause they're like, "No, the, the thing that we need to be talking about is the harm that you can cause with these models right now today, which is actually happening," and the science fiction stuff kind of ends up distracting from that.

218
00:34:36.504 --> 00:34:41.784
I love it. You, you... Okay, so, so, uh, y- Eliezer, I, I don't know how to pronounce his name.

219
00:34:42.084 --> 00:34:52.824
Eliezer has a list of ways that AI will kill us post, and I think, Simon, you could write a list of ways that AI will harm us but not kill us, right? Like the, the, the non-science fiction actual harm ways.

220
00:34:53.364 --> 00:35:02.924
I think- Right... I haven't seen a, a actual list of like, hey, romance scams, spam. Uh, I, I don't, I don't know what else, but that could be very interesting as a- Hmm. Okay...

221
00:35:03.244 --> 00:35:09.504
practical, like here are the situations we need to guard against because they are more real today than that we need to think about, warn about.

222
00:35:09.744 --> 00:35:17.764
Obviously, you've been a big advocate of prompt injection awareness even though you can't really solve them, and [laughs] I, I walked through a scenario with you. But yeah. Yeah.

223
00:35:17.804 --> 00:35:20.464
Prompt injection's a whole other side of this, which is, I mean, that...

224
00:35:20.524 --> 00:35:31.204
If you want a risk from AI, the risk right now is everyone who's building prompts are building systems that attackers can trivially subvert into stealing all of their private data, unlocking their house, all of that kind of thing.

225
00:35:31.564 --> 00:35:39.204
So that's another very real risk that we have today. Yeah. I think in all our personal bios, we should edit in prompt injections already.

226
00:35:39.444 --> 00:35:48.884
Like in, on my website, I wanna add in a personal prompt injection so that if I get scraped, like I'll know if someone's like reading from a script, right, that, uh, that is generated by an AI bot.

227
00:35:49.264 --> 00:35:58.984
I've seen people do that on LinkedIn already, and they get, they get recruiter emails saying, "Hey, I didn't read your bio properly, and I'm just an AI script, but would you like a job?" Yeah, it's fascinating.

228
00:35:59.084 --> 00:36:09.924
[laughs] Okay. All right. So tr- trying to stay, uh, roughly on topic. Um, I, I, I think, I think this, this, this moat is, you know, is a peek under the curtain of the, the, the internal panic within Google.

229
00:36:10.224 --> 00:36:20.704
I think it is very va- very validated. I'm not so sure they should care so much about small models or, or like on-device models, but the other stuff is interesting. There was a comment at the end that you had,

230
00:36:21.764 --> 00:36:26.184
Simon, about as for OpenAI, OpenAI is themselves, OpenAI doesn't matter, right?

231
00:36:26.264 --> 00:36:39.264
So this is a Google document talking about Google's position in the market and what Google should be doing, but they had a comment here about OpenAI, and they also say OpenAI had no moat, which is a interesting and brave comment given that OpenAI, you know, is the leader in, in a lot of these innovations.

232
00:36:40.024 --> 00:36:49.704
Well, one thing I will say is that I think we might have identified who within Google wrote this document now. There's a version of it floating around with a name, and I looked them up on LinkedIn.

233
00:36:49.824 --> 00:36:57.304
They're heavily involved in the AI corner of Google. So my guess is that at Google, and this, when I've worked for companies, I'll put out a memo.

234
00:36:57.404 --> 00:37:04.084
I'll write up a Google Doc, and I'll email, email it around, and it's nowhere near the official position of the company or of the executive team.

235
00:37:04.324 --> 00:37:13.312
It's somebody's opinion And so I think it's more likely that this particular document is somebody who works for Google and has an opinion and distributed it internally and then it, and then it got leaked.

236
00:37:13.392 --> 00:37:19.812
I don't know if it's necessarily represents Google's sort of institutional thinking about this. I think it probably should.

237
00:37:19.912 --> 00:37:28.792
Again, this is such a well-written document, it's so well argued, that if I was an executive at Google and I read that, I would, I would be thinking pretty hard about it. But yeah Oh...

238
00:37:28.832 --> 00:37:37.631
I don't think we should see it as, as sort of the official secret internal position of the company. Yeah. Fir- first of all, I might promote that person, 'cause he's clearly [laughs] more, uh- Oh, definitely.

239
00:37:37.671 --> 00:37:44.772
He's, he's, he's really... Th- this is a, it's... I, I would hire this person on the strength of that document. [laughs] But second of all, this is more about OpenAI.

240
00:37:44.832 --> 00:37:52.512
Like, I, I'm not interested in Google's, you know, official statements about OpenAI, but I was just interested, like his assertion that OpenAI doesn't have a moat, that's a bold statement.

241
00:37:52.712 --> 00:37:57.512
I, I don't know, it's got, it's got the best people. [laughs] Well, I, I would, I would say two things here.

242
00:37:57.812 --> 00:38:05.152
One, it's really interesting, just at a meta- meta po- point, that, that they even approached it this way of having this public leak.

243
00:38:05.212 --> 00:38:18.092
It, it, it kind of tal- talks a little bit to the fact that they, they, they felt that, that doing, doing internally, like wasn't going to get anywhere or, or maybe this speaks to, to some of the like middle management type stuff or, or within Google.

244
00:38:18.512 --> 00:38:33.072
And then to the, the, the, the point about like OpenAI not having a moat, I think for, for large language models, it, it, it will be over, over time kind of a race to the bottom just because the switching costs are, are, are so low compared with traditional cloud and SaaS.

245
00:38:33.572 --> 00:38:40.612
And, you know, yeah, there will be differences in, in, in quality, but, but like over time, if you, you look at the limit of these things, like the...

246
00:38:40.672 --> 00:38:52.912
I, I think Sam Altman, you know, has been quoted a few times saying that the, the, the price of marginal price of intelligence will go to zero over time, and the marginal price of energy powering that intelligence will, will also head over time.

247
00:38:53.252 --> 00:38:59.872
And in that world, if you're, you're providing large language models, they become commoditized. Like, yeah, what, what is, what is your moat at that point? I don't know.

248
00:38:59.932 --> 00:39:05.212
I think they're e- extremely well-positioned as a team and as a company for leading this space. I'm not that, that worried about that.

249
00:39:05.252 --> 00:39:14.952
But it is something from a strategic point of view to keep in mind about large language models becoming a commodity. So it's quite short, so I think it's worth just reading the en- that, that entire section.

250
00:39:14.991 --> 00:39:24.472
It says, "Epilogue: What about OpenAI? All of this talk of open source can feel unfair given OpenAI's current closed policy. Why do we have to share if they won't?" That's talking about Google sharing.

251
00:39:24.732 --> 00:39:35.412
"But the fact of the matter is we are already sharing everything with them in the form of the steady flow of poached senior researchers. Until we stem that tide, secrecy is a moot point." I love that. That's so salty.

252
00:39:35.432 --> 00:39:45.412
[laughs] And, and in the end, OpenAI doesn't matter. They are making the same mystiques, mistakes that we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question.

253
00:39:45.772 --> 00:39:52.492
Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least we can make the first move.

254
00:39:52.832 --> 00:40:05.892
So the argument this, this paper is making is that Google should go, go like Meta and, and just lean right into open sourcing it and engaging with the wider open source community much more deeply, which OpenAI have very much signaled they are not willing to do.

255
00:40:06.432 --> 00:40:13.852
But yeah, it's, it's, it... Read the whole thing. The whole thing is full of little snippets like that. It's just super fun. Yes. Yes, read the whole thing.

256
00:40:14.012 --> 00:40:21.512
I, I, I also appreciated the timeline because it set a lot of really great context for people who are out of the loop. So yeah. Yeah. All right.

257
00:40:21.532 --> 00:40:31.072
And the, the final conspiracy theory is that this got leaked right before Sundar and Satya and Sam all went to the White House this morning, so. Yeah. Did it happen? I haven't caught up.

258
00:40:31.092 --> 00:40:41.972
But did, like the White House issue a statement? No, that I, I just saw, I just saw the photos of them going into the, the White House. I haven't, I haven't seen any post-meeting updates. Yeah.

259
00:40:42.012 --> 00:40:51.792
I, I think it's a big win for Anthropic to be at that table. Oh, yeah. Yeah. For sure. And Cohere's not there. I was like, "Hmm, interesting." Well- Anyway. Yeah. They need, they need some help. Okay.

260
00:40:51.832 --> 00:41:00.852
Well, I, I, I pr- I promise to keep this relatively tight. These spaces do tend to have a, have a tendency of dragging on. But before we go, anything that you all want to plug, anything that you're working on currently?

261
00:41:01.372 --> 00:41:09.152
Maybe go around to Simon. Are you still working on Datasette? [laughs] I am. I am. I'm having a bit of a... So Datasette's my open source project that I've been working on.

262
00:41:09.152 --> 00:41:19.192
It's about helping people analyze and publish data. I am having an existential crisis of it at the moment because I've got access to the ChatGPT code interpreter mode, and you can upload- Are you-...

263
00:41:19.472 --> 00:41:28.592
a SQLite database to that, and it will do all of the things that I, that were on my roadmap for the next 12 months. Oh my God. So that's frustrating. So I'm basically, I'm leaning Datasette...

264
00:41:28.712 --> 00:41:31.772
My interest in Datasette and AI are g- are rapidly crossing over.

265
00:41:31.812 --> 00:41:39.992
I'm, got harder about the AI features that I need to build on top of Datasette to make sure it stays relevant, and that ChatGPT can do most of the stuff that it does already.

266
00:41:40.452 --> 00:41:46.912
But yeah the thing I'll plug, my blog, simonwilliaston.net. I am now updating it daily with stuff because AI mo- moves so quickly.

267
00:41:47.152 --> 00:42:01.572
And I have a Substack newsletter, which is effectively my blog, but in email form sent out a couple of times a week, which please subscribe to that or RSS feed of my blog or, or whatever, 'cause I, I'm trying to keep track of all sorts of things, and I'm publishing a lot at the moment.

268
00:42:02.632 --> 00:42:12.212
Yes. You, you are, and we love you very much for it because you, you are a very good reporter and technical deep diver into things, into all the things. Thank you, Simon. Travis, are you ready to announce the...

269
00:42:12.452 --> 00:42:23.792
I, I guess you've announced it some- somewhat. Yeah. Yeah. So I'm I, I, I just founded a company. I'm working on a framework for building reliable agents that aren't toys, and focused on more constrained use cases.

270
00:42:24.092 --> 00:42:40.552
You know, I, I, I look at kind of AGI and these, these AutoG type, type projects as like jumping all the way to str- to, to self-driving and, and we, we, we kind of wanna, wanna start with some more anchor and really focus on, on reliable primitives to, to start that, and that'll be an open source TypeScript project.

271
00:42:40.592 --> 00:42:51.832
I'll be releasing the first version of that soon. And that's, that's it. Follow me, you know, on here for, for this type of stuff. I, I, I, everything AI. And, and spam his ChatGPT bot while you still can. Oh, yeah.

272
00:42:51.872 --> 00:43:00.812
The ChatGPT Twitter bot has about 125,000 followers now. It's still running. I, I'm not sure if it's Earn your credits... you know? [laughs] Yeah. Well, I think I've actually- Can you say how much you spent, actually?

273
00:43:01.332 --> 00:43:11.532
No, no. Well, I think probably totally like, like 1,000 bucks or something. But, I, it's, it's sponsored by OpenAI, so I haven't, I haven't actually spent any real money. What? That's awesome. Yeah. Yeah.

274
00:43:11.552 --> 00:43:20.232
[laughs] Well, once, once I changed... Originally the logo was the ChatGPT logo, and it was the green one, and then they, they hit me up and asked me to change it, so it's now it's a purple logo. Yeah, yeah.

275
00:43:20.232 --> 00:43:28.372
Um, and they're, they're, they're cool with that. Yeah. Yeah. Yeah, OpenAI is sending take down notices to people with GPT stuff apparently now. So it's, yeah, it's a little bit of a gray area.

276
00:43:28.452 --> 00:43:37.692
I wanna write more on, on modes. I've been actually collecting and meaning to write a piece on modes, and today I saw the memo, I was like, "Oh, okay. Like, I guess today's the day we talk about modes." So thank you all.

277
00:43:37.732 --> 00:43:45.852
Thanks. Thanks, Simon. Thanks, Travis, for, for jumping on. And thanks to all the audience for engaging on this with us. We'll continue to engage on Twitter, but thanks to everyone. Cool. Thanks, everyone.

278
00:43:45.852 --> 00:43:48.732
Thanks, everyone. Bye. Thanks, everyone. Bye. All right. Thanks, everyone. Bye.
