WEBVTT

1
00:00:00.060 --> 00:00:12.580
[on-hold music] Hey, everyone. Welcome to the Latent Space Podcast.

2
00:00:12.800 --> 00:00:19.960
This is Alessio, partner and CTO in residence at Decibel Partners, and I'm joined by my co-host Swyx, writer and editor of Latent Space.

3
00:00:20.300 --> 00:00:28.400
Hey, and today we're back in the studio again, uh, after a little break, and we have Aman Sanger in the house. Hey, Aman. Hey. Thanks for coming. [chuckles] Thanks for having me.

4
00:00:28.860 --> 00:00:39.020
So we like to introduce our guests and then have you fill in the blanks. So you worked at Gamelan, Bridgewater, McKinsey, Google, and You.com, all on sort of kind of AI-related things and some finance-related things.

5
00:00:39.040 --> 00:00:50.240
Mm-hmm. You also ran your own, uh, consultancy, Abelian AI, and you, you graduated in CS/MF from MIT recently. Worked on a few projects, uh, including Instill, which I think we'll cover a little bit later.

6
00:00:50.780 --> 00:00:55.240
And most recently Cursor.so, which we'll cover for the vast majority of the podcast.

7
00:00:55.240 --> 00:01:04.900
[chuckles] But, uh, just on a personal side, what's one thing that people should know about you that, you know, might not be so obvious on LinkedIn? Oh, interesting. In a previous life, I played a lot of squash.

8
00:01:05.140 --> 00:01:16.460
You were a top seed? Yeah. So in high school I kind of competed in tournaments, and most people probably don't really know what squash is. It's like tennis in many ways. It's like a racket sport, but it's indoors.

9
00:01:16.920 --> 00:01:22.500
You play against a wall. I guess now pickleball is all the rage- Yeah... with, with racket sports. But, um- Outdoor squash. [chuckles] Yeah.

10
00:01:22.560 --> 00:01:30.580
The story is I used to play tennis, and then I moved to a building that had a squash court in it, and then I picked it up. I loved it, and I've been playing ever since.

11
00:01:30.640 --> 00:01:40.600
So I competed a lot in high school, played a bunch at MIT. Have not had the chance to play much here. Uh, in San Francisco, there aren't too many courts. We can organize a squash tournament.

12
00:01:40.600 --> 00:01:53.760
[chuckles] And then you'll crush it, of course. Is there anything about the athlete mentality that you take with you as a founder? Yeah. I think it can be at times a bit too much, but I'm very competitive.

13
00:01:54.200 --> 00:02:01.680
I really hate losing. Like, now I think I'll go on runs and if someone tries passing me, [chuckles] I won't let it happen.

14
00:02:01.760 --> 00:02:08.680
[chuckles] I'll just kick it into overdrive and, like, maybe I'll turn the corner if I know they're gonna beat me, but I, I can't let someone pass me when I'm running.

15
00:02:08.740 --> 00:02:18.360
And I, I think the same is true with, with startups where the competitive nature just, I think it, it in general helps motivate me and makes me, I, I guess just, like, work harder. Yeah. Okay.

16
00:02:18.400 --> 00:02:28.920
Well, we'll, we'll have a bunch of competitive questions later. Okay. But, uh, we'll go over the, the timeline. Yeah. Let's jump into the how you got to Cursor. So in August 2022, you launched something called Instill.

17
00:02:29.500 --> 00:02:36.580
Can you talk a little bit about that? Yeah. And maybe before I go into Instill, I should talk about what I was even doing before that. Mm-hmm.

18
00:02:36.600 --> 00:02:41.860
Because Instill was actually a very brief array from what I was doing with my original co-founder, Michael.

19
00:02:42.420 --> 00:02:49.200
So we had both actually gone to the same high school together, gone to MIT together, and then after graduating, we knew we wanted to start something.

20
00:02:49.820 --> 00:02:59.020
And in June, what we were working on was also called Cursor, but very different. We basically were very, very fanatical users of Copilot.

21
00:02:59.060 --> 00:03:05.920
We, we, we loved it, and we had a little bit of experience with computer-aided design or, or CAD software.

22
00:03:06.300 --> 00:03:14.540
A lot of our friends, in fact, were mechanical engineers, and we'd heard a lot about how tedious it was to design these parts in software like SolidWorks and whatnot.

23
00:03:14.980 --> 00:03:29.440
And it was pretty obvious to us that if you could train a transformer on the task of predicting the next token, not just for code, but for CAD, then you could get a really useful product that could speed up mechanical engineering.

24
00:03:29.900 --> 00:03:37.880
So that's actually what we'd worked on up until Instill, even a little bit after Instill. And yeah, I can go into more detail about that. It was, it was pretty interesting.

25
00:03:37.940 --> 00:03:47.580
That's probably how despite these days doing less stuff with model training than in the past, for that, it was all just kind of rolling our own models from scratch. A lot of training, a lot of inference.

26
00:03:48.100 --> 00:03:58.660
I'm always curious to hear about what made you interested in that. You know, obviously you've been at the forefront of a lot of this AI work. Why was that the most interesting thing to you?

27
00:03:58.800 --> 00:04:05.760
Did you think there were not as many people going after that? Did you think you had a unique insight into it? Because we got a lot of people listening- Yeah...

28
00:04:05.780 --> 00:04:13.960
that wanna be founders and wanna figure out how to make that decision. Yeah. Well, first off, I've always been incredibly fascinated by AI.

29
00:04:14.220 --> 00:04:23.220
The first time I originally learned how to program, actually, because I'd seen the results from ImageNet, and I'd heard deep learning, and that just sounded insanely cool to me.

30
00:04:23.560 --> 00:04:32.960
And so my first programming project was building and training a neural network in Java because that was the only language I, I knew from my AP computer science class.

31
00:04:33.360 --> 00:04:44.000
But ever since then, uh, kind of everything I've done has been involving ML, AI. And the reason I wanted to, I guess, found a company is, first off, I had been working with Michael on a couple of other things.

32
00:04:44.040 --> 00:04:49.420
We'd done an AI consultancy in the past. We worked really well together and really just enjoyed working on stuff on our own.

33
00:04:50.140 --> 00:04:57.600
And with CAD, we were doing a little bit of ideation, and I think we were quite worried about competition in a lot of other areas.

34
00:04:57.660 --> 00:05:05.520
I think that worry has definitely subsided a little bit in-- with what we're working on now. Obviously, a lot of competition in the coding, coding space.

35
00:05:06.060 --> 00:05:16.580
But it seemed like the kinda thing where not a lot of eyes were on this. It seemed very technically possible, at least at the time, and the market was, was pretty sizable if you, if you looked into it.

36
00:05:16.660 --> 00:05:26.460
So it was both a really interesting technical problem, and then if you just tried to analyze the space, it seemed like a good idea. How do you decide to move off of it? That's another- Yeah...

37
00:05:26.480 --> 00:05:32.820
important, uh, answer as a founder. I think there are a few key things that we did not take into account when we were working on this.

38
00:05:33.380 --> 00:05:44.700
One was, if you look at the original Codex paper, our assumption was this is the model that powers Copilot. It was trained on 100 billion tokens, or it was something like 50 billion tokens of Python.

39
00:05:45.260 --> 00:05:55.360
And one interesting insight from it was that you actually get no transfer benefits from the pre-trained model on text to code.

40
00:05:56.020 --> 00:06:06.800
So that means they took GPT-3, and for the smaller models, for the models that weren't trained in all of the Python data, there were some benefits where GPT-3 transferred really well faster.

41
00:06:07.240 --> 00:06:17.580
But then for like the final Codex model, it turns out that There were no transfer benefits, meaning you just took a model, you trained it from scratch on those 100 billion tokens of code, of Python code.

42
00:06:17.600 --> 00:06:29.540
It would do just as well as the GPT-3 12 billion model that was fine-tuned. The issue was that it was only true for GPT-3 and 100 billion tokens of Python code.

43
00:06:29.600 --> 00:06:36.800
These days, I mean, the jury's still out on this, but it seems pretty clear that the benefits from language or learning language are quite helpful with code.

44
00:06:37.260 --> 00:06:44.420
I guess that kind of goes into the issues with CAD, where, one, you're dealing with much less data than code.

45
00:06:44.920 --> 00:06:51.800
If you assume first off that 50 billion, 100 billion tokens is all you need, then maybe with like 10 X less, you could get a pretty useful model.

46
00:06:52.240 --> 00:06:57.680
In reality, Copilot today is powered by probably trillions of tokens of code as well as text.

47
00:06:58.120 --> 00:07:06.040
And when you're dealing with, at most, from scraping every single bit of CAD data you can find, 10 billion tokens, it's just not enough to train a useful model.

48
00:07:06.080 --> 00:07:14.340
We tried scaling, and no matter what kinds of regularization techniques we used, we just couldn't get it past a few billion parameters without overfitting. That was a big thing.

49
00:07:14.380 --> 00:07:16.260
And then the other is that there's no transfer.

50
00:07:16.479 --> 00:07:28.840
If you try to test these models today, and even with GPT-4, there's a prompt that I like to use, which is good for testing like 3.5 versus 4 if you don't know which one's behind the scenes, and even 4 sometimes struggles with it.

51
00:07:28.880 --> 00:07:38.100
And, and the prompt is you kind of lay out... I think it's like a, a famous kind of Gary Marcus prompt as well, where you lay out a bunch of kind of cubes on- Yeah... on a table, right?

52
00:07:38.120 --> 00:07:44.380
And you describe it and a- as you increase the complexity, you know, 3.5 drops out. You increase the complexity more, 4 drops off.

53
00:07:44.900 --> 00:07:54.700
But it's clear that these models are not that good at spatial reasoning, and that's exactly what's needed for CAD. And- Oh, yeah. That's right...

54
00:07:54.840 --> 00:08:02.480
what you want to do is if I were to, to design this table with CAD in front of you, I would first draw a rectangle.

55
00:08:03.260 --> 00:08:15.040
Then I would do an extrusion operation, which would basically take the rectangle and then extend it orthogonal to the plane so that, such that it's like a, a volume, right?

56
00:08:15.660 --> 00:08:23.300
And then the model has to realize that, okay, the shape now that exists is this, this structure here, this table.

57
00:08:23.960 --> 00:08:30.520
And then the really difficult thing is for other operations, it'll need to point to the constructed geometry that was built.

58
00:08:30.540 --> 00:08:40.600
And basically, the model effectively to work, to work well, it needs to kind of, in its mind, imagine this 3D structure, and the models are not good at that. If you try fine-tuning code models in this task, it's just...

59
00:08:40.940 --> 00:08:48.040
Or language models, they're just not gonna transfer well at all. Do you think in like two, three years there will be a good, uh, AI-powered CAD software or? Yeah.

60
00:08:48.320 --> 00:08:53.720
My perspective now is that I think the best way here is probably redesigning the entire system.

61
00:08:53.740 --> 00:09:03.640
[chuckles] If you-- Like, one other big pain point was we tried to build plug-ins with all the major pieces of CAD software like SolidWorks, Onshape, and so on and so forth.

62
00:09:04.320 --> 00:09:10.880
And if you think it's hard to build a plug-in for some of the older IDEs, you've not, [chuckles] you've not seen these pieces of software.

63
00:09:11.560 --> 00:09:19.180
And so I think even if you got a good model, it might be really hard to actually get distribution and create a good plug-in that works.

64
00:09:19.260 --> 00:09:32.360
So it feels like with, uh, the advancements you have in, in kind of text images, and there's some new companies kinda doing stuff with text to 3D, it feels like the reasonable approach is actually just scrap the way that people are doing CAD right now.

65
00:09:32.760 --> 00:09:42.380
And I suspect some, a company or some companies will come around and do this quite well. That's really good insight and, uh, we have more sort of general LLM product thoughts to ask you at the end. Mm-hmm.

66
00:09:42.800 --> 00:09:54.660
We wanted to get into Cursor since that, that is, uh, your primary product right now. In January 2023, you, you announced it to the world. Maybe take us into the, the, I guess, idea maze, uh, leading up to Cursor. Yeah.

67
00:09:55.020 --> 00:10:05.600
I guess Instill was one kind of brief, brief pivot period where we tried doing text to images. The reason we decided against it was that I don't think we're the founders for that kind of company.

68
00:10:06.180 --> 00:10:16.200
We learnt this from CAD, and we strongly believe this now, that it is much better to be a user of the product you're building, and we just weren't big users of any of the text image tools.

69
00:10:16.820 --> 00:10:32.379
So it was around December where we managed to actually get early access to GPT-4, and before then, we had played around a little bit with using earlier versions of 3.5 on writing code, and we'd kinda given up.

70
00:10:32.499 --> 00:10:40.740
It just seemed like if you looked at Text DaVinci-2 or Code DaVinci-2, those older 3.5 variants, they just couldn't really do anything meaningful.

71
00:10:41.340 --> 00:10:52.860
But then we opened up the playground, started copy-pasting code into there, and it was ridiculous. This is before everyone started using human- Did you use the early version? Yes. Or the- So- Okay...

72
00:10:53.220 --> 00:11:02.700
so, um, it was an earlier version. The unhinged, uh, raw [laughs] Oh, it was still... No, it was still... It, it wasn't... Yeah, it was still s- very safe. [laughs] But before people started using...

73
00:11:02.980 --> 00:11:11.680
Human eval was the thing, but before everyone started talking about it and knowing about it, we, we kind of pasted it in and it got 85%, and we were just like, "Wow."

74
00:11:11.960 --> 00:11:23.100
Best open source model at the time got 30%, and Code DaVinci-2 got something like 47%. And yeah, GPT-4 today, it gets about the same score.

75
00:11:23.580 --> 00:11:32.000
And so we then started, you know, writing code in there, just copying and pasting random pieces of code from whatever kind of things we were testing and developing.

76
00:11:32.040 --> 00:11:45.040
We found that it was not just good at creating net new things, but refactoring code, editing code, helping you debug. Kind of every single aspect of software development felt so different with these models.

77
00:11:45.620 --> 00:11:53.620
And then we kind of, in our heads, just plotted out the future, and this is GPT-4. Like, what happens when you have 4.5, GPT-5? It's...

78
00:11:53.800 --> 00:12:05.480
These models are just gonna get better and better and better at programming, and the future is probably not going to be more and more things that you Tab Enter for autocomplete. I think that's a very useful tool.

79
00:12:05.540 --> 00:12:15.920
We use Copilot every day. We find it quite useful. But you can't have a world in which language models are able to produce 90%, 95% of the code, and it still follows that form factor.

80
00:12:16.360 --> 00:12:21.910
I think you have to redesign the entire way, the entire UX of- Writing software.

81
00:12:22.390 --> 00:12:31.770
And that was our take with Cursor, where you need to own the full IDE and completely redesign the flow of producing, producing software and just doing software development in general.

82
00:12:32.470 --> 00:12:40.650
Those are big statements that we need to dig into a little bit more [laughs]. Yes. Uh, so I, I wanna backtrace a little bit. So you got a early access- Mm-hmm... to, to GPT-4. Mm-hmm.

83
00:12:41.250 --> 00:12:48.150
That actually means that you got into the... You were backed by OpenAI, you joined OpenAI fund before you were Cursor. Yeah, basically, kind of. Oh, okay.

84
00:12:48.210 --> 00:12:54.410
'Cause I'm trying to get the chronology and I, I assumed you were... They funded you because of Cursor, but- Yeah, so OpenAI has this program, Converge.

85
00:12:54.610 --> 00:13:06.030
That was the program we participated in, and through that, the main thing was early access to unreleased models that we got to play with. Obviously, none of this went to production. None of this could go into production.

86
00:13:06.070 --> 00:13:14.830
It was just kind of a sneak peek of, of GPT-4. And so yeah, before we actually built out Cursor, we didn't take money from OpenAI, but we were a part of this program. Got it. Yeah.

87
00:13:15.370 --> 00:13:25.890
And then you also mentioned one more thing which was interesting, that you still use Copilot- Mm-hmm... but you also use Cursor. Yes. You also mentioned that Copilot is probably trained on trillions of tokens. Mm-hmm.

88
00:13:25.930 --> 00:13:35.050
Which means it's, that's, that's extensive training since the original Codex. That's my guess. Okay. I mean, if you look at the stack, for example, right? Yeah. It's, it's what, one to two trillion tokens? Okay.

89
00:13:35.070 --> 00:13:44.850
Something around that. I'm very skeptical that Copilot's training on less, especially with all the lawsuits you see with whether or not it's, quote-unquote, "fair use." So yeah, my guess is trillions of tokens.

90
00:13:44.910 --> 00:13:52.150
I don't really know. But yeah, I'm sure if you, if you did the math on how much public code there is on GitHub, it's, it's almost certainly in the trillions.

91
00:13:52.390 --> 00:14:02.490
One of the reasons I harp on this is one of our pet themes is tracking the dataset-to-parameter ratio. Mm. And it... Copilot cannot be that big 'cause it's, it's- Yeah... returns relatively quickly.

92
00:14:02.850 --> 00:14:11.610
So it's gonna be in, like, the low billions, right? Well- So how do you do trillions of tokens to the low billions, you know? That's interesting. Um. Yeah.

93
00:14:11.710 --> 00:14:19.530
I think I have some, some thoughts on this because there's the whole thing with Chinchilla scaling, and then people are now saying, "Oh, Chinchilla scaling doesn't matter because of, of inference."

94
00:14:19.970 --> 00:14:30.369
But Copilot could be mixture of experts. That's one other speculation. I don't know if that's true. I mean, it, it probably wasn't the case at least a year or two ago.

95
00:14:31.030 --> 00:14:42.890
But yeah, my guess it's probably a, a small model that's, that's very over-trained. From what I've heard, there are also lots of tricks you can do with caching, where even if the model's quite big, it doesn't take...

96
00:14:42.930 --> 00:14:48.030
It effectively takes no time to ingest the entire prompt. Yeah, semantic caching is what they are calling it, right? Yeah.

97
00:14:48.050 --> 00:14:52.850
Like, it sort of roughly embeds to the same thing, just returns- Well, I, I think it's partially that, right?

98
00:14:52.950 --> 00:15:03.070
Where let's say the suffix or the code before where your cursor is has changed slightly, they might not a-actually go ahead and use a different, um- That seems dangerous for code.

99
00:15:03.370 --> 00:15:11.310
It does seem a little dangerous, but it gives you this, like, incredibly snappy response. And the other thing is the KV cache, right? Where you can just...

100
00:15:11.670 --> 00:15:19.990
I don't think there's any open source framework that does this right now, but what you can do is if you've already computed something over the KV cache, then you can just- This is the attention KV cache, right?

101
00:15:19.990 --> 00:15:23.390
Exa- Yeah, yeah. For people following. So this is the attention KV cache, right?

102
00:15:23.430 --> 00:15:31.870
And if you've already computed all the keys and values, you can just store that in memory and then load that back up in the GPU, and you don't need to process the prompt again.

103
00:15:32.290 --> 00:15:41.890
And I speculate they're, they're doing something like that behind the scenes. That's a lot of memory. [laughs] That's a lot of storage. It is. Well, unless they're using something like multi-query attention or... Yeah.

104
00:15:42.070 --> 00:15:48.790
We'll talk about that- Yeah... uh, in your Llama 2 piece. And then the final big opinion that you drop in there was you must write your own IDE- Yes...

105
00:15:48.850 --> 00:15:58.050
as opposed to write a VS Code extension, which there's plenty of them out there. Sourcegraph is doing one, um, and I've been working closely with, uh, Morph- Mm-hmm... uh, which, which just put out Rift. There's a bunch.

106
00:15:58.290 --> 00:16:10.970
Mm-hmm. So this is obviously a big undertaking. Maybe explain more a little bit about why, why build your own IDE. Yeah. The reason we decided to do this is I think in the future...

107
00:16:11.330 --> 00:16:18.330
Today, what Cursor can provide and what any of these tools can provide isn't that much different than, I guess, what you get in VS Code.

108
00:16:18.670 --> 00:16:27.550
But it was more of a long-term decision, where in the long term, you're going to need to design just a very different UX that kind of the, the extensions don't give you.

109
00:16:27.870 --> 00:16:36.690
Like, one story we'd heard is that with Copilot, in order to actually get the multi-line ghost text implemented, it wasn't actually a part of the extension.

110
00:16:37.270 --> 00:16:51.450
I think the team at GitHub had to call up VS Code and have them make a change to the source in order for that extension API to be enabled. That's what allows for multi-line ghost text completion, and this is scary.

111
00:16:51.770 --> 00:16:59.190
If you look today, there are other things that VS Code, in their source code, has enabled as APIs that are just closed off to everyone but Copilot.

112
00:16:59.710 --> 00:17:09.210
So I think there's this fundamental platform risk where you're competing with the incumbent that owns the platform you're building on, and we thought it would just not really be tenable in that sense.

113
00:17:09.290 --> 00:17:17.310
And then the other thing is if you wanna do other kind of fancy things... So one example of a feature we're kind of building right out right now is instead of just...

114
00:17:17.470 --> 00:17:31.270
So Copilot's great for compu- completing the next line, completing the next few lines, but what if you wanted to do a kind of sort of edit, right, where instead of just completing this line, it, like, changes the line above or deletes something, right?

115
00:17:31.310 --> 00:17:39.650
There's no way that you can do something like that in VS Code, but we have the UI for this that we've kind of built out in Cursor. We're currently training models in order to get it to work well.

116
00:17:40.190 --> 00:17:49.930
But again, this is a feature that we think once we get it to work will be quite useful, could be on par with Copilot level, um, in terms of usefulness, and it's just fundamentally impossible unless you own the IDE.

117
00:17:50.390 --> 00:17:54.310
There are a lot of other ones like those that we're kind of cooking up. A-and then there are small things.

118
00:17:54.690 --> 00:18:09.790
I do think, like, in terms of inline edits, which means inside the editor making a, just you can press Command K in Cursor and then ask for some kind of modification to the code or ask for a generation of the code, and I do think we have probably the best UX for that.

119
00:18:09.830 --> 00:18:20.190
'Cause if you look at what someone like Sourcegraph does, I mean, Sourcegraph Code is a great product, but, like, they basically have to use the GitHub pull request comment feature in order to do it.

120
00:18:20.710 --> 00:18:27.490
And I think, like, these paper cuts kind of add up over time. They do, and it's very impressive, uh, how quickly you can try it out.

121
00:18:27.690 --> 00:18:38.410
You know, I obviously encourage everyone listening to try out a Cursor and, and the download is really quick, the binary is super small, and then when you spin it up, it, like, boots up really fast, and it's just a text file that guides you through the tutorial.

122
00:18:38.450 --> 00:18:46.210
It's really, it's really great. Yeah. I was using it today, actually, I have it open right now. The first thing I like, you guys have, like, bring your own keys.

123
00:18:46.690 --> 00:18:54.650
So that's, like, one of the things that I don't see in enough products, like, bring your own API key- Yeah... instead of, like, sign up for an account and, and do all of that. Yeah. Well, so- Um...

124
00:18:54.670 --> 00:19:05.730
like, you have to trust them that they won't- Look, look at this guy. I know. [laughs] This guy. I just wonder if, like, OpenAI could do one more thing, which is just, you know, do a limit, the spend limit per key. Mm.

125
00:19:05.770 --> 00:19:10.610
Right? Yeah. So, like, that, that leaves space for, like, other companies to come in and do that, but, I mean, OpenAI could just build it tomorrow. Yeah.

126
00:19:10.690 --> 00:19:16.730
I saw Logan tweeted about whether or not it would be interesting to have per-key billing. Yeah. I mean, they're clearly thinking about it. You know, I think that would be- Yeah.

127
00:19:17.010 --> 00:19:24.970
That would be- They have more important things, like GPT 4.5. [laughs] We can talk about that one. Yeah. [laughs] Uh, yes. Let's talk a bit about what you do.

128
00:19:25.030 --> 00:19:33.790
So first of all, unlike some of the other tools, you guys have, like, a system prompt kind of thing- Yeah... which are like rules for AI. What was, like, the decision behind that?

129
00:19:33.890 --> 00:19:43.650
Did you see people being frustrated with always having to repeat the same thing in the prompt? Yeah. So this was for encoding some small rules that the model will tend to get wrong.

130
00:19:43.710 --> 00:19:52.670
So for example, we use Solid instead of React, and so- Oh, right... yeah. Quick, quick two seconds and... [laughs] So yeah, Solid is just another reactive UI framework.

131
00:19:53.210 --> 00:19:59.130
It's a decent bit faster and, um, my co-founders, uh, know a lot more about the details in this than I do.

132
00:19:59.490 --> 00:20:08.510
But, like, the, the other really nice benefit is that with the VS Code fork that we're, we're using, you can kind of inject Solid into multiple roots.

133
00:20:08.890 --> 00:20:19.690
While React is meant to be kind of like it takes over the very root of the entire DOM. Solid instead, you can inject it inside of, like, multiple HTML components. It's much more performant that way.

134
00:20:20.030 --> 00:20:32.070
And so yeah, we, we use Solid because of that, and then the issue is every time you create a TSX file, write a component, GPT-4 by default will assume it's React, right? And so it'll get the code wrong.

135
00:20:32.110 --> 00:20:35.750
And so just encoding rules like that are pretty helpful on the side of that problem.

136
00:20:35.790 --> 00:20:45.070
For some of our users who are less familiar with English too, it's, it's helpful to kind of add a prompt to say, "Describe this in whatever language they're most comfortable with."

137
00:20:45.390 --> 00:20:55.750
And so for those who don't know, you primarily... the main model for most people is GPT-3.5. Mm-hmm. And Pro users can, can use GPT-4. You're prompting GPT-3.5 with these system prompts- Mm-hmm... first.

138
00:20:56.090 --> 00:21:04.250
Any other tips apart from your, like, your company-specific ones, apart from the English as a second language ones, how do you prompt GPT-3 or 4 for code?

139
00:21:05.130 --> 00:21:16.830
So this is interesting, 'cause I think in general, these models are good at just producing net new code or rewriting code from scratch. The thing that they're not great at is producing edits or modifications.

140
00:21:17.250 --> 00:21:27.610
So producing a diff is incredibly painful, and I'm sure you guys may have encountered this if doing stuff with agents, but they just get line numbers wrong pretty often.

141
00:21:27.930 --> 00:21:39.230
And when you're producing a diff, you know, it's fewer tokens of compute, and there's, there's some theories that, like, you know, the more tokens of compute you kind of use up, the more the model is kind of expending on thinking, um- Chain of thought...

142
00:21:39.250 --> 00:21:40.710
quote, unquote, thinking. Yeah, chain of thought.

143
00:21:41.250 --> 00:21:58.150
That's one thing we kind of struggle with, and so that takes probably chaining to get it to work well, where o- one kind of technique we do is we have GPT-4 kind of propose a d- a draft PR, and then we have 3.5 go and kind of heal a draft, uh, diff, and then we have 3.5 go and, go and heal the- those changes.

144
00:21:58.450 --> 00:22:09.490
So you'll have to do things like this in order to, to get it work around those limitations with edits. In terms of general code writing, I think with 4 it's just, it's been super, super straightforward. 4 is fantastic.

145
00:22:09.670 --> 00:22:24.910
3.5 would strongly recommend using the Azure model because there you get access to completions, meaning you can put kind of words in GPT-3.5's mouth and let it, let it finish it, kind of like what you, you can do with Claude, and that's really helpful.

146
00:22:25.050 --> 00:22:32.910
I always assumed that was going away as a API 'cause OpenAI is, like, clearly not interested in maintaining that. I mean, they're, they're straight up deprecating it now. Yeah.

147
00:22:33.370 --> 00:22:38.270
It's a little frustrating because- [laughs]... I think, I think it's really useful for code, right?

148
00:22:38.330 --> 00:22:45.409
'Cause when you can do stuff in the middle of the line, it's impossible to do that with the chat format, but with the completion format, it becomes trivial. Yeah.

149
00:22:46.010 --> 00:22:55.930
So one thing I learned from working with Jesse on GPT-4 OpenAI- Mm-hmm... he always asked GPT to comment your code before writing the code, and that's the chain of thought- Yeah... for code, right?

150
00:22:56.050 --> 00:23:04.070
So when I ask you for code, give me fully commented code with only a brief explanation of how, on how it works, bias towards the most efficient solution, and offer an alternative implementation if it fits.

151
00:23:04.470 --> 00:23:10.190
If it's unclear what environment or library versions I'm working with that might significantly change your answer, please ask me to clarify. Hmm.

152
00:23:10.230 --> 00:23:18.130
That's my custom instructions right now for, for code, and I'm just like, "Hey, we should come together as a community and just share these custom instructions or system prompts." Yeah.

153
00:23:18.170 --> 00:23:27.110
The thing is, when you get it to be more, more verbose, I do worry a bit bec- in terms of UX 'cause more tokens means it takes a lot longer to get to the answer- Speeds everything, right...

154
00:23:27.150 --> 00:23:36.050
and then it's also just I don't wanna read a massive answer. I just often want the answer immediately, or I just want kind of a short block of code to, to, to answer that.

155
00:23:36.350 --> 00:23:46.750
That is a trade-off you kind of h- will have to deal with. And the same thing with diffs, right? Where the diffs are gonna be so much faster if you get them to work, but it's just gonna result in lower quality edits.

156
00:23:47.310 --> 00:23:52.010
One nice thing you do in the chat is actually remove some of the code you don't touch.

157
00:23:52.490 --> 00:23:59.710
So I was using it to make some changes to the code base, and in each function it would say, like, add a comment with, like, existing code- Mm-hmm...

158
00:23:59.770 --> 00:24:09.970
and then tell you just the stuff to change and the stuff to, to add to it, which kind of frustrates me with GPT-4 sometimes is like it just re-gives you the whole function- Yeah... definition instead of just that.

159
00:24:10.470 --> 00:24:18.410
I noticed that in the chat you cannot, like, apply change, you know- Mm-hmm... and put it into the code if you don't start- Mm-hmm... the conversation from the file itself. Mm-hmm. Why is that so hard?

160
00:24:18.750 --> 00:24:26.310
Like, so many products have it. Is it actually hard, or is it just, like, a UX decision to have you- So, uh, uh, so there are two ways of, of doing that, right?

161
00:24:26.370 --> 00:24:36.750
So when you say apply change, do you mean you select a region in the code, press the button, and then it makes it just, like, pasted in? Yeah, like right here it told me to, like, add these three lines of Python. Mm.

162
00:24:36.770 --> 00:24:42.850
And I'm like, "I don't wanna copy paste them." You know, I did it, but- Yeah. Yeah, yeah... it would be good to just to- So if it just makes the change- Yeah... for you.

163
00:24:43.230 --> 00:24:51.610
Yeah, this is something we're, we're gonna be adding, um, this week. So yeah, this is definitely, like, something a lot of users have asked for, and it should be reasonably straightforward to do.

164
00:24:52.150 --> 00:25:02.738
I think the issue is we want to use 4 sparingly because of how expensive it is- Yeah... and 3.5 actually kind of struggles with this. Hmm, interesting. And then I noticed, so you can co-...

165
00:25:02.818 --> 00:25:11.078
chat either with or without context. Mm-hmm. So with context, you pass it parts of your code base. Without, you don't. Every time it loads the license file.

166
00:25:11.658 --> 00:25:17.498
So is there anything that you're working on to make sure that, like, you don't have, like, license infringement and stuff like that?

167
00:25:17.578 --> 00:25:25.718
Or is it just, like, the model for some reason thinks the, the license file is really important? Yeah. Right now it probably uses vanilla embeddings.

168
00:25:25.758 --> 00:25:35.177
We're working on a couple of interesting techniques for, for much better retrieval. One of them is basically fine-tuning a model to kind of memorize a code base.

169
00:25:35.258 --> 00:25:43.898
So there's a paper that came out a little while, a while ago from Google, which is called Documents, or it's called Transformers as a Differenciable Search Index.

170
00:25:44.378 --> 00:25:58.118
The idea here is you train a transformer on a code base, or you train it on a corpus of documents in order to basically directly answer questions about which document is relevant given the question.

171
00:25:58.218 --> 00:26:12.858
So the mapping would be some query, some question, and in this case, a question about a piece of code, and then the model would directly output the not just file, but let's say the actual function or the class that, that solves it.

172
00:26:13.198 --> 00:26:20.718
It wouldn't output all the code for it, but it would only just output, like, the symbol that corresponds to it. And we've seen some initially promising results with this direction.

173
00:26:20.778 --> 00:26:30.278
If you look at the original paper and then there are some follow-on work, it actually does a lot better than very old-school retrieval techniques like BM25 and even embedding-based techniques.

174
00:26:30.578 --> 00:26:36.738
And so this is an approach we're experimenting with, and we think it could prove quite helpful. The other direction is just improving embeddings.

175
00:26:37.258 --> 00:26:51.378
If you looked at the recent paper by, um, I think it was Alibaba, so there was a recent model, if you do the math, it costs them $1,000, less than $1,000 to train this thing, and it beats OpenAI on non-code related tasks.

176
00:26:51.458 --> 00:26:57.078
Sadly, not in code related. [laughs] OpenAI still kind of own- holds, holds the crown for code rel- code related embeddings.

177
00:26:57.598 --> 00:27:04.858
But we think there's, there's some promise in potentially training our own embeddings and then fine-tuning it on particular code bases so it performs better there.

178
00:27:04.898 --> 00:27:16.278
So these are both directions we're kind of independently exploring to improve the performance of, of retrieval. But in the short term, we do have the ability to use kind of re-rankers and more kind of advanced tuning.

179
00:27:16.358 --> 00:27:25.518
So if you look in the chat, I think there, there may be, but you can click, which lets you enable re-rankers- Mm... which should improve the performance a decent bit. Awesome.

180
00:27:25.578 --> 00:27:35.958
Anything else in the product that we're missing? We have inline generation, kinda inline question asking- Mm-hmm... to the model. You have the chat interface on the right. What else should people really dive into? Yeah.

181
00:27:36.018 --> 00:27:43.498
So one thing that our users have found quite helpful is being able to add files or add documentation.

182
00:27:43.718 --> 00:27:56.098
So if you want to add Next.js docs, the most recent docs, you can just do @Nextjs in the chat or in, in Command K, and you'll be able to then basically get that information in your context.

183
00:27:56.518 --> 00:28:07.658
We've a lot of features that will be coming up quite soon. One that we're quite excited about is basically code interpreter style mode of using, using the chat.

184
00:28:07.938 --> 00:28:10.878
And so I don't mean that, I guess, in the traditional sense of code interpreter. Mm-hmm.

185
00:28:11.278 --> 00:28:20.498
But code interpreter is probably the one example of, as far as I know, the one example of an agent that works really well, that has some sort of kind of product market fit, right?

186
00:28:20.918 --> 00:28:33.678
And I think the reason it works super well is because when you try to get agents to do some massive task, I don't know, many people who like reviewing PRs or reviewing large diffs, it's much more fun to kind of be in flow.

187
00:28:34.138 --> 00:28:43.438
And I think the way that the code interpreter is able to deal with this is it breaks it down to these kind of small units that are very auditable and understandable.

188
00:28:43.878 --> 00:28:53.218
When you ask the model to produce a graph, you just see the graph, and then you can kind of tell more or less it's wrong, and then you can go and see the code, and the code's very understandable.

189
00:28:53.258 --> 00:29:04.398
So I think it's pretty important to kind of have the agent do these very small discrete units and then show the output in a way that's very easy for the user to understand and then go in and fix.

190
00:29:04.858 --> 00:29:10.758
And so we're building a kind of flow like that in the chat that should be coming out in the next two weeks, which we're ve-very excited by.

191
00:29:10.818 --> 00:29:20.698
Because we've done a bunch of experimental stuff with agents, and the big thing has always been this problem where it just produces a bunch of code, and it's just so hard to tell whether or not it's correct or not.

192
00:29:20.738 --> 00:29:29.678
And it's, it's, it's less efficient because it'll end up having some bugs, and then it would've been better if the user just went and wrote all of it, all of it themselves. Yeah.

193
00:29:30.078 --> 00:29:42.558
There's one approach with a former guest of ours, uh, Itamar on Codium, whose essentially approach is you need to develop the spec, the tests, and the source code in harmony, kind of, uh, together.

194
00:29:42.618 --> 00:29:53.878
Like, well, the spec is the prompt, and then the spec could generate a test or a spec could generate code, and the only way to validate the code is to run it with tests, is kind of his analysis of what the agent space, what the code agent space may look like.

195
00:29:54.258 --> 00:30:04.578
I think tests are pretty promising, uh, direction. If you have a really, really rigorous set of tests where you can completely confirm whether or not the agent has done the right thing, I think that solves it.

196
00:30:04.598 --> 00:30:10.218
But I think it's only one part of, of the overall puzzle here. I do think you're gonna want the model to...

197
00:30:10.378 --> 00:30:17.018
Like, the issue is it- it's really kind of painful to go and write this massive, massive prompt describing everything.

198
00:30:17.178 --> 00:30:22.278
I wanna be able to kind of do it in flow and just, like, see a change, then go step by step from there.

199
00:30:22.658 --> 00:30:30.318
I think that's just a more fun way of doing it, and I think the more fun and more easy to use kind of product will win, assuming the capabilities are, are about equal.

200
00:30:30.758 --> 00:30:37.098
So that's kind of, yeah, that's kind of our bet here. Yeah. That's great. Have you thought about, like... So you, you said you can, like, add docs- Yeah...

201
00:30:37.138 --> 00:30:45.618
which is really cool, and I've thought about this before, but, like, I always get hung up on versioning. Mm. Do you just choose to not care about it and just, just embed the, the most current docs? Yes.

202
00:30:45.678 --> 00:30:54.258
We embed the most current docs. You can, um, add whatever docs you want if you just have a URL. Ah. We... You can paste the URL for the docs in. We- You give a crawler. Yeah...

203
00:30:54.518 --> 00:31:04.738
we crawl it in the background, um, and embed it. And so you can have a custom, basically a custom version or whatever version you use- That's wild... um, kind of- Well, I mean-... it's stored locally for you...

204
00:31:04.798 --> 00:31:12.282
what kind of crawl diff? Like, the, if you've just wr- like, yeah, that means you've written a search engine Kind of. Ah, it's, it's very, very basic. [laughs] Yeah. It's...

205
00:31:12.462 --> 00:31:20.142
Docs are very, very easy to crawl relative to other things because it's like- The structure... they're all like this kind of sort of markdown-like format. Yeah, yeah.

206
00:31:20.182 --> 00:31:29.282
Definitely have not written a, a crawler for the entire, the entire net. The other thing on Code Interpreter, we've also done an episode on that. Mm-hmm. I'm very excited about it. I think it's GPT-4.5, you know- Mm...

207
00:31:29.322 --> 00:31:38.242
officially, because it's GPT-4 that has been trai- fine-tuned on more code. Yeah. Plus it has inference time capabilities that you cannot do in the traditional LLM setting.

208
00:31:38.422 --> 00:31:41.501
Anyway, the most important thing about GPT-4 is that it has the sandbox.

209
00:31:41.942 --> 00:31:52.542
So main question for you is, are you gonna run the sandbox in your environments or do you want to run it on our local machine since you have access to that too? Yeah, I think we wanna be very careful with this. Yeah.

210
00:31:52.562 --> 00:32:02.182
You don't wanna do rm, pseudo rm- [laughs]... dash RF star or something. Our plan is to run it on the local machine, but always kind of prompting the user whether or not they want it.

211
00:32:02.722 --> 00:32:06.282
I think if we want to do things where the agent takes many...

212
00:32:06.342 --> 00:32:11.942
So for the Code Interpreter style thing, the great thing is because you're breaking it down to these units, you can kind of batch together a bunch of commands.

213
00:32:12.382 --> 00:32:15.502
At each step, just kind of ask the user because they're always kind of watching.

214
00:32:16.022 --> 00:32:25.042
For agents that are running completely in the background, I think there you probably will need to have some kind of contained environment where it's safe for agents to execute arbitrary code.

215
00:32:25.482 --> 00:32:43.462
One pretty bad attack is if one team wanted to, let's say, prompt inject the model, they could just kind of in a piece of code, just like have a comment that said something like, "Okay, when you're doing this kind of edit, um, you should do rm dash RF," or do something like really, really dangerous, right?

216
00:32:43.842 --> 00:32:52.982
And then the issue is if an agent's kind of running in the background and then it does that and it, it grabs that piece of information and then it gets like actually successfully prompt injected, it'll just execute that thing.

217
00:32:53.022 --> 00:32:56.322
The same actually may be true with documentation, where someone could...

218
00:32:56.902 --> 00:33:06.042
Someone malicious, if they had access to some piece of documentation that other people use, could try to prompt inject agents that are then going and running code and running terminals, uh, terminal commands.

219
00:33:06.442 --> 00:33:13.862
Today people just hijack NPM packages. [laughs] Yeah. Yeah, there'll be more, uh, of that, I'm sure. Shenanigans- Hmm... as they call it.

220
00:33:14.142 --> 00:33:21.942
But yeah, I think probably the safest way is to have sandboxes in the cloud and- Yeah... uh, I've been calling this the sort of the agent cloud phenomenon.

221
00:33:22.002 --> 00:33:33.342
I think Fly.io, Mo- Model, uh, and E2B are in that space already. And then I think Re- uh, uh, Replit is, uh, is exploring it. Yeah. Um, it'd be interesting for you guys to get in that game.

222
00:33:33.642 --> 00:33:41.602
I have trouble articulating what's different about an agent cloud versus like a typical serverless sandbox- Mm-hmm... thing that you can spin up. Basically, I think for people to...

223
00:33:42.002 --> 00:33:49.222
If agent cloud is, is like a real category, we have to give... We have to identify what kinds of feedback do we wanna give the AI that's different to a human.

224
00:33:49.682 --> 00:34:00.102
That's my, that's the extent of my thoughts on like what this would take. I think like the key thing that not enough people are probably doing is giving the AI access to like a lot more tools.

225
00:34:00.482 --> 00:34:07.062
So like the classic example I, I like to, to bring up is if you look at the old kind of AlphaCode model- Mm...

226
00:34:07.242 --> 00:34:15.482
which went and got 50% on some programming contest, a competition, 50th percentile of, you know, pretty good programmers, right?

227
00:34:16.102 --> 00:34:22.342
This was a, a base model that basically got, I think, something around 28% on HumanEval, right?

228
00:34:22.482 --> 00:34:31.302
And they used this interesting inference strategy of having the model generate a bunch of test cases and then running the test cases, seeing which one passed, and then u- they use some other like...

229
00:34:31.342 --> 00:34:39.522
There's other, some other details there where they do clustering and whatever. But, um, the key thing is kind of letting the model generate tests, run the tests on all the outputs that it's generated.

230
00:34:39.802 --> 00:34:53.462
And that brings a 28% Codeforces model to 50th percentile. GPT-4, you just add like a very basic prompt, "Please complete this Python function," and it gets, you know, 85%, 87% on HumanEval.

231
00:34:53.802 --> 00:35:04.762
Now, who knows how tainted that benchmark is. [laughs] But assuming it's reasonable, like what score do you think GPT-4 with the same kind of inference strategy as AlphaCode would get on that benchmark?

232
00:35:04.842 --> 00:35:05.902
It would do really well.

233
00:35:06.042 --> 00:35:19.542
And then GPT-4 is at this level where it can actually not just run the test and like binary yes or no use that answer, but it would see the results of the test and be able to modify the code or the test based on those.

234
00:35:19.902 --> 00:35:25.861
And so I think just like that's just one tool. The other tools you could have access to would be language servers.

235
00:35:25.902 --> 00:35:31.562
So this is a great thing with VS Code, where VS Code kind of invented the language server or the language server protocol.

236
00:35:32.002 --> 00:35:45.442
And so as a result, when working with a VS Code 4, we kind of have access to every single part of the language server protocol, which means we can go to definition, get all the symbols in your entire workspace, kind of everything you do in a modern IDE.

237
00:35:45.962 --> 00:35:52.522
And what we've been working on is kind of giving these models access to those tools, and that like dramatically improves performance, right?

238
00:35:52.562 --> 00:36:00.802
'Cause the way that humans usually will search for something is they'll kind of click around, go to definition, read some code, do all that. But you use the tools in the IDE to search for things more efficiently.

239
00:36:01.282 --> 00:36:10.362
And if you're just trying to have a model just do a brute force kind of semantic search and then get the answer from that, I think it's not gonna work nearly as well as kind of an agent that's able to use those tools.

240
00:36:10.922 --> 00:36:23.042
Awesome. And you guys are growing the team right now, I would assume. Yes, we are. Um, so we are currently five people based in SF, and we're looking to hire engineers and designers.

241
00:36:23.222 --> 00:36:27.042
We think there's a lot of interesting work that we're doing that's left to be done.

242
00:36:27.282 --> 00:36:39.262
So some of it involves model training, kind of training some open source models for things like embeddings or areas where it perhaps is too expensive or not, uh, or too slow to use OpenAI.

243
00:36:39.682 --> 00:36:50.822
And then lots of interesting things with pushing these models to kind of the boundaries. So getting GPT-4 to work really well in this kind of agent loop in a way that's really in flow and intuitive for users to use.

244
00:36:51.362 --> 00:36:59.822
So yeah, lot... I think lots of exciting work. Cool. Um, and then maybe to sketch out a little bit more about the company- Mm-hmm... and then we'll zoom out to just general LLM observations.

245
00:37:00.222 --> 00:37:06.882
You're also working on a prompt tooling thing called Priompt? Yeah. So this is just an internal tool that we use.

246
00:37:07.442 --> 00:37:21.450
It's called Priompt, and we built this because we didn't really find a good way of solving for the problem of when you have a variable number of kind of inputs that you wanna stuff into the prompt and you have like a fixed length...

247
00:37:21.490 --> 00:37:29.370
prompt, right? You can only use 4,096 tokens. Like, how do you encode for rules on how to properly kind of order the inputs that go into it?

248
00:37:29.890 --> 00:37:45.230
And preempt or priority prompting are intermediary solution for this, where you can kind of encode very custom rules into how you build up the prompt based on how, I guess, overflowing it is, right? So let's say

249
00:37:46.210 --> 00:37:55.930
I have a bunch of previous chat messages, and then I also have the code from the current file. So maybe what you wanna do is you want some rules where if everything can fit in, you put it all in.

250
00:37:56.010 --> 00:38:05.490
But then you start by, like, first removing, like, all the old chat messages. Then once it gets to a certain length, you don't wanna remove any more chat messages, and you wanna start removing parts of the file.

251
00:38:05.530 --> 00:38:10.270
And then you wanna remove parts of the file in this particular truncation strategy, which tends to work quite well.

252
00:38:10.670 --> 00:38:18.510
So this kind of thing where, like, as you kind of slide the window on how many tokens you're allotted, you can see, like, the prompt is, like, very, very differently constructed.

253
00:38:18.570 --> 00:38:27.070
So it's, like, optimal kind of at all sizings. And we found that quite helpful internally. There's... And you chose the JSX approach. Yes. Uh, now I'm not gonna ask you too much about, like, design choice.

254
00:38:27.130 --> 00:38:35.910
I mean, it's popular with React. Mm-hmm. Uh, Fixie has also put out AI JSX. Mm-hmm. Do you find that, like, helpful? Do you think that, like, some kind of DSL might emerge for prompting?

255
00:38:36.790 --> 00:38:46.270
Yeah, I think it's still pretty early, and it's not clear what the best way to do. I think for very lightweight, easy prompting, like, you should just use strings.

256
00:38:46.270 --> 00:38:56.410
[laughs] When you're doing kind of prompt engineering and really, like, rigorous prompt construction, where you can have a bunch of different possible inputs in the prompt, we think JSX makes a lot of sense.

257
00:38:56.930 --> 00:39:07.810
It's because it's kind of like website development, where you'll have different kind of screen sizes on different kinds of devices that can look at it. And in a similar way, you have different kinds of prompts, right?

258
00:39:07.850 --> 00:39:18.890
You have different... Or prompt context lengths. And you basically want, across all different context lengths, context le- context lengths to get a very, very good prompt for the model.

259
00:39:19.210 --> 00:39:27.150
And so yeah, that's why I think, like, JSX kinda makes sense. It, it's not clear if it, it, if it is, like, the best way of doing it. I think the jury's still out on that. Yep, yep.

260
00:39:27.650 --> 00:39:38.210
One way to deal with the context length model issue is to train your own model that has a very long context length- Yeah... like Magic.dev, which is doing- Mm... which announced a 5 million context length window.

261
00:39:38.470 --> 00:39:50.450
I don't know how credible that is. [laughs] I haven't tried it. But, uh, your thoughts. Yeah. I think the issue with context length, long context length right now is that costs scale linearly, right?

262
00:39:51.070 --> 00:40:09.270
Costs technically scale quadratically in terms of attention, but the interesting thing is that for, for really, really large models, in terms of FLOPS or actual floating point operations that the models are doing, attention tends to be a pretty negligible part compared to the actual, I guess, feed forward part of the, the neural network.

263
00:40:09.650 --> 00:40:13.050
And so up to, like, 8K, it tends to look pretty linear.

264
00:40:13.590 --> 00:40:22.870
I guess when you're going to, like, higher and higher context lengths, it starts to get more and more tricky, and then there are some other optimizations or some other difficulties with memory bandwidth that we can get into.

265
00:40:23.010 --> 00:40:36.010
But yeah, it just, it just feels like the key issue is even if it is linear, it's still so expensive, right? Paying for 32,000 tokens at whatever the pricing is right now feels, like, exorbitantly high.

266
00:40:36.930 --> 00:40:47.250
My perspective is that there probably will be, at some point in the future, or there might be at some point in the future, like, a better approach for really, really long context, something that looks more kind of recurrent.

267
00:40:47.750 --> 00:40:55.050
It feels more elegant. I don't know if it'll happen, 'cause I think there are, like, interesting ways of hacking together or chaining together these language models, even with short prompts.

268
00:40:55.570 --> 00:41:04.450
But I'm not super bullish on kind of scaling up attention, um, the way that we're doing right now in, like, 100, 200K context windows. Like Claude is doing. Yeah.

269
00:41:04.750 --> 00:41:12.150
Are you monitoring, like, RWKV, which is one of the- Mm... recurrent, uh, approaches? I've been meaning to read that, that paper. Okay. I have not been monitoring that.

270
00:41:12.530 --> 00:41:19.790
I looked into a few of the papers from state space, like the state space- State space models... based models. Yeah. Yeah, those are pretty interesting. Can you give an intuition?

271
00:41:19.910 --> 00:41:27.050
'Cause you seem to be explaining it really well. Like, why are they different? Why, why is that interesting? Yeah. I think the interesting thing with, at...

272
00:41:27.150 --> 00:41:39.890
So at least with the original state space model, is that you get kind of two benefits. One, for training, you get the parallelizability of a transformer, and you can kind of run it, I believe in about N log N- Yeah...

273
00:41:40.150 --> 00:41:46.230
um, for some N length sequence. And then for inference, it's also, like, the way that it's formulated is also somewhat recurrent.

274
00:41:46.310 --> 00:41:52.650
So you can kind of store everything in this fixed state, and then because of that, you get, I believe, an O of one kind of cost towards inference.

275
00:41:52.710 --> 00:42:02.670
It could be slightly higher, but yeah, it's, it's much less than the O of N cost per token for the transformer. And so that makes it really tractable to then do for, for very, very long sequences.

276
00:42:02.730 --> 00:42:12.830
There's some follow-on work with, um, Hungry Hungry Hippos and Hyena. And again, I think the key piece is that for, like, very, very long sequence lengths, it ends up being N log N rather than N squared.

277
00:42:13.270 --> 00:42:23.830
'Cause I did say that the cost of, like, I guess even linear attention is pretty high, but that's because, you know, the 32K model is priced, you know, a decent bit higher than the original.

278
00:42:24.190 --> 00:42:34.870
It is surprising that Claude, uh, or... I'm actually not familiar with Claude's pricing. Is it higher for the 100K one than for the normal? No, I believe it's the same. Yeah, that, that is actually quite surprising.

279
00:42:34.950 --> 00:42:45.170
[laughs] I'm not sure if they're doing attention under the hood, 'cause even with, like, a lot of tricks, with 100K or even 200K, I would assume that cost will eventually start to build up.

280
00:42:45.210 --> 00:42:56.010
So they might be doing something fancy there. Well, my guess was Alibi, which is a trick which is replacing proper attention with kind of like a exponentially declining, uh, forgetting curve- Mm...

281
00:42:56.110 --> 00:43:04.430
is what I'm thinking about. Oh, interesting. Someone has to put the 100K to the test. I haven't done it. [laughs] Yeah, they have this, they have this graph that looks promising for- Yeah... 200K. Yeah.

282
00:43:04.770 --> 00:43:11.470
But I feel like anecdotally, from everything that I've heard, it just seems like it forgets things. Like they don't actually pay attention to things. Yeah.

283
00:43:11.530 --> 00:43:18.770
I just opened source small podcast there yesterday, which is what we used for- We used it to summarize this podcast. Yeah, yeah. Oh, wow. Like, I'm, I'm, I'm looking at- Appreciate it... my logs.

284
00:43:19.090 --> 00:43:28.034
Prompt length of all my recent ones is 55,400. Tokens And how much? And it works How much per, how much per call? Free, because it's not commercial. Oh, yeah, yeah, yeah.

285
00:43:28.034 --> 00:43:38.114
[laughs] And I'm like, uh, hopefully nobody from Entropic is listening, but, but yeah, it works. But I, I think that's kind of, like, the sweet spot. And then the completion length is, like, 1,800, you know? Yeah.

286
00:43:38.114 --> 00:43:44.514
So it's not like... It stays within, like, the 60K band. But anyway. Yeah. Yeah, yeah. Yeah, curious to see.

287
00:43:44.554 --> 00:43:54.634
And I think another thing from your, uh, Twitter model threads that I really like, [chuckles] uh, is actually differentiating between the type of workload, you know? Yeah.

288
00:43:54.654 --> 00:44:07.314
I feel like people talk about these models as, like, anything you do is, like, the same thing. But your post said about GPT-3.5 being cheaper than Llama 2- Yes... for completion-heavy workloads. What does that mean?

289
00:44:07.854 --> 00:44:12.574
Yeah. So there are different terms, I guess, based on, like, whatever community you're in.

290
00:44:12.694 --> 00:44:22.454
So I think in the research co- community, they probably call it pre-filling, is handling prompt tokens, and then I believe maybe decoding is what they call generating completion tokens.

291
00:44:22.994 --> 00:44:25.234
We'll just use prompt tokens and completion tokens. But

292
00:44:26.154 --> 00:44:43.614
for prompt tokens, the work, it's entirely compute bound, and the reason why is, is the same reason why transformers are so good at being, at, at kind of being trained in parallel, and it's that you can paralyze the entire sequence, or you can paralyze an input, not just along the batch dimension, but the sequence dimension.

293
00:44:44.074 --> 00:44:50.034
So that means let's look at the first layer of the transformer. Imagine, like, that entire layer could fit in memory.

294
00:44:50.174 --> 00:44:58.414
I just read that to memory, and then I basically apply a mu- the matrix multiplication of the entire sequence on this layer.

295
00:44:58.974 --> 00:45:10.254
If you're doing token generation, instead you have to read the layer, then take in the first input, and then you have to read the next layer and then do that same input.

296
00:45:10.294 --> 00:45:17.374
You have to do it all the way to the end of the, the model, and then you generate the next token, and then that next token passes through all the layers again. Mm.

297
00:45:17.414 --> 00:45:26.314
So before, what you were doing is you have all your input tokens. In parallel, they're going through the first layer, so you read the first layer. Then in parallel, they're going through the second layer.

298
00:45:26.374 --> 00:45:29.454
You read the second layer. So on and so forth till the end.

299
00:45:29.674 --> 00:45:37.794
But when you're doing it for one token at a time, you read the first layer, second layer, third layer, fourth, blah, blah, blah, then you do it all over again for the next token.

300
00:45:38.194 --> 00:45:46.334
And so as a result, for your sequence length N, you end up using N times more memory bandwidth than compute. And time as well, like wall clock time.

301
00:45:46.794 --> 00:45:57.794
Yeah, I mean, so with wall clock time it's weird because transformers are far more compute efficient than- I comment on that because in the RWKV interview that I did- Mm-hmm... uh, same thing.

302
00:45:57.814 --> 00:46:05.414
They have a visual actually of- Mm... of this. So the thing you're trying to describe with words, they actually have a visual and an animation. [laughs] Yeah, it's hard to... Yeah. [laughs] But it's helpful. It's, it...

303
00:46:05.434 --> 00:46:12.594
Because o- once you see it, you're like, "Oh, okay, that's why." It's, like, a different graph. Yeah, exactly. [laughs] Yeah, so when you're dealing with the prompt, it's completely compute bound.

304
00:46:12.614 --> 00:46:21.414
And because GPUs can handle some crazy number of floating-point operations per second, it's, like, almost instant. That's why time to first token feels super instant.

305
00:46:21.794 --> 00:46:33.914
And then when you're generating one token at a time, it now becomes completely memory bound, where for each token, you're bound by how fast you can read all the weights into memory, and that's, like, a- around, like, 200X slower in general.

306
00:46:34.274 --> 00:46:41.254
Yeah. So your, your specific recommendations, which I pulled out from the post, uh, people should read it. It's, it's really good. I feel like the title undersells it a little bit. Yeah.

307
00:46:41.274 --> 00:46:49.774
[laughs] Um, you should not serve Llama 2 for completion-heavy workloads. Llama is best for prompt-dominated tasks like classification and, like, I feel like I can ru- I can run with that.

308
00:46:49.814 --> 00:46:58.714
That makes, that makes a lot of sense. And re-ranking is one thing we, we find useful for it internally. You use Llama 2 right now? We don't have it in production, but we've experimented with it with, for a few things.

309
00:46:59.094 --> 00:47:06.294
You also had an interesting observation, because I think we talk a lot about quantization in the podcast just for running locally- Mm-hmm... or, like, more efficient running.

310
00:47:06.594 --> 00:47:11.894
You say quantization and imperfect utilization cancel each other out. Yes. That's a cool observation. Yeah.

311
00:47:11.934 --> 00:47:21.734
So this is, like, a little bit hand-wavy, but the core thing is, yeah, we expect that when you don't have, like, complete utilization, right, you're never gonna, like, saturate, um, all your GPUs.

312
00:47:21.774 --> 00:47:31.654
There's gonna be some idle time. Like, um, from things that we've experimented with in the past, it ends up being, you know, 50% is a reasonable amount as a more liberal estimate of how much you can get.

313
00:47:32.134 --> 00:47:43.913
So the interesting thing about quantization is that there's a bunch of these kind of new quantization libraries that have cropped up, and they're all very good at reducing costs for low batch inference when you're memory bound.

314
00:47:43.954 --> 00:47:55.834
But the key thing is when you increase the batch size, they actually end up resulting in no real speedups over FP16. The reason why is because they only quantize the model weights, right?

315
00:47:55.934 --> 00:48:05.854
So that operation of kind of reading the model weights when they're now, you know, 4X smaller, instead of FP16, they're, you know, 4 bits or something. It's still the same number of weights.

316
00:48:06.214 --> 00:48:09.514
The operation of reading weights is, is, like, it ends up being three, 4X faster.

317
00:48:10.114 --> 00:48:21.094
But the issue is when you increase your batch size enough, and for large batch inference, the key thing is it now moves back from being memory to being compute bound again.

318
00:48:21.474 --> 00:48:27.294
And when you're compute bound, quantization of model weights basically does nothing, and so it ends up being effectively the same cost.

319
00:48:27.334 --> 00:48:38.014
And then the other interesting thing is it's even worse for small models because for, or at least the small Llama models, because I, I believe the smaller ones relative to model size have a much bigger KV cache.

320
00:48:38.494 --> 00:48:45.774
I'm not sure if the smaller ones use multi or group query attention. They might not. Uh, they do not. The, only the large ones use. Okay, exactly, yeah.

321
00:48:45.814 --> 00:48:57.334
So then because they use normal multi-head attention, the thing is when your batch size increases enough, then the memory bottleneck is not your small quantized model weights. No, it's actually the KV cache.

322
00:48:57.834 --> 00:49:01.814
And so quantizing the model weights effectively will do nothing then.

323
00:49:02.094 --> 00:49:16.274
So the key insight there is, like, all these new techniques are fantastic when you're just kind of playing with these models, running them low batch sizes, but when you really try to increase the batch size and serve it in production, they're probably gonna be lower or more expensive than FP16.

324
00:49:16.514 --> 00:49:23.874
Because there are these optimizations with things like text generation inference, which uses vLLM or, like, page attention, which are much, much faster.

325
00:49:24.374 --> 00:49:36.554
And so the best that I think you could probably do right now with open source is, like, full 8-bit quantization, which means not just quantizing the weights but also, like, the actual activations and the KV cache so that, like, none of those things end up being bottlenecks.

326
00:49:36.814 --> 00:49:37.834
That's a great breakdown. We...

327
00:49:37.874 --> 00:49:48.470
The post goes into much more detail with a lot of math, actually, which I love, and you also spec out some ru- rules of thumb, which I think people can use to Figure out their limitations and pricing and all that good stuff.

328
00:49:48.570 --> 00:49:58.130
Yeah. So it's good. And one, one big caveat I'd say is that the other massive benefit of Llama too is that you can fine-tune it. Fine-tune it. Yeah. Right? Well, you'll be able to fine-tune OpenAI soon enough. We'll see.

329
00:49:58.170 --> 00:50:03.610
[chuckles] So we'll just get your general takes on, um, LLM topics, uh, just kind of quick fire, and then we'll go to lightning round.

330
00:50:04.190 --> 00:50:14.350
So HumanEval, that is the predominant way to benchmark code models because OpenAI benchmarks code models that way. There's some issues with, with it. Yeah.

331
00:50:14.650 --> 00:50:21.650
With open source models and even probably with some closed source models, it's unclear how much of it has actually leaked into the train set, right?

332
00:50:21.710 --> 00:50:28.790
So there's a recent model, New Hope, which it looked like they had some leakage, which is why it had really, really good performance.

333
00:50:28.810 --> 00:50:38.410
[chuckles] But I think there was an interesting approach taken by Palm 2, where I think this is actually possible for someone to do right now. I've been meaning to do it at some point.

334
00:50:38.750 --> 00:50:47.230
But there's this paper called BabelCode, and they have a library which I think literally translates HumanEval into all other languages, and I think that would be a really good test.

335
00:50:47.250 --> 00:50:56.390
'Cause the other issue is a lot of the models that perform really well on HumanEval are pure Python, right? And that doesn't really give you a sense of if it's a good coding model overall.

336
00:50:56.790 --> 00:51:07.550
So yeah, I think at some point it would be really helpful if just someone did the work and, like, ran the Babel Code engine and, and translated HumanEval into all these other languages and then was able to run it.

337
00:51:07.830 --> 00:51:18.650
I think that, that would probably be a better me- benchmark, but still, I think if the original HumanEval problems leaked, I suspect it would also be helpful for solving the, the problems translated into other languages.

338
00:51:19.010 --> 00:51:27.430
But the issue is it's just so easy to run. [laughs] And, like, anything else is probably gonna be quite painful, right? Well, it'd be better if, like, there was a sandbox- Yeah...

339
00:51:27.510 --> 00:51:35.410
you know, to run it, so Agent Cloud, hashtag. [laughs] Uh, hot take on training. Yeah, another one from your endless Twitter quality.

340
00:51:35.410 --> 00:51:45.790
"Training will look like researchers offloading large scale training jobs to specialized training companies, a state of the art that resembles chip design and fabrication." Yeah. How do you think about that?

341
00:51:45.830 --> 00:51:53.830
And obviously Mosaic, uh, was on the podcast, just got acquired. So you, you tweeted that in May in 2022, and then one year later Mosaic gets acquired. Like, I think- Yeah... that's a pretty prescient, uh- Yeah.

342
00:51:53.869 --> 00:52:03.430
I was probably wrong about it in a lot of ways too because I assumed the future would kind of look like a lot of startups would have their own models.

343
00:52:03.990 --> 00:52:12.230
And this is me kind of in the CAD frame of mind, where I thought, okay, if you look at GPT-3, at that point it's just, like, GPT-3, maybe a little bit 3.5.

344
00:52:12.610 --> 00:52:18.970
It wasn't, like, that good a generalist model, and I thought prompting is not the way to do things. It's just completely fine-tuning or training your own models.

345
00:52:19.470 --> 00:52:27.550
And it was also a similar time that we kind of saw a lot of the open source earlier efforts in training models, which proved, like, not that great.

346
00:52:27.890 --> 00:52:37.470
I think Bloom and OPT were two, two models that came around at about that time. And if you looked at the OPT logs, they manually, like, tuned their learning rate several times.

347
00:52:37.530 --> 00:52:44.690
I think they switched the optimizer from Adam to s... They, they did something really weird where they switched the optimizer in the middle. And don't quote me on this- [laughs]...

348
00:52:44.690 --> 00:52:52.950
'cause I could be wrong, but I remember it was, like, some really, really sketchy stuff done in the middle. And I just thought, "Wow, if it's this hard, it seems like there's a company to bu- be built around it."

349
00:52:53.470 --> 00:53:05.510
The key difference is that there are just massive foundation model companies, and I think most AI product companies are not gonna be u- mostly training their models. They're mostly using, like- Right... custom models.

350
00:53:05.550 --> 00:53:14.130
It's more so gonna look like them kind of using these APIs out of the box, and then maybe using, you know, the fine-tuning endpoints there. Oh. Which, I mean, it's the same in chips.

351
00:53:14.130 --> 00:53:21.250
So you changed your mind a little bit. I did change my mind a little bit. I assumed... Like, with the CAD thing, I thought, okay, you're gonna need a foundation model for CAD.

352
00:53:21.290 --> 00:53:29.970
You're gonna need a foundation model for- No, that's old-school thinking. Yeah. And now it's just like you have the one generalist model. The one God model. And the one God model transfers fantastically- [laughs]...

353
00:53:30.010 --> 00:53:36.490
well for everything. Okay, quickly move along. You had another one which I loved. "The size of all code history on GitHub public repos is 92 terabytes.

354
00:53:37.030 --> 00:53:46.090
The size of Google's monorepo is 86 terabytes of much higher quality code. If Google were willing to deploy code models trained on their own data, they would have a noticeable advantage over everyone else." Yeah.

355
00:53:46.170 --> 00:53:53.070
Again, this is one thing that I, I think is probably a little wrong because this is based on the Big Science paper. Yeah.

356
00:53:53.430 --> 00:53:59.190
And the Big Science paper, like, basically said they scraped all of GitHub, and they got 92 terabytes.

357
00:53:59.230 --> 00:54:06.590
And I think if you look closely, which I did kind of after some people kind of pointed out some mistakes, I think GitHub's, like, a lot, a lot bigger than that.

358
00:54:07.090 --> 00:54:18.750
The Big Science paper said they git cloned, and so I was assuming, okay, git clone means you get the full working tree, right? But if you look a little deeper, I think GitHub is, like, a lot bigger than people think.

359
00:54:19.150 --> 00:54:30.970
My expectation is that GitHub probably has something like 5 to 10 trillion tokens of code, usable code. And so that's a lot more than what they ended up, ended up getting.

360
00:54:31.030 --> 00:54:39.030
But yeah, Google still has, like, a pretty meaningful fraction. And they just put out IDX, which is a- Yeah... somewhat of a competitor. Yeah, yeah. I think it's more like...

361
00:54:39.190 --> 00:54:45.330
It looks more like a Replit kind of competitor- Yes... where it's, like, a in-browser thing. But yeah, I think a lot of people can be viewed as competitors.

362
00:54:45.350 --> 00:54:53.190
[laughs] Uh, but you're very competitive, as we established- Yes... you know? And then final question, why is the company called Anysphere, and you have this whole manifesto on your landing page- Yes...

363
00:54:53.410 --> 00:54:54.770
about why humans should focus on bigger problems?

364
00:54:55.110 --> 00:55:07.690
It's an interesting story where Michael and I were in this program, Converge, and two of our friends, um, Arvin and Swale, who we knew, like, reasonably well at MIT, and we knew them because they're, like, some of the best engineers at MIT.

365
00:55:07.730 --> 00:55:18.930
And they... So they were independently kind of working on their own company. It was called Anysphere. And we both independently from after playing with GPT-4 realized, "Oh, wow, like, the IDE is the thing to build."

366
00:55:19.510 --> 00:55:27.630
After a few months of independently working on it, we realized, okay, like, why are we doing this separately? We should just kind of join forces, and that's kind of what we did.

367
00:55:27.690 --> 00:55:37.390
And so, like, right now, the overall company is called Anysphere, but yeah, the product and the core thing is Cursor. I- It's lovely. I, I recommend people actually check out anysphere.co- Mm-hmm...

368
00:55:37.490 --> 00:55:46.430
and read the manifesto 'cause I think it's a broader message to builders out there. Yeah. Uh, yeah, let's jump into lightning round. Okay. We got three questions for you. Mm-hmm.

369
00:55:46.470 --> 00:55:52.866
The first one is, what is something that already happened in AI that you thought would take much longer? I think code.

370
00:55:52.966 --> 00:56:00.626
And specifically, I think just being generalist at code, where before you had these special- specialized models, right?

371
00:56:00.686 --> 00:56:06.046
Where Codex was supposed to be kinda specialized for code, and then there's a general language model.

372
00:56:06.386 --> 00:56:14.146
But it's kind of unification of capabilities towards, like, this one model that's not just really good at text, but it's also fantastic at code.

373
00:56:14.566 --> 00:56:22.766
I was not expecting, like, the generalist model, I guess, to come super, super soon and be this good at code. That's why you pivoted or, or you started your whole company. Yes. Exploration.

374
00:56:22.906 --> 00:56:25.586
What do you think is the most interesting unsolved question in AI?

375
00:56:26.486 --> 00:56:40.966
I really think it's this kind of long-term memory piece, where I, I think it's possible to get to maybe AGI superhuman level systems that, that still kind of hack around memory using, like, something that kind of resembles transformers.

376
00:56:41.346 --> 00:56:51.946
But it feels like the more elegant thing is how do you get models that really, like, continuously learn? Some kind of recurrent-based system would be able to do this, where there's, like, a state.

377
00:56:52.326 --> 00:57:01.386
But right now, like, models can only really learn in context super efficiently. Fine-tuning is incredibly inefficient. It requires tons of data points to actually learn, learn new things.

378
00:57:01.766 --> 00:57:09.546
So yeah, I'm really interested to see how we solve this lifelong learning efficiency problem. Yeah. I'm interested in using knowledge graphs to do that- Mm....

379
00:57:09.606 --> 00:57:18.366
because I think that's kind of like a forgotten piece of the, the puzzle. And if you could have models update their own knowledge graphs and query their own knowledge graphs, that might be it.

380
00:57:18.546 --> 00:57:26.566
I think LlamaIndex is basically working itself into, into what that is. Oh, interesting. Yeah, and then there's the techniques where the models directly kind of learn to...

381
00:57:26.966 --> 00:57:34.786
Like, inside the weights or inside the architecture, you learn how to be able to read from databases and retrieval-based, like, the, the retro-based techniques.

382
00:57:35.126 --> 00:57:41.466
Like, those seemed interesting, but it's surprising, like, you haven't really seen anything from that in a while after that initial paper.

383
00:57:41.946 --> 00:57:54.746
And just to wrap the episode up, what's, uh, one message you want everyone to remember and think about as they keep building and exploring in AI? Yeah. I mean, GPT-4 is now a few months old.

384
00:57:54.946 --> 00:58:01.106
At some point, we're gonna get much, much better models, and I think it'll be pretty soon. And so what does the world look like then?

385
00:58:01.226 --> 00:58:08.806
And specifically for coding, like, what does the world look like when you have another step that's just as large as it was from GPT-3 to GPT-4?

386
00:58:09.426 --> 00:58:20.386
I think it's just so incredibly different, and I think it just completely changes how people write software. In what direction, though? 'Cause... So I've, I've said my piece on, like, 4.5 being more inference time. Mm.

387
00:58:20.446 --> 00:58:31.426
I don't actually know if that's true. That's just my theory. I think the direction that we'll probably see is... I mean, the language models will just get better at doing intense reasoning, right?

388
00:58:31.486 --> 00:58:38.946
So they'll be able to tackle harder problems. They'll probably pick up on more nuances on how, like, software engineering is done. They'll probably have longer context windows.

389
00:58:39.306 --> 00:58:44.446
And so I expect, yeah, more agentic-type things will end up being more prominent in the future.

390
00:58:44.826 --> 00:58:56.826
I don't know how far you can take the agent stuff with a four-level model, but I think with, like, a 4.5 or a 5, like, I think agent models will work for almost any kind of coding task.

391
00:58:57.346 --> 00:59:04.646
Um, at least almost any kind of reasonably well-scoped coding task. Yeah. Agents are the future. Yeah. Well, thanks so much for coming in. Thanks, Aman. Of course. Thanks for having me.

392
00:59:04.726 --> 00:59:24.546
[upbeat music]
