WEBVTT

1
00:00:00.040 --> 00:00:15.900
[upbeat music] Hey everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CGO in residence at Decibel Partners. I'm joined by my co-host Swyx, founder of Small AI.

2
00:00:16.200 --> 00:00:25.620
And today we have Dylan Patel, and welcome. So you are, you're the author of the extremely popular Semi Analysis blog. We have both had a little bit of claim to fame in breaking details of GPT-4.

3
00:00:25.800 --> 00:00:32.060
George Hotz came on our pod and talked about the mixture of experts thing, and then you had a lot more detail on your- Let's be clear, I talked about mixture of experts in January.

4
00:00:32.060 --> 00:00:39.000
[laughs] It's just people didn't really notice it- Yeah... I guess. I don't know. [laughs] You, you went into a lot more detail and, and I'd love to dig into some of that. Yeah. Thank you so much.

5
00:00:39.040 --> 00:00:42.120
I've been doing consulting in the industries, semiconductor industry since '17.

6
00:00:42.300 --> 00:00:52.200
2021, got bored, and in November I started writing a blog, and then like 2022 was good, and I started hiring folks at, for my firm, and then all of a sudden 2023 happens, and it's like the perfect intersection.

7
00:00:52.440 --> 00:00:59.680
I used to do data science, but not like AI, not really, like multivariable [laughs] progression is not AI, right?

8
00:00:59.740 --> 00:01:04.360
But also I've been involved in the semiconductor industry for a long, long time, posting about it online since I was 12, right?

9
00:01:04.519 --> 00:01:10.720
You know, all of a sudden this all kind of came to fruition, so it's cool to have the blog sort of blow up in, uh, that way.

10
00:01:11.120 --> 00:01:15.520
I used to cover semis at Dalyacny as well, and it was, for a long time it was just the mobile cycle- Mm-hmm...

11
00:01:15.580 --> 00:01:23.080
and then a little bit of PCs, but like not that much, and then I, I, maybe some cloud stuff, you know, like public cloud, you know, semiconductor stuff.

12
00:01:23.160 --> 00:01:33.740
But it really wasn't anything until this, this wave, and I was, I was actually listening to you on one of the previous podcasts that you've done and, and it was surprising that high-performance computing also kinda didn't really take off.

13
00:01:33.780 --> 00:01:36.680
Like AI is just the first form of high-per- high-performance computing that worked.

14
00:01:37.380 --> 00:01:51.800
One of the theses I've had for a long time that I think people haven't really caught on, but it's coming to fruition now, is that the largest tech companies in the world, their software is important, but actually having and operating a very efficient infrastructure is incredibly important.

15
00:01:51.860 --> 00:02:06.440
And so, you know, people talk about, you know, "Hey, Amazon is great, AWS is great," because yes, it is easy to use and they've built all these things, but behind the scenes they've done a lot on the infrastructure that is super custom that Microsoft Azure and Google Cloud just don't even match in terms of efficiency.

16
00:02:06.500 --> 00:02:16.420
If you think about the cost to rent out SSD space or the cost to rent, you know, offer database service on top of that obviously, a cost to rent out a certain level of CPU performance, Amazon has a massive advantage there.

17
00:02:16.440 --> 00:02:26.440
And likewise, like Google spent all this time doing that in AI, right? With their TPUs and infrastructure there and, and optical switches and all this sort of stuff. And so in the past it wasn't immediately obvious.

18
00:02:26.640 --> 00:02:35.820
I think with AI especially, like with how scaling laws are going, it's like incredibly important for infrastructure is like so much more important. And then like when you just think about software cost, right?

19
00:02:35.860 --> 00:02:36.780
Like the cost structure of it.

20
00:02:37.120 --> 00:02:48.820
There was always a bigger component of R&D and like SaaS businesses, you know, all over SF, all these SaaS businesses did crazy good 'cause, you know, they just start as they grow and then all of a sudden they're so freaking profitable for each incremental new customer.

21
00:02:49.120 --> 00:02:52.400
And AI software looks like it's gonna be very different in my opinion, right?

22
00:02:52.460 --> 00:03:02.860
Like the R&D cost is much lower in terms of people, but the cost of goods sold in terms of actually operating the service I think will be much higher. And so in that same sense, infrastructure matters a ton.

23
00:03:03.180 --> 00:03:12.100
And I think you wrote once that training costs effectively don't matter. Yeah. In, in my opinion, I think that's a little bit spicy. But yeah, it's like training costs are irrelevant, right? Like GPT-4, right?

24
00:03:12.160 --> 00:03:18.920
Like 20,000 A100s, that's, that's chi- like I know it sounds like a lot of money. Child's play. 500 million all-in, is that a reasonable estimate?

25
00:03:18.960 --> 00:03:25.560
The supercomputer, it's, it's o- it's slightly more, but yeah, I think the 500 million is a fair enough number. I mean, if you think about just the pre-training, right?

26
00:03:25.680 --> 00:03:33.800
Three months, 20,000 A100s at, you know, a dollar an hour is like, that is way less than 500 million, right? But of course there's data and all this sort of stuff. Yeah.

27
00:03:33.860 --> 00:03:41.780
So people that are watching this on YouTube, they can see a GPU poor and a GPU rich, uh, on the table, which is inspired by your- I was gonna ask you about that. Yes...

28
00:03:41.780 --> 00:03:50.100
yeah, yeah, your Google Gemini hits the world blog post. One, did you know that this thing was gonna blow up so much? Sam Altman even tweeted about it.

29
00:03:50.120 --> 00:03:59.560
He said, "Incredible Google got the Sem- Semi Analysis guy to publish their internal [laughs] marketing recruiting chart." And yeah, tell people who are the GPU poor, so who are the GPU rich?

30
00:03:59.600 --> 00:04:01.540
Like what's this framework that they should think about?

31
00:04:01.560 --> 00:04:11.740
So it's, it's, um, you know, some of this work we've been doing for a while is just on infrastructure and like, hey, like when something happens, I think it's like a, you know, sort of competitive ad- advantage of our firm, right?

32
00:04:11.780 --> 00:04:21.160
Me, myself and my colleagues is we go from software all the way through to like low-level manufacturing and it's like, who... You know, oh, Google's actually ramping up TPU production massively, right?

33
00:04:21.600 --> 00:04:26.940
And like I think people in AI would be like, "Well, duh," but like, okay, like who, who has the capability of figuring out the number?

34
00:04:26.980 --> 00:04:30.460
Well, one, you could just get Google to tell you, but they, they don't, they won't tell you, right?

35
00:04:30.500 --> 00:04:37.420
That's like a very closely guarded secret, and most people that work at Google DeepMind don't even know that, know that number, right? Two, you go through the supply chain and see what they've placed in orders.

36
00:04:37.560 --> 00:04:45.580
Three is sort of like, well, who's actually winning from this? Hey. Oh, Celestica's building these boxes. Wow. Oh, interesting. Okay, um, this company's involved in testing for them. Oh, okay.

37
00:04:45.920 --> 00:04:51.600
Oh, this company's providing design IP to them. Okay. That's very valuable in a monetary sense, but you know, you have to understand the whole technology stack.

38
00:04:51.620 --> 00:04:59.700
But on the flip side, right, is, well, why is Google building all these? What could they do with it, and what does that mean for the world? Especially in SF, right? Like I'm sure you folks have been to parties.

39
00:04:59.900 --> 00:05:04.580
People just brag about [laughs] how many GPUs they have. Like I've, it's happened to me multiple times where somebody's just like...

40
00:05:05.000 --> 00:05:12.580
I'm, I'm just witnessing a conversation where somebody from Meta is bragging about how many GPUs they have versus someone from another firm, uh, that it's like [laughs]...

41
00:05:12.680 --> 00:05:19.940
Or like a startup person's like, "Dude, can you believe we just acquired, we have 512 H100s coming online in, in August?" And it's like, "Oh, cool."

42
00:05:19.980 --> 00:05:29.860
Like, you know, going through the supply chain, it's like, dude, you realize there's 400,000 manufactured last quarter and like 530,000 this quarter being sold of H100s. And it's like, oh crap, that's, that's a lot.

43
00:05:29.900 --> 00:05:37.040
That's a lot of GPUs. But then like, oh, how, how does that compare to Google? And like there's one way to look at the world, which is just like, hey, scale is all you need.

44
00:05:37.080 --> 00:05:44.700
Like obviously data matters, obviously all this stuff matters, but given any data set, a larger model will just do better, I think is... It's gonna be more expensive, but it's gonna do better.

45
00:05:45.020 --> 00:05:56.300
Okay, there's all these GPUs going to production. Nvidia's gonna sell well over 3 million, you know, total GPUs next year. Over a million H100s this year alone. There's a lot of GPU capacity coming online.

46
00:05:56.360 --> 00:06:02.760
It's, it's an incredible amount and... Well, what are people doing? What are people working on? I think it's very important to like just think about what are people working on, right?

47
00:06:03.180 --> 00:06:13.260
What actually are you building that's, that's gonna advance what is monetizable, but what also makes sense? And so like a lot of people were doing things that I thought felt counterproductive, right?

48
00:06:13.640 --> 00:06:25.240
In a, in a world where in less than a year there's gonna be more than 4 million high-end GPUs out there, you know, we can talk about the concentration of those GPUs, but if you're doing really valuable work as a, as a good person, right?

49
00:06:25.300 --> 00:06:34.648
Like you're, you're contributing in some way, should you be focused on like- Well, I don't have access to any of those four million GPUs, right? I actually only have access to gaming GPUs.

50
00:06:34.668 --> 00:06:44.688
Should I focus on, like, being able to fine-tune a model on that, right? Like, no, it's not really that important. Or like, should I be focused on batch one inference on a cloud GPU? Like, no, that's like pointless.

51
00:06:44.728 --> 00:06:52.828
Like, why would you do batch size one inference on an H100? That's just, like, ridiculously dumb. There's a lot of counterproductive work, and at the same time, there's a lot of things that people should be doing.

52
00:06:53.088 --> 00:06:54.928
I mean, obviously, most people don't have resources, right?

53
00:06:54.968 --> 00:07:05.988
And I love the open source, and I want the open source to win, and I hate the people who want to, like, "No, we're X lab, and we think this is the only way you should do it, and if people don't do it this way, they should be regulated against it," and all this kind of stuff.

54
00:07:06.048 --> 00:07:13.888
I hate that attitude. So I want the open source to win, right? Like companies like Mistral and like what Meta are doing, you know, Mosaic and, and you know, all these folks together.

55
00:07:13.948 --> 00:07:21.708
All these people doing, you know, huge stuff for the open source, you know, want them to succeed, but it's like there's certain things that are, you know, like fo- hyper-focusing on leaderboards at Hugging Face.

56
00:07:21.868 --> 00:07:27.408
No, like Truthful QAs a garbage benchmark. Some of the models that are very high on there, if you use it for five seconds, you're like, "This is garbage."

57
00:07:27.468 --> 00:07:37.128
There was things I wanted to say also, you know, we're in a world where compute matters a lot. Google is gonna have more compute than any other company in the world, period, by, by, like a large, large factor.

58
00:07:37.448 --> 00:07:46.208
It's just like framing it into that, like mindset of like, hey, like what are the counterproductive things? What do I think personally, or what have people told me that are involved in this should they focus on?

59
00:07:46.588 --> 00:07:57.048
The pace of acceleration from twenty-twenty to twenty-twenty two is less than twenty-twenty two to twenty-twenty four. You know, GP two to four, two to four is like twenty-twenty to twenty-twenty two, right?

60
00:07:57.428 --> 00:08:06.468
Is, is less than I think from GPT-4 in twenty-twenty two, which is when it was trained, right, to what OpenAI and, and Google and, and Anthropic do in twenty-twenty five. Right?

61
00:08:06.488 --> 00:08:12.988
Like I think the pace of acceleration is, is increasing. Um, and, and it's just good to like think about, you know, that sort of stuff. That makes sense.

62
00:08:13.028 --> 00:08:25.648
And the chart that Sama mentioned is about, yeah, Google TPU V5's completely overtaking OpenAI by orders of magnitude. Let's talk about the TPU a bit. We had Chris Lattner on the show, which I know you know.

63
00:08:25.828 --> 00:08:35.548
He used to work on TensorFlow and Google, and he did mention that the goal of Google is like make TPUs go fast with TensorFlow. But then he also had a post about PyTorch stealing the thunder.

64
00:08:35.888 --> 00:08:43.288
How do you see that changing? If like now that a lot of the compute will be TPU based and Google wants to offer some of that to the public too.

65
00:08:43.508 --> 00:08:49.908
Google internally and I think, you know, is, is obviously on JAX and XLA and all that kind of stuff. Externally, like they've done a lo- really good job.

66
00:08:49.968 --> 00:08:55.808
Wouldn't say like TPUs through PyTorch XLA is amazing, but it's, it's not bad, right?

67
00:08:55.848 --> 00:09:04.888
Like some of the numbers they've shown, some of the, you know, code they've shown for TPU v5e, which is not the TPU v5 that I was referring to which was in, in the sort of the post, the GPU poor post I was referring to.

68
00:09:04.928 --> 00:09:11.688
But TPU v5e is like, uh, the new one, but it's mostly, mostly an inference chip. It's a small chip. It's, it's about half the size of a TPU v5.

69
00:09:11.808 --> 00:09:19.108
That chip, you know, you can get very good performance on, of Llama seven-ty B inference. Very, very good performance when you're using PyTorch and XLA.

70
00:09:19.388 --> 00:09:26.628
Now, of course you're gonna get better if you go JAX XLA, but I think Google is doing a really good job after the restructuring of focusing on external customers too.

71
00:09:26.668 --> 00:09:33.548
Probably won't focus too much on TPU v5 for everyone externally, but v5e, we're also building a million of those, right? Hey, a lot of companies are using them, right?

72
00:09:33.748 --> 00:09:42.568
Or, or will be using them because it's gonna be an incredibly cheap form of compute. I think the world of frameworks and all that, right, like that's obviously something a researcher should talk about, not myself.

73
00:09:42.668 --> 00:09:49.948
But you know, the, the stats are clear that PyTorch is way, way dominating everything. But JAX is like doing well. Like there's external users of JAX.

74
00:09:50.148 --> 00:09:55.968
The forever shouldn't be that the person doing PyTorch level code, right, that high should also be writing custom CUDA kernels, right?

75
00:09:56.008 --> 00:10:02.508
There should be, you know, different layers of abstraction where people hyper-optimize and make it much easier for everyone to innovate on separate stacks, right?

76
00:10:02.548 --> 00:10:11.608
And then every once in a while someone comes through and pierces through the layers of abstraction and innovates across multiple or a group of people. But I think frameworks are important, compilers are important, right?

77
00:10:11.668 --> 00:10:18.708
Chris Lattner's what he's doing is really cool. I, I don't know if it'll work, but it's, it's super cool and it certainly works on CPUs. Um, we'll see about accelerators.

78
00:10:19.108 --> 00:10:27.128
Likewise, there's Open3.js Triton and like what they're trying to do there and you know, everyone's really coalescing around Triton. Third party hardware vendors, uh, there's Palace.

79
00:10:27.348 --> 00:10:39.068
I, I don't wanna mischaracterize it, but you can write in Palace and, and it'll go through-- you can lower level code and it'll work to, uh, TPUs and GPUs, kinda like Triton, but it's like there's a backend for Triton.

80
00:10:39.108 --> 00:10:44.728
I don't know exactly everything about it, but I think there's a lot of innovation happening on make things go faster, right? How do you, how do you go brr?

81
00:10:44.788 --> 00:10:52.988
Because every single person working in ML, it would be a travesty if they had to write like custom CUDA kernels always, right? Like that would just slow down productivity. But at the same time, you kinda have to.

82
00:10:53.348 --> 00:11:03.008
By the way, I like to quantify things when you say make things go brr. Is there a target range of MFU that you typically talk about? Yeah, there's, there's sort of two metrics that I like to think about a lot, right?

83
00:11:03.068 --> 00:11:04.908
So in training everyone just talks about MFU, right?

84
00:11:05.108 --> 00:11:16.448
But then on inference, right, which I think is one LLM inference will be bigger than training or multimodal whatever blah, blah, blah inference will be bigger than training probably next year in fact, at least in terms of GPUs deployed.

85
00:11:17.048 --> 00:11:21.008
And the other thing is like, you know what, what's the bottleneck when you're running these models?

86
00:11:21.128 --> 00:11:30.748
The simple stupid way to look at it is training is, you know, there's six flops floating point operations you have to do for every byte you read in, right? Every, every parameter you read in.

87
00:11:30.808 --> 00:11:38.128
So if it's FP8 then it's a byte, if it's FP16 it's two bytes, whatever, right? The on training. But on inference side the ratio is completely different. It's two to one, right?

88
00:11:38.388 --> 00:11:47.308
There's two flops per parameter that you read in and parameters maybe one byte, right? Because it's FP8 or intake. But then when you look at the GPUs, the GPUs are very, very different ratio.

89
00:11:47.648 --> 00:11:55.208
The H100 has three point three five terabytes a second of memory bandwidth and it has a thousand teraflops of FP16, bfloat16. Right?

90
00:11:55.268 --> 00:12:01.268
So that ratio is like, I, I'm sorry I'm gonna butcher the math here and people are gonna think I'm dumb, but two fifty-six to one, right?

91
00:12:01.328 --> 00:12:11.628
Call it two fifty-six to one if you're doing FP16 and, uh, same applies to FP8, right? Because, uh, yeah anyways, uh, per, per parameter, parameter read to, uh, number of floating point operations, right?

92
00:12:11.828 --> 00:12:18.268
'Cause if you, if you, if you quantize further then you but you also get double the performance on that lower quantization. You know that, that, that does not fit the hardware at all.

93
00:12:18.448 --> 00:12:24.008
So if you're just doing LLM inference at batch one then you're always gonna be underutilizing the flops. You're only paying for memory bandwidth.

94
00:12:24.248 --> 00:12:34.068
And the way hardware is developing that ratio is actually only gonna get worse. H200 will come out soon enough which will help the ratio a little bit, you know improve memory bandwidth more than it improves flops.

95
00:12:34.388 --> 00:12:40.408
Just like the A-A100 eighty gig did versus the A100 forty gig. But then when the B100 comes out the flops are gonna increase more than memory bandwidth.

96
00:12:40.488 --> 00:12:44.608
And then when future generations come out and the same with AMD's side, right? MI three hundred versus four hundred.

97
00:12:44.688 --> 00:12:55.118
As you move on generations just due to fundamental like semiconductor scaling DRAM memory is not scaling as fast as logic has been, and you can do a lot of interesting things on the architecture.

98
00:12:55.438 --> 00:13:02.138
So, so you're gonna have this problem get worse and worse and worse. And so on training it's very, you know, who cares, right? Because my flops are still my bottleneck mostly.

99
00:13:02.178 --> 00:13:09.538
I mean, memory bandwidth is obviously a bottleneck, but like well, I, you know, batch sizes are freaking crazy s- right? Like, people train like two million batch size is trivial, right?

100
00:13:09.578 --> 00:13:19.118
Like, that's what Llama I think did. Llama 70B was two million batch size. Unlike you talk to someone at one of the frontier labs and they're like, "[laughs] Just two million?" Two million token batch size, right?

101
00:13:19.158 --> 00:13:26.898
That's crazy. Or sequence, sorry. But when you go to inference side, well, it's impossible to do one, to do two million batch size. Also, your latency would be horrendous if you tried to do something that crazy.

102
00:13:27.078 --> 00:13:36.098
So you kinda have this like differing problem where on training everyone just kept talking MFU, model flop utilization, right? How many flops? Six times the number of parameters basically, more or less.

103
00:13:36.518 --> 00:13:44.958
And then what's the quoted number, right? So if I have three- 312 teraflops out of my A100, and I was able to achieve 200, that's all. That's really good, right? You know? Some people are achieving higher, right?

104
00:13:44.998 --> 00:13:53.298
Some people are achieving lower. That's a very important like metric to think about. Now you have like people thinking MFU is like a security risk. But on inference, MFU is not nearly as important, right?

105
00:13:53.457 --> 00:13:59.018
It's, it's memory bandwidth utilization. You know, batch one is, you know, what memory bandwidth can I achieve, right? 'Cause if...

106
00:13:59.078 --> 00:14:08.998
as I increase batch from batch size one to four to eight to, you know, even 256, right, is sort of where the crossover happens on an H100 inference-wise, where it's flops limiting you more and more.

107
00:14:09.318 --> 00:14:20.998
But like you, you should have very high memory bandwidth utilization. So when people talk about A100s, like 60% MFU is decent. On a H100s it's more like 40, 45% because the flops increase more than the memory bandwidth.

108
00:14:21.458 --> 00:14:32.298
Uh, but people over time will probably get above 50% on, on H100 on MFU on training. But on inference it's not being talked about much, but MBU, model bandwidth utilization is the important factor, right?

109
00:14:32.638 --> 00:14:40.218
So of my 3.3 ter- five ter- terabytes a second of memory bandwidth that I have on my H100, can I get two? Can I get three, right? That's the important thing.

110
00:14:40.238 --> 00:14:44.978
And, and right now if you look at everyone's inference stuff... So I, I dogged on this in the GPU four thing, right?

111
00:14:45.078 --> 00:14:53.398
But it's like Hugging Face's libraries are actually very inefficient, like incredibly inefficient for inference. You get like 15% MBU on, on s- on, on some configurations.

112
00:14:53.438 --> 00:15:04.098
Like 8 A100s and Llama 70B you get like 15%, which is just like horrendous. Because at the end of the day, your latency is derived from what memory bandwidth you can effectively get, right?

113
00:15:04.378 --> 00:15:15.678
You know, so if, if, if you're doing Llama 70 billion, 70 billion parameters, if you're doing it int eight, okay, that's 70, uh, gigabytes a second, uh, gigabytes you need to read for every single inference, every single forward pass.

114
00:15:15.738 --> 00:15:25.258
Plus, you know, the attention but, you know, again we're simplifying it. 70 gigabytes you need to read for every forward pass. What is an acceptable latency for a user to have? I would argue 30 milliseconds per token.

115
00:15:25.478 --> 00:15:34.798
Some people would argue lower, right? But at the very least you need to achieve human reading level speeds and probably a little bit faster 'cause we like to skim to have a usable model for chatbot style applications.

116
00:15:34.818 --> 00:15:39.938
Now, there's other applications of course, but chatbot style applications you want it to be human reading speed. So 30 tokens per second.

117
00:15:40.398 --> 00:15:53.338
30 tokens per second is 33 or, or 30 tokens, milliseconds per token is 33 tokens per second times 70 is... Let's say three times seven is 21 and then add two zeros. So 2,100 gigabytes a second. Yeah. Right?

118
00:15:53.578 --> 00:16:04.858
To achieve human reading speed on Llama 70B, right? So one, you can never achieve Llama 70B human reading speed on even if you had enough memory capacity on a model on, on an A100, right?

119
00:16:05.338 --> 00:16:14.258
Even in, e- even in H100 to achieve human reading speed, right, of course you couldn't fit it 'cause it's 80 gigabytes versus 70 billion parameters so you're, you're, you're kind of butting up against the limits already.

120
00:16:14.738 --> 00:16:20.978
70 billion parameters being 70 gigabytes at in- int eight or FP8. You end up with one, how do I achieve human reading level speeds, right?

121
00:16:21.038 --> 00:16:25.538
So if I go with two H100s, then now I have, you know, call it six terabytes a second of memory bandwidth.

122
00:16:25.958 --> 00:16:36.358
If I achieve just 30 milliseconds per token, then I'm, you know, which is, uh, 33 tokens per second which is 30 ti- you know, is three terabytes a second. What is it? Thir- three times... Uh, 21...

123
00:16:36.758 --> 00:16:44.958
2.1 terabytes a second of memory bandwidth, then I'm only at like 30% m- bandwidth utilization. So I'm not using all my flops on batch one anyways, right?

124
00:16:45.238 --> 00:16:52.418
'Cause 70, you know, the, the, the flops that you're using there is tremendously ri- low relative to inference, and I'm not actually using a ton of the tokens on inference.

125
00:16:52.458 --> 00:17:02.898
So with two H100s I only get 30 milliseconds of token. That's a really bad result. You should be striving to get, you know, so upwards of 60%. And that's like, 60% is kinda low too, right?

126
00:17:02.958 --> 00:17:06.398
Like, I've, I've heard people getting 70, 80% model bandwidth utilization.

127
00:17:06.537 --> 00:17:14.818
And then, you know, obviously you can increase your batch size from there and, and your model bandwidth ut- utilization will start to fall as your flops utilization increases. But, you know, there...

128
00:17:14.858 --> 00:17:22.818
You have to pick the sweet spot for where you wanna hit on the latency curve for your user. Obviously as you increase batch size you get more throughput per GPU, so that's more cost effective.

129
00:17:22.838 --> 00:17:35.618
There's a lot of like things to think about there, but I think those are sort of the two main things that people wanna think about and there's obviously a ton with regards to like networking and inter GPU connection 'cause, uh, most of the useful models don't run on a single GPU.

130
00:17:35.838 --> 00:17:38.998
They can't run on a single GPU. Is there, is there a TPU equivalent of Mellanox?

131
00:17:39.358 --> 00:17:48.038
The Google TPU is like super interesting because Google's been working with Broadcom who's the number one networking company in the world, right? So Mellanox was nowhere close to number one.

132
00:17:48.338 --> 00:17:55.238
They had a niche that they were very good at which was the network card, the, the card that you actually put in the server, but they didn't do much. They didn't have...

133
00:17:55.258 --> 00:18:03.238
They weren't doing successfully in the switches, right? Which is, you know, you connect all the networks cards to switches and then the switches to all the servers. So Mellanox was not that great. I mean, it was good.

134
00:18:03.258 --> 00:18:12.758
They were doing good and Nvidia bought them, you know, in '19 I believe or '18. But Broadcom has been number one in networking for decade plus, and Google partnered with them on making the TPU, right?

135
00:18:12.818 --> 00:18:21.118
So Google does a lot of the design especially on the ML hardware side on how you pass stuff around internally on the chip. But Broadcom does a lot on the network side, right?

136
00:18:21.198 --> 00:18:25.338
They specifically how to get really high connection speed between two chips, right?

137
00:18:25.398 --> 00:18:35.658
They've do, they've done a to- ton there and obviously Google works ton there too, but this is sort of Google's like le- less discussed partnership that's truly critical for them and why Go- Google's tried to get away from them many times.

138
00:18:35.938 --> 00:18:44.538
Their latest target to get away from Broadcom is 2027, right? But like, you know, that's, that's four years from now. Chip design cycle's four years so they already tried to get away in 2025 and that failed.

139
00:18:44.878 --> 00:18:51.938
They have this equivalent of very high speed networking. It works very differently than the way GPU networking does and, and that's important for people who code on a lower level.

140
00:18:52.298 --> 00:19:01.978
I've seen this described as the ultimate rate limit on how big models can go. It's not flops, it's not memory, it's n- networking. Like it has, it has the lowest scaling law.

141
00:19:02.058 --> 00:19:09.708
It's like the lowest Moore's laws, and I don't know what to do about that because no one else has s- any solutions. [laughs]Yeah. Yeah.

142
00:19:09.768 --> 00:19:17.728
So I think, I think what you're referring to is that, like network speed has increased slower- M- much slower than the other two... than, than FLOPS. Yeah, and bandwidth. Yeah. Yeah.

143
00:19:17.768 --> 00:19:21.548
And yeah, that's a tremendous problem in the industry, right? That's why NVIDIA bought a networking company. Yeah.

144
00:19:21.568 --> 00:19:30.128
That's why Broadcom is, is, is, is working with, uh, on Google's chip right now, but of course on Meta's, Meta's internal AI chip, which they're on the second generation of working on that.

145
00:19:30.168 --> 00:19:37.388
And what's the main thing that, uh, Meta's doing interesting is networking stuff, right? Multiplying tensors is kind of... There's a lot of people who've made good matrix multiply units, right?

146
00:19:37.708 --> 00:19:44.308
But it's about like getting good utilization out of those and interfacing with the memory and interfacing with other chips really efficiently makes designing these chips very hard.

147
00:19:44.388 --> 00:19:49.428
Most of the startups obviously have not done that really well. I think the startups point is the most interesting, right?

148
00:19:49.668 --> 00:19:58.868
You mentioned companies that are GPU poor, they raise a lot of money, and there's a lot of startups out there that are GPU poor and did not raise a lot of money. What should they do?

149
00:19:59.148 --> 00:20:11.548
How, how do you see like the space dividing? Are we just supposed to wait for like the, the big labs to do a lot of this work with a lot of the GPUs? What's like the GPU poor's beautiful version of the, of the article?

150
00:20:11.968 --> 00:20:19.448
OpenAI who everyone would be like, "Oh yeah, they have more GPUs than anyone else," right? But they have a lot less FLOPS than Google, right? That was the point of the like thing, and it was, but not just them.

151
00:20:19.488 --> 00:20:27.988
It's like, okay, you, you, you know, it's like a relative totem pole right now. Of course, Google doesn't use G- GPUs as much for training and inference. Like they do use some but mostly TPUs.

152
00:20:28.408 --> 00:20:31.748
So kind of like the whole point is that everyone is GPU poor [laughs]

153
00:20:31.808 --> 00:20:38.648
because we're gonna continue to scale faster and faster and faster and faster and compute will always be a bottleneck, just like data will always be a bottleneck.

154
00:20:38.708 --> 00:20:44.448
You can have the best data set in the world, and you can always have a better one. And same with you can have the biggest compute system in the world and you can al- but you'll always want a better one.

155
00:20:44.588 --> 00:20:52.648
I mean, so it's like there's things that like Mistral, they trained a fricking awesome model on relatively fewer GPUs, right? And, and now they're scaling up higher and higher and higher, right?

156
00:20:52.908 --> 00:21:03.108
There's a lot that the GPU poor can do though, right? We all have phones. We all have laptops, right? There is a world for running GPUs or models on device. The Replit folks are, you know, trying to do stuff like that.

157
00:21:03.348 --> 00:21:12.348
Their models, they can't follow scaling laws, right? Why? Because there is a fundamental limit to how much memory bandwidth and capacity you can get on a laptop or a phone.

158
00:21:12.408 --> 00:21:18.347
You know, I mentioned the ratio of FLOPS to bandwidth on a GPU is actually really good compared to like a MacBook or like a phone.

159
00:21:18.588 --> 00:21:26.528
To run Llama seventy billion requires two terabytes a second of memory bandwidth, two point one at reading human reading speed. Yeah, but my phone has like fifty gigabytes a second.

160
00:21:26.608 --> 00:21:34.608
Your laptop, even if you have an M1 Ultra, has what like, I don't remember, like a couple hundred gigabytes a second of memory bandwidth. You can't run Llama seventy B just by doing the classical thing.

161
00:21:34.648 --> 00:21:40.108
So there's like, there's stuff like speculative decoding. You know, Together did something really cool, and they put it in open source, of course. Medusa, right?

162
00:21:40.148 --> 00:21:46.848
Like things like that that are, you know, they work on batch size one. They don't work on batch size, you know, high. And then so there's like the world of like cloud inference.

163
00:21:47.048 --> 00:21:50.768
And so in the cloud it's all about what memory bandwidth and MFU I can achieve.

164
00:21:50.928 --> 00:22:00.868
Whereas on the edge, I don't think Google's gonna deploy a model that I can run on my laptop to help me with code or help me with, you know, X, Y, Z. They're always gonna wanna run it on the cloud for control.

165
00:22:01.088 --> 00:22:05.208
Or maybe they let it run on the device, but it's like only their Pixel phone, you know? It's kinda like a walled garden thing.

166
00:22:05.428 --> 00:22:14.408
There's obviously a lot of reasons to do other things for security, for openness, to not be at the whims of a trillion-dollar plus company who wants my data, right?

167
00:22:14.548 --> 00:22:20.528
You know, there's a lot of stuff to be done there, and I think folks like Replit, they open source their model, right? Things like what, what, like Together what I just mentioned, right?

168
00:22:20.688 --> 00:22:28.548
That developing Medusa, that didn't take much GPU at all, right? That's... They're very G- well, they do have quite a few GPUs. They made a big announcement about having four thousand H100s.

169
00:22:28.868 --> 00:22:36.668
That's still relatively poor, right? When we're talking about hundreds of thousands of like the big labs, uh, like OpenAI and, and so on and so forth, or millions of TPUs like Google.

170
00:22:36.908 --> 00:22:46.588
But still, they were able to develop Medusa with probably just one server, one server with eight GPUs in it. And its usefulness of something like Medusa, something like speculative decoding is, is on device, right?

171
00:22:46.608 --> 00:22:53.028
And that's what like a lot of people can focus on. You know, people can focus on all sorts of things like that. I don't, I don't know, right? Like w- a new model architecture, right?

172
00:22:53.068 --> 00:23:00.188
Like are we only gonna use transformers? I'm pretty pilled to think like transformers are it, right? My hardware brain can only know something that loves hardware, right?

173
00:23:00.348 --> 00:23:08.988
People should continue to try and innovate on that, right? Like, you know, asynchronous training, right? Like that kind of stuff is like super, super interesting. I think it's Tim Demetres. He had like this- Demirs.

174
00:23:09.208 --> 00:23:13.588
The same guy as QLoRA. Yes. He had the Swarm paper and Petal. That research is super cool.

175
00:23:13.948 --> 00:23:21.248
The universities will never have much compute, but like, hey, to prepare to do things, to, you know, all these sorts of stuff, like they should try to build, you know, super large models.

176
00:23:21.268 --> 00:23:27.548
Like you look at what Tsinghua University is doing in China. Actually, they open source their model too. I think the largest like by parameter count at least open source models.

177
00:23:27.588 --> 00:23:34.208
I mean, of course they didn't train it on much data, but it's like, you know, it's like still like you could do some still cool stuff like that. I don't know. I think there's a lot that people can focus on.

178
00:23:34.648 --> 00:23:43.828
One, scaling out a service to many, many users. Distribution's very important. So figuring out distribution, right? Like figuring out useful fine tunes, right? Like doing LLMs that OpenAI will never make.

179
00:23:44.168 --> 00:23:51.428
Sorry for the crassness. A porn DALL-E 3, right? [laughs] Open source is doing crazy stuff with stable diffusion, right? [laughs] Right? Like, I don't, I don't mean to...

180
00:23:51.448 --> 00:23:59.928
Yeah, but it's like, it's like and there is a legitimate market. I think there's a couple companies who make tens of millions of dollars of revenue from LLMs or diffusion models for porn, right?

181
00:23:59.988 --> 00:24:01.128
Or, or, you know, that kind of stuff.

182
00:24:01.168 --> 00:24:09.988
Like, I mean, there's a lot of stuff that people can work on that will be successful businesses or doesn't even have to be a business but can advance humanity tremendously that doesn't require crazy scale.

183
00:24:10.348 --> 00:24:19.528
How do you think about the depreciation of like the hardware versus the models? If you buy a H100, sure the next years is gonna be better, but like at least the hardware is good.

184
00:24:19.588 --> 00:24:28.688
If you're spending a lot of money on like training a smaller model, it might be like super obsolete in like three months and you got now all this compute coming online.

185
00:24:28.988 --> 00:24:36.908
I'm just curious if like companies should actually spend the time to like, you know, fine-tune them and like work on them where like the next generation is gonna be out of the box so much better.

186
00:24:37.088 --> 00:24:56.028
Unless you're fine-tuning for on-device use, I think fine-tuning current existing models, especially the smaller ones, is a useless waste of time because the cost of inference is actually much cheaper than you think once you achieve good MBU and you batch at a decent size, which any successful business in the cloud is gonna achieve, you know?

187
00:24:56.068 --> 00:25:03.628
And then, and then two, fine-tuning like people are like, "Oh, you know, this seven billion parameter model if you fine-tune it on a data set is almost as good as three point five," right?

188
00:25:03.788 --> 00:25:10.928
Why don't you fine-tune three point five and look at your performance, right? And, and like there's nothing open source that is anywhere close to three point five yet. There will be.

189
00:25:11.048 --> 00:25:13.868
People also don't, don't quite grasp- Falcon was supposed to be.

190
00:25:14.108 --> 00:25:23.868
Falcon one forty B Uh, it's, it's less parameters than 3.5 and, and also, uh, I don't know about the exact token count, but I believe it- Do we know the parameters of 3.5? Um, it's not 175 billion.

191
00:25:23.928 --> 00:25:33.888
People keep saying this. No. 'Cause we know 3, but we don't know 3.5. The 3.5- It's definitely smaller. No, it's bigger than 175. I think it's sparse. MoE, I'm pretty sure, and, uh- Yeah...

192
00:25:33.988 --> 00:25:38.088
you know, you can, you can do some, like, gating around the size of it by looking at their inference latency.

193
00:25:38.408 --> 00:25:44.708
Well, what's the theoretical bandwidth if they're running it on this hardware and, and doing tensor parallel in this way, so they have this much memory bandwidth and maybe they get...

194
00:25:44.948 --> 00:25:47.607
Maybe they're awesome and they get 90% memory bandwidth utilization. I don't know.

195
00:25:47.648 --> 00:25:56.508
That's an upper bound and you can see the latency that 3.5 gives you, especially at, like, off-peak hours or if you do fine tuning and you have your, if you have a private enclave, they'll, like, Azure will quote you latency.

196
00:25:56.568 --> 00:26:04.768
So you can, you can figure out how many parameters per forward pass, which I think is somewhere in the, like, 50 to 40 billion range, but I could be very wrong.

197
00:26:04.788 --> 00:26:10.888
That's just, like, my guess based on that sort of stuff, you know, 50-ish. But actually I think open source will have models of that quality.

198
00:26:11.088 --> 00:26:17.268
I mean, I, I assume Mosaic or, like, Meadow will open source and Mistral will be able to open source models of that quality.

199
00:26:17.328 --> 00:26:26.528
Now furthermore, right, like, if you just look at the amount of compute, obviously data's very important and the abil- all these tricks and dials that you turn to be able to get good MFU and good MBO, right?

200
00:26:26.568 --> 00:26:37.048
Like, depending on inference or training is there's a ton of tricks, but at the end of the day, there's, like, 10 companies that have enough compute in one single data center to be able to beat GPT-4, right?

201
00:26:37.768 --> 00:26:41.368
Like straight up, like to- if not today, within the next six months, right?

202
00:26:41.828 --> 00:26:51.588
4,000 H100s is, is I, I think you need about 7,000 maybe and with some algorithmic improvements that have happened since GPT-4 and some of the, some data quality improvements probably.

203
00:26:51.628 --> 00:27:00.068
Like, you could probably get to even like, you know, less than 7,000 H100s running for three months to beat GPT-4. But of course that's gonna take a really awesome team.

204
00:27:00.328 --> 00:27:02.608
But there's quite a few companies that are gonna have that many, right?

205
00:27:02.988 --> 00:27:17.908
Open source will match GPT-4 but then it's like what about GPT-4 Vision or what about, you know, uh, 5 and 6 and you know, all this kind of stuff and like interact tool use and Dolly and like, that's the other thing is like there's a lot of stuff on tool use that the open source could also do that the, the GPU poor could do.

206
00:27:18.248 --> 00:27:25.567
Um, I think there are some folks that are doing that kind of stuff, agents and all that kind of stuff. I don't know. Uh, that's way over my head, the agent stuff. Yeah. It's over everyone's head.

207
00:27:25.768 --> 00:27:35.328
One more question on the sort of Gemini GPU rich essay. We've, we've had a very wide-ranging conversation already, so it's hard to categorize, but I tried to look for the Meena Eats the World document.

208
00:27:36.268 --> 00:27:45.948
Oh, it's not public. No, no, no, no, no, no. You've read it? Yeah. I've, I, I read it. So, so Noam Shazeer is like, I, I, I don't know. I think he's like- The GOAT. The GOAT. Yeah. I think he's the GOAT.

209
00:27:46.108 --> 00:27:53.928
In one year he published like Switch Transformers, like some, some, uh, Attention is All You Need obviously, but he also did the speculative decoding stuff. Yeah, yeah. Exactly.

210
00:27:53.968 --> 00:28:01.128
It's like, it's like all this stuff that we were talking about today was like hey, Noam. And obviously there's other people that are awesome that were, you know, helping and all that sort of stuff.

211
00:28:01.308 --> 00:28:06.148
Meena Eats the World was basically he wrote an internal document around the time where Google had Meena, right?

212
00:28:06.308 --> 00:28:11.668
But it was like he wrote it and he was like basically predicting everything that's happening now, which is that like large language models are gonna eat the world, right?

213
00:28:12.028 --> 00:28:18.268
In terms of, you know, compute and he's like the total amount of deployed FLOPS within Google data centers will be dominated by large language models.

214
00:28:18.488 --> 00:28:27.688
But back then a lot of people thought he was like silly for that, right? Like internally at Google. But you know, now if you look at it it's like, oh wait, [laughs] millions of TPUs.

215
00:28:27.728 --> 00:28:33.508
You're right, you're right, you're right. Okay. [laughs] We're totally d- getting dominated by like both, you know, Gemini training and inference, right?

216
00:28:33.548 --> 00:28:39.068
Like, you know, total FLOPS being dominated by LLMs was completely right. So my question was he had a m- bunch of predictions in there.

217
00:28:39.428 --> 00:28:45.928
Do you think there were any like underrated predictions that may not have yet have come, come true? Was he wrong on anything? Meena sucked, right?

218
00:28:46.068 --> 00:28:57.228
If you look at the total FLOPS, right, uh, you know, parameters times tokens times six, it's like tiny, tiny fraction of GPT-2 which came out just a few months later, which was like, okay, so he wasn't right about everything, but like maybe he knew about GPT.

219
00:28:57.288 --> 00:29:08.588
I have no clue. OpenAI clearly was like way ahead of Google on LLM scaling even then. People didn't really recognize it back in GPT-2 days maybe. Or the pe- the number of people that recognized it was maybe hundreds.

220
00:29:08.728 --> 00:29:21.488
Yeah. Tens. So we talked about transformer alternatives. The other thing is GPU alternatives. The TPU is obviously one, but there's Cerebras, there's Graphcore, there's Madax, Lemurian Labs, there's a lot of them.

221
00:29:21.668 --> 00:29:29.868
Thoughts on what's real, who's alive, who's kind of like a zombie company walking? You know, I, I mean, I mentioned like transformers were the architecture that won out.

222
00:29:29.908 --> 00:29:37.868
But I think, you know, the number of people who recognized that in 2020 was, you know, as you mentioned, probably hundreds, right? You know, for natural language processing maybe in 2019 at least, right?

223
00:29:38.188 --> 00:29:44.148
You think about a chip design cycle, it's like years, right? You know, so, so it's kinda hard to bet your architecture on the type of model that develops.

224
00:29:44.408 --> 00:29:53.608
But what's interesting about all the first wave AI hardware startups, you know, there's a ratio of, of memory, capacity, compute, and memory bandwidth, right?

225
00:29:53.888 --> 00:30:02.088
Everyone kind of made the same bet, which is I have a lot of memory on my chip, which is A, really dumb because the models have grew way past that, right? Even Cerebras, right?

226
00:30:02.128 --> 00:30:11.248
I mean, you know, like I'm talking about like Graphcore, it's called SRAM, which is the memory on chip. Much lower density but much higher speeds versus, you know, DRAM memory off chip.

227
00:30:11.408 --> 00:30:21.808
And so everyone was betting on more memory on chip and less memory off chip, right? And to be clear, right, for image networks and, and models that are small enough to just fit on your chip, that works.

228
00:30:22.008 --> 00:30:25.928
That is a superior architecture. But scale, right? Scale, scale, scale, scale.

229
00:30:26.288 --> 00:30:34.968
Nvidia was the only company that bet on the other side of more memory bandwidth and more memory capacity external- Also the right ratio of memory bandwidth versus capacity.

230
00:30:35.128 --> 00:30:42.448
A lot of people like Graphcore specifically, right, they had ton of memory on chip and then they had a lot more memory off chip, but that memory off chip was a much lower bandwidth.

231
00:30:42.608 --> 00:30:49.868
Same applies to SambaNova, same applies to Cerebras. They had no memory off chip but they, they thought, "Hey, I'm gonna make a chip the size of a wafer," right?

232
00:30:49.968 --> 00:30:58.048
Like, you know those guys they're, they're, they're silly, right? Hundreds of megabytes. We have 40 gigabytes. There's no, you know, and then oh crap, models are way bigger than 40 gigabytes, right?

233
00:30:58.348 --> 00:31:03.228
You know, kind of as, as the ones that people deploy. Everyone bet on sort of the left side of this curve, right?

234
00:31:03.548 --> 00:31:11.368
The interesting thing is that these, these New Age startups like Lumeria, like Madax, I won't get into what they're doing, but they're making much more rational bets.

235
00:31:11.468 --> 00:31:16.568
I don't know, you know, it's, it's hard to say with a startup like it's gonna work out, right? Obviously there's tons of risk embedded.

236
00:31:16.828 --> 00:31:23.488
But tho- those folks, you know, Jay Dewani of Lumerium and like Mi- Mike and, uh, and, and, and, and Ranier, they understand models.

237
00:31:23.708 --> 00:31:31.128
They understand how they work and if transformers continue to reign supreme, whatever innovations those folks are doing on hardware are gonna need to be fitted for that.

238
00:31:31.248 --> 00:31:36.308
Or you have to predict what the model architecture is gonna look like in a few years, right? You know, and, and, and hit that spot correctly.

239
00:31:36.348 --> 00:31:45.268
So that's kind of a background on those but like now you look today, hey, Intel bought, uh, Nervana which was Naveen Rao's MosaicMLs. He started MosaicML and sold it to Databricks somewhere recently.

240
00:31:45.308 --> 00:31:54.988
He's obviously leading LLMs and stuff there, AI there. In- Intel bought that company from him and then shut it down and bought this other AI company and, and now that company is kind of, uh, you know, got new chips.

241
00:31:55.228 --> 00:32:03.808
They're gonna release a better chip than the H100 within the next quarter or so. AMD, they have a GPU, MI300 that will be better than the H100 in a quarter or so.

242
00:32:03.848 --> 00:32:08.624
Now that says nothing about how hard it is to program it, but at least hardware-wise on paper It's better. Why?

243
00:32:08.664 --> 00:32:15.204
Because it's, you know, a year and a half later, right, than in the H100 or a year later than the H100, of course, and, you know, a little bit more time and all that sort of stuff.

244
00:32:15.244 --> 00:32:26.244
But they're at least making similar bets on memory bandwidth versus flops versus capacity, kinda following Nvidia's lead. The questions are like, what is the correct bet for three years from now?

245
00:32:26.584 --> 00:32:29.204
How do you engineer that? Uh, and will those alternatives make sense?

246
00:32:29.224 --> 00:32:39.004
The other thing is, if you look at total manufacturing capacity, right, for this sort of bet, right, you need high bandwidth memory, you need HBM, and you need large five nanometer dies, you know, soon three nanometer, whatever, right?

247
00:32:39.744 --> 00:32:44.144
Y- you need both of those components, and you need the supp- whole supply chain to go through that. We've written a lot about it.

248
00:32:44.164 --> 00:32:50.104
But, you know, to simplify it, Nvidia has a little bit more than half, and Google has, like, thirty percent, right, through Broadcom.

249
00:32:50.164 --> 00:32:54.964
So it's like the total capacity for everyone else, much lower, and they're all sharing it, right?

250
00:32:55.064 --> 00:33:03.504
Amazon's Trainum and Inferentia, Microsoft's in-house chip, and, you know, you go down the list and it's like Meta's in-house chip and also AMD and also all the...

251
00:33:03.604 --> 00:33:12.803
So all of these companies are sharing like a much smaller slice. Uh, their chips are not as good, or if they are, they're-- even though they're... You know, I mentioned, y- y- uh, Intel and AMD's chips are better.

252
00:33:12.984 --> 00:33:20.424
That's only because they're throwing more money at the problem kind of, right? You know, Nvidia charges crazy prices. I think everyone knows that. Their gross margins are pro- are insane.

253
00:33:20.644 --> 00:33:29.084
AMD and, and Intel and, and, and others will charge more reasonable margins, and so they're able to give you more HBM and et cetera for a similar price.

254
00:33:29.104 --> 00:33:34.904
And so that ends up letting them beat Nvidia, if you will, but their manufacturing costs are twice that in some cases, right?

255
00:33:34.944 --> 00:33:43.764
In the case of AMD, their manufacturing costs for MI300 are more than twice that of H100, and it only beats H100 by a little bit from, you know, performance stuff I've seen.

256
00:33:43.804 --> 00:33:50.904
So it's like, you know, it's, it's tough for anyone to like bet the farm on a alternative hardware supplier, right?

257
00:33:50.944 --> 00:33:57.964
Like, in my opinion, like, you should either just like be like, you know, a lot of like ex-Google startups are just using TPUs, right?

258
00:33:58.004 --> 00:34:05.844
And, and hey, that's Google Cloud, you know, after moving the TPU team into the cloud team, infrastructure team, sort of they're much more aggressive on external selling.

259
00:34:05.864 --> 00:34:10.344
And so companies like-- you even see companies like Apple using TPUs for training LLMs as well as GPUs.

260
00:34:10.404 --> 00:34:20.544
But either bet heavily on TPUs because that's where the capacity is, bet heavily on GPUs of course, and stop worrying about it and, and leverage all this amazing open source code that is optimized for Nvidia.

261
00:34:20.684 --> 00:34:33.624
If you do bet on AMD or, or Intel or, or any of these startups, then you better make damn sure you're really good at low-level programming and damn sure you also have a compelling business case and that the hardware supplier is giving you such a good deal that it's worth it.

262
00:34:33.904 --> 00:34:41.244
And also, by the way, Nvidia's releasing a new chip in... You know, they're gonna announce it in March, and they're gonna release it and ship it Q2, Q3 next year anyways, right?

263
00:34:41.284 --> 00:34:46.004
And that chip will probably be three or four times as good, [chuckles] right? And maybe it'll cost twice as much or fifty percent more.

264
00:34:46.084 --> 00:34:51.084
I, I hear it's three x the performance on an LLM a- and fifty percent more expensive is what I hear.

265
00:34:51.464 --> 00:34:57.664
So it's like, okay, yeah, nothing, nothing is gonna compete with that, even if it is fifty percent more expensive, right? And then you're like, okay, well, that kicks the can down further.

266
00:34:57.984 --> 00:35:04.004
And then Nvidia is moving to a yearly release cycle. So it's like very hard for anyone to catch up to Nvidia really, right?

267
00:35:04.044 --> 00:35:11.524
Are, are, you know, investing all this in other hardware, like if you're Microsoft, obviously who cares if I spend five hundred million dollars a year on my internal chip?

268
00:35:11.764 --> 00:35:15.024
Who cares if I spend five hundred million dollars a year on AMD chips, right?

269
00:35:15.064 --> 00:35:23.044
Like, if it lets me knock the price of Nvidia GPUs down a little bit, puts the fear of God within Jensen Huang, right, like, you know, then, then it is what it is, right?

270
00:35:23.084 --> 00:35:31.604
And, and likewise, you know, with Amazon and so on and so forth. You know, of course, their hope is that their chips succeed or that they can actually have an alternative that is much cheaper than Nvidia.

271
00:35:31.704 --> 00:35:39.104
But to throw a couple of hundred million dollars at a company, you know,'s product is, is completely reasonable. And in the case of AMD, I think it'll be more than a couple of hundred million dollars, right?

272
00:35:39.484 --> 00:35:53.984
But, uh, yeah, I think, I think alternative hardware is like it really does hit like sort of a peak hype cycle kind of end of this year, early next year, because all Nvidia has is H100 and then H200, which is just better, more mem- more memory bandwidth, higher memory capacity H100, right?

273
00:35:54.384 --> 00:36:01.984
But that doesn't beat what, you know, AMD are doing. It doesn't beat what, you know, even Intel's Gaudi 3 does. But then very quickly after, Nvidia will crush them.

274
00:36:02.224 --> 00:36:08.724
And then those other companies are gonna take two years to get to their next generation. You know, so it's g- it's just a really tough place and, and no one decides...

275
00:36:08.884 --> 00:36:16.684
You know, the, the, the main thing about hardware is like, hey, that bet I talked about earlier is like, you know, that's very oversimplified, right? Just memory bandwidth, flops and, and memory capacity.

276
00:36:16.884 --> 00:36:24.863
There's a whole lot more bets. There's a hundred different bets that you have to make and guess correctly to get good hardware, not even have better hardware than Nvidia, get close to them.

277
00:36:24.904 --> 00:36:27.664
And that takes understanding models really, really well.

278
00:36:27.863 --> 00:36:37.244
That takes understanding so many different aspects, whether it's power delivery or cooling or design layout, all this sort of stuff, and it's like how many companies can do everything here, right?

279
00:36:37.284 --> 00:36:44.404
It's like I'd argue Google probably understands models better than Nvidia. I don't think people would disagree. Um, and Nvidia understands hardware better than, than Google.

280
00:36:44.664 --> 00:36:54.004
And so you end up with like Google's hardware is competitive, but like does Amazon understand models better than Nvidia? I don't think so. And does Amazon better hardware-- understand hardware better than N- Nvidia? No.

281
00:36:54.344 --> 00:37:02.244
I'm, I'm also of the opinion that the labs are useful partners. They're convenient partners. They're not gonna buddy up as close as people think, right? I don't, I don't even think like...

282
00:37:02.584 --> 00:37:06.964
You know, I, I, I expect in the next few years that the OpenAI Microsoft probably falls apart too.

283
00:37:07.124 --> 00:37:14.604
I mean, I mean, they'll still continue to use GPUs and stuff there, but like I think that the level of closeness you see today is probably the closest they get.

284
00:37:14.984 --> 00:37:21.244
At some point, they become competitive if OpenAI becomes its own cloud. The level of the value that they deliver to the world,

285
00:37:22.184 --> 00:37:26.944
if you talk to anyone there, they truly believe it'll be tens of trillions, if not hundreds of trillions of dollars, right?

286
00:37:27.044 --> 00:37:33.924
In which case, obviously, you know, I know, I know weird corporate structure aside, you know, this is the same playing field as companies like Microsoft and Google.

287
00:37:34.184 --> 00:37:41.104
Google wants to also deliver hundreds of trillions of dollars of value, and it's like obviously you're competing, and Microsoft wants to do the same, and you're gonna compete.

288
00:37:41.384 --> 00:37:52.744
In general, right, like these, these lab partnerships are gonna be nice, but they're probably incentivized to, uh, you know, "Hey, Nvidia, you should... You know, can you, can you design the hardware in this way?"

289
00:37:52.804 --> 00:38:00.688
And Nvidia's like, "No, it doesn't work like that. It works like this." And they're like, "Oh, so this is the best compromise," right? Like I think- OpenAI would be stupid not to do that with Nvidia, but also with AMD.

290
00:38:00.828 --> 00:38:06.728
And but also, hey, like how much... And, and Microsoft's internal silicon, but it's like, how much time do I actually have, right? Like, you know, should I do that?

291
00:38:06.768 --> 00:38:13.868
Should I spend all my, you know, super, super smart people's time and limited, you know, this caliber of person's time doing that?

292
00:38:14.108 --> 00:38:20.868
Or should they focus on like, "Hey, can we get like asynchronous training to work," or like, you know, figure out this next multimodal thing? Or I don't know, I don't know, right?

293
00:38:20.908 --> 00:38:28.548
Like, it's probably better, you know, hey, can I eke out 5% more MFU and work on designing the next supercomputer, right? Like these kind of things. How much more valuable is that, right?

294
00:38:28.588 --> 00:38:38.828
So it's like, you know, it's, it's tough to see, you know, even OpenAI helping Microsoft enough to get their knowledge of models so, so, so good, right? Like, Microsoft's gonna announce their chip soon.

295
00:38:39.108 --> 00:38:46.588
It's worse performance than the H100, but the cost effectiveness of it is better for Microsoft internally, just because they don't have to pay the Nvidia tax.

296
00:38:47.088 --> 00:38:55.768
But again, like by the time they ramp it and, and all these sorts of things, it... Oh, hey, that only works on a certain size of models. Once you exceed that, then it's actually, you know, again, better for Nvidia.

297
00:38:55.828 --> 00:39:04.548
So it's like, it's really tough for OpenAI to be like, "Yeah, we, we wanna bet on, on Microsoft," right? Like, and hey, we have, you know... I don't know, what's the number of people they have now? Like 700 people?

298
00:39:04.888 --> 00:39:12.328
You know, of which how many do low level code? Do I wanna have separate code bases for this and this and this and this? And, you know, it's like, it's just like a, a big headache to... I, I don't, I don't know.

299
00:39:12.388 --> 00:39:20.088
I think it'd be very difficult to see anyone truly pivoting to anything besides a GPU and a T- a TPU, especially if you have-- if you need that scale.

300
00:39:20.328 --> 00:39:29.708
And, and that scale that the la- at least the labs, right, require is absurd. Google says millions, right, of TPUs. OpenAI will save millions of GPUs, right?

301
00:39:29.748 --> 00:39:35.868
Like, I truly do believe they think that that number of, of next generation GPUs, right? Like the numbers that we're gonna get to are like...

302
00:39:36.108 --> 00:39:42.428
I bet you-- I, I mean, I don't know, but I bet Sam Altman would say, "Yeah, we're gonna build a hundred billion dollar supercomputer in three years or two years," right?

303
00:39:42.468 --> 00:39:52.328
Like, and, and like after GPT 5 releases, if he goes to the market and says like, "Hey, I wanna raise a hundred billion dollars at five hundred billion dollar valuation," I'm sure the market would give it to him, right?

304
00:39:52.368 --> 00:39:59.708
Like [laughs] and then they build that supercomputer, right? Like, I mean, like, I think that's like truly the path we're on, and so it's hard to, hard to imagine. Yeah, I don't know.

305
00:40:00.308 --> 00:40:10.188
One point that you didn't touch on, and Taiwan companies are famously very chatty about the fruit company. Should we take Apple seriously at all in this game, or they're just in a different world altogether?

306
00:40:10.648 --> 00:40:18.168
I, I respect their products, but like I don't think Apple will ever release a model that you can get to say really bad things.

307
00:40:18.328 --> 00:40:24.088
There's all these jailbreaks, but also like as soon as they happen, like, you know, it gets fed back into OpenAI's like platform, and it gets them...

308
00:40:24.388 --> 00:40:32.048
It's, it's like being public and open is accelerating their like ability to make a better and better model, right? Like the RLHF and all this kind of stuff.

309
00:40:32.328 --> 00:40:42.368
I, I, I don't see how Apple can do that structurally like as a company. Like, the, the fruit company ships perfect products or like, or else, right? That's why everyone loves iPhones, right?

310
00:40:42.408 --> 00:40:50.948
Like, and, and all these like, uh, uh, open source firms and like all these folks are doing exactly that, right? Building a bigger and better model every, you know, every few months.

311
00:40:51.288 --> 00:40:56.748
And I don't know how Apple gets on that train. But, you know, at the same time, there's no company that has more powerful distribution, right?

312
00:40:56.888 --> 00:41:07.208
Are people in Taiwan concerned that it will come to a point where China will just claim Taiwan? I think, I think that a lot of people there are not super concerned, but there's some people that are super concerned.

313
00:41:07.988 --> 00:41:16.048
I think, I think especially after like, you know, instability across the world in, in, in Europe and, uh, in, in the Middle East and, uh, even Africa.

314
00:41:16.088 --> 00:41:26.948
If you look at any of the stuff they're building up, it seems very clear, and if you talk to a lot of people, they, they think Chi- China will invade Taiwan in '27 and, or '26 in April or, or in, uh, September as sort of the best, uh, time frames, right?

315
00:41:26.988 --> 00:41:36.568
Like a lot of people believe that's what will happen, right? Maybe the semi analysis analyst point of view is, is it feasible to build this capacity up in the US? Uh, no. No, right?

316
00:41:36.948 --> 00:41:45.708
People don't understand how fragmented the semiconductor supply chain really is and how many monopolies there are. The US could absolutely shut, shut down the Chinese semiconductor supply chain. They won't, but...

317
00:41:45.928 --> 00:41:52.648
And China could absolutely shut down the US one, actually, by the way. But more, more relevantly, right, is like, you know Austria has two companies?

318
00:41:52.868 --> 00:42:01.408
Like the country of Austria in Europe has two companies that have super high market share and very specific technologies that are required for every single like, like chip period, right?

319
00:42:01.688 --> 00:42:09.508
There is no chip that is less than seven nanometer that doesn't get touched by, uh, this one Austrian company's tool, right? And, and there is no alternative really.

320
00:42:09.708 --> 00:42:16.308
And there's another Austrian, you know, and I, uh, it's, it's-- And there's another Austrian company, likewise, everything two nanometer and beyond will be touched by their tool.

321
00:42:16.528 --> 00:42:21.828
And it's like b-both of these companies are like doing well less than a billion dollars in revenue, right? So it's like you think it's so inconsequential.

322
00:42:22.248 --> 00:42:29.048
No, there's actually like three or four Japanese chemical companies. Same, same idea, right? It's like the, the supply chain is so fragmented, right?

323
00:42:29.088 --> 00:42:36.328
Like people only ever talk about where are the fabs, where, where they actually get produced. But it's like, I mean, TSMC in Arizona, right? TSMC is building a fab in Arizona.

324
00:42:36.628 --> 00:42:42.668
It's, it's quite a bit smaller than the fabs in, in, in Taiwan, but even ignoring that, those fabs still have to ship everything to Taiwan back anyways.

325
00:42:42.828 --> 00:42:47.368
And also they have to get what's called a mask from Taiwan and get sent to, get sent to Arizona.

326
00:42:47.548 --> 00:42:55.048
And by the way, there's these Japanese companies that make these chemicals that need to ship to, you know, like Tok and Shinetsu and, you know, it's like... And, and hey, it needs this tool from Austria no matter what.

327
00:42:55.088 --> 00:43:04.788
It's like, oh, wow, wait, actually like the entire supply chain is just way too fragmented. You can't like re-engineer and rebuild it, uh, on a snap, right? It's just like, uh, it's just complex to do that.

328
00:43:05.108 --> 00:43:16.028
Semiconductors are more complex than any other thing that humans do. Without a doubt, there's more people working in that supply chain with XYZ backgrounds and more money invested every year in R&D plus CapEx.

329
00:43:16.068 --> 00:43:22.868
You know, it's like, it's just by far the most complex supply chain that humanity has, and to think that we could rebuild it in a few years is absurd.

330
00:43:23.008 --> 00:43:26.928
Alternative universe, the US kept Morris Chang and he built it here [laughs] right?

331
00:43:26.988 --> 00:43:37.868
Like it, it was just one guy that [laughs]- Yeah, in an alternative universe, Texas Instruments, uh, communicated to Morris Chang that he would become CEO, and so he never goes to Taiwan and, you know, blah, blah, blah, right?

332
00:43:37.908 --> 00:43:46.188
Yeah, no. But I, I, you know, that's just... Also, I think, I think the world would probably be further behind in terms of technology development if that didn't happen, right?

333
00:43:46.548 --> 00:43:50.528
Like technology proliferation is how you accelerate the pace of innovation, right?

334
00:43:50.588 --> 00:43:59.688
So the, the, you know, the dissemination to, oh, wow, hey, it's not just a bunch of people in Oregon at Intel that are leading everything, right? Or, you know, hey, a bunch of people in Samsung, Korea, right?

335
00:43:59.968 --> 00:44:01.048
Or Hsinchu, Taiwan, right?

336
00:44:01.088 --> 00:44:12.908
It's actually all three of those, plus all these tool companies across the country, in the, the Netherlands and, and in Japan and the US and, you know, it's, it's millions of people innovating on a disseminated technology that's led us to get here, right?

337
00:44:13.188 --> 00:44:18.612
I don't even think, you know, if, if Morris Chang didn't go to Taiwan, would we even be at- Five nanometer? Would it be at seven nanometer? Probably not, right?

338
00:44:18.672 --> 00:44:26.772
Like there's innovations that, that, you know, happen because of that, right? Let's get a quick lightning round done on SemiAnalysis branded one.

339
00:44:27.412 --> 00:44:38.572
Uh, so the first one is, what are like foundational readings that people that are listening today should read to get up to speed on like semis? I think the easiest one is, is the PyTorch 2.0 and Triton one that I did.

340
00:44:38.812 --> 00:44:43.392
You know, there's the advanced packaging series. There's the Google infrastructure supremacy piece.

341
00:44:43.472 --> 00:44:51.932
I think that one's really, uh, critical because it explains Google's infrastructure quite a bit from networking through chips, through all that sort of history of the TPU a little bit.

342
00:44:51.972 --> 00:45:00.652
Maybe like AMD's MI300 piece, it talks a lot about the one that we did on that are very good. And then obviously like, you know, like, I don't know, probably like Chip Wars by Chris Miller.

343
00:45:00.692 --> 00:45:03.332
Who doesn't recommend that book, right? Um, it's a really good book, right?

344
00:45:03.392 --> 00:45:14.232
I mean, like I would say Gordon Moore's book is freaking awesome because you gotta think about, right, like, you know, LLM scaling laws are like Moore's law on crack, right?

345
00:45:14.512 --> 00:45:24.912
Uh, kind of like, you know, in a different sense. Like, you know, if you think about all of human productivity gains since the '70s is probably just off of the base of semiconductors and technology, right?

346
00:45:24.972 --> 00:45:35.952
Of course, of course, people across the world are getting, you know, access to oil and gas and all this sort of stuff, but like, at least in the Western world since the '70s, everything has just been mostly innovated because of technology, right?

347
00:45:36.372 --> 00:45:43.292
Oh, we're able to build better cars because semiconductors enable us to do that. Or we're able to build better software because we're able to connect everyone because semiconductors enabled that, right?

348
00:45:43.352 --> 00:45:53.952
So it's like, that is like, I think that's why it's the most important industry in the world, but like seeing the frame of mind of what Gordon Moore has written, you know, he's got a couple, you know, papers, books, et cetera, right?

349
00:45:54.392 --> 00:45:55.692
Only the Paranoid Survive, right?

350
00:45:55.732 --> 00:46:07.172
Like, I think, I think like that philosophy and thought process really translates to the now modern times, except maybe, you know, humanity has been an exponential S-curve, and this is like another exponential S-curve on top of that.

351
00:46:07.212 --> 00:46:18.272
So I think that's probably a good, good readings to do. Has there been a equivalent pivot? So Gordon, like that classic tale was more of like his, the pivot to memory. From memory to logic. Yeah, yeah. Yeah.

352
00:46:18.292 --> 00:46:24.192
And then was there, is there, has there been an equivalent pivot, um, in, in semi's history of, of that magnitude?

353
00:46:24.512 --> 00:46:41.732
I mean, I, I mean, like, you know, some people would argue that like, you know, Jensen, you know, he, he basically didn't care about- [laughs] He only cared about, you know, like gaming and, and 3D professional visualization and like rendering and things like that until like he started to learn about, uh, AI, and then all of a sudden he's going to like universities like, "You want some GPUs?

354
00:46:41.772 --> 00:46:50.512
Here you go." Right? Like, I think there's even stories of like, you know, not so long ago, Nerips, when it used to have the more unfortunate name, he would go there and just give away GPUs to people, right?

355
00:46:50.572 --> 00:46:59.212
Like, there's like stuff like that. Like, you know, very grassroots, like pivoting the company. Now, like you, you look on gaming forums, and it's like everybody's like, "Oh, Nvidia doesn't even care about us.

356
00:46:59.272 --> 00:47:05.812
They only care about AI." And, you know, it's like, [laughs] yes, you're right. [laughs] They only care... They mostly only care about AI.

357
00:47:05.852 --> 00:47:12.872
And, and the gaming innovations are only because of like, they're putting more AI into it, right? It's like, but also like, hey, they're doing a lot of chip design stuff with AI.

358
00:47:12.912 --> 00:47:21.592
And, you know, I think, I think that's a like, not a-- I don't know if it's equivalent pivot quite yet, but, you know, because the digital, you know, logic is a pretty big innovation, but I think that's a big one.

359
00:47:21.612 --> 00:47:26.972
And, you know, likewise, it's like, you know, what did, what did OpenAI do, right? What did they pivot? How did they pivot?

360
00:47:27.012 --> 00:47:34.272
They left the like-- A lot of-- most people left the culture of like Google Brain and DeepMind and, and decided to build this like company that's crazy cool, right?

361
00:47:34.312 --> 00:47:40.052
Like, and does things in a very different way and like is innovating in a very different way. So could you consider that a pivot even though it's not inside Google?

362
00:47:40.412 --> 00:47:50.412
They were, they were on a very different path with like the Dota games and all that before they eventually found like GPTs as the, as the thing. So it, it was a full like started in 2015 and then- Yeah...

363
00:47:50.432 --> 00:47:58.112
like really pivoted in 2019 to be like, "All right, we're the GPT company." Yeah. Yeah. If I could classify them. I, I don't-- I, I'm sure there's OpenAI people who are yelling at me right now.

364
00:47:58.292 --> 00:48:07.712
Okay, so just a general question about, you know, I'm a fellow writer on, on Substack. You are obviously managing your, your consulting business while you're also publishing these amazing posts.

365
00:48:07.852 --> 00:48:16.752
How do you-- What's your writing process? How do you source info? Like when do you sit down and go like, "Here's the theme for the week"? Do you, do you have a pipeline going out? Just anything you can describe.

366
00:48:17.212 --> 00:48:24.472
I'm thankful for my, you know, my teammates 'cause they are actually awesome. Like, and they're much more, um, you know, directed, focused to working on one thing.

367
00:48:24.812 --> 00:48:33.092
You know, or not one thing, but a number of things, right? Like, you know, someone who's just expert on X and Y and Z in the semiconductor supply chain. So that really helps with the con- the, that side of the business.

368
00:48:33.672 --> 00:48:40.552
I most of the times only write when I'm very excited or, you know, it's like, hey, like we should work on this and we should write about this.

369
00:48:40.592 --> 00:48:53.652
So like, you know, one of the most recent posts we did was we explained the manufacturing process for 3D NAND, you know, flash storage, uh, gate all around transistors and 3D DRAM and all this sort of stuff 'cause there's a company in Japan that's going public, Kokusai Electric, right?

370
00:48:53.712 --> 00:49:01.912
It was like, okay, well, we should do a post about this, and we should explain this. But like, it's like, okay, we, we you know... And so Myron, he, he did all that work, Myron Shi in, in most of the work, and awesome.

371
00:49:01.952 --> 00:49:08.872
But like usually it's like there's a few like very long in-depth back burner type things, right? Like that took a long time.

372
00:49:09.172 --> 00:49:13.572
Took, you know, over a month of, of research, and Myron knew- knows this stuff already really well, right?

373
00:49:13.712 --> 00:49:21.032
Like there's stuff like that that we do and, and that like, like builds up a body of work for our consulting and, and, and some of the reports that we sell that aren't, you know, newsletter posts.

374
00:49:21.172 --> 00:49:36.412
But a lot of times the process is also just like, well, like Mina Eats the World is the culmination of reading that, having done a lot of work on the supply chain around the TPU ramp and CoWoS and HBM capacities and all this sort of stuff to be able to, you know, figure out how many units and that Google's ordering all this sort of stuff.

375
00:49:36.432 --> 00:49:44.352
And then like also like looking at like open source is like all just that, all that culminated in like I wrote that in four hours, right? I sent it to a couple people, and they were like, "No, change this, this, this.

376
00:49:44.612 --> 00:49:52.892
Oh, you know, add this 'cause that's really gonna piss off, you know, the open source community." I'm like, "Okay, sure." Um, [laughs] and then posted it, right? So it's like there's no like specific process.

377
00:49:53.192 --> 00:49:59.872
Unfortunately, like the most viral posts, especially in the AI community, are just like those kind of pieces rather than the like the really deep, deep like...

378
00:50:00.192 --> 00:50:10.472
You know, obviously, like what was in the, uh, Gemini Eats the World post, you know, obviously, hey, like we, we do deep work. There's a lot more like factual, not leaks. You know, it's just factual research.

379
00:50:10.512 --> 00:50:13.312
Hey, we, across the team, we go to forty-plus conferences a year, right?

380
00:50:13.412 --> 00:50:23.612
All the way from like a photoresist conference to a photomask conference to a lithography conference, all the way up to like AI conferences and, you know, all, everything in between, networking conferences and piecing everything across the supply chain.

381
00:50:23.652 --> 00:50:27.552
So it's like that's like the true like work and like, yeah, I don't know.

382
00:50:27.652 --> 00:50:36.832
It, it is sometimes bad to like have the infa- in- inf- infamousness of, you know, only people caring about this or the GPT-4 leak or the Google has no moat leak, right?

383
00:50:36.852 --> 00:50:39.072
It's like, but like, you know, that's just like stuff that comes along, right?

384
00:50:39.112 --> 00:50:46.832
You know, it's really focused on like understanding the supply chain and how it's pivoting and who's the winners, who's the losers, what technologies are inflecting, things like that.

385
00:50:47.032 --> 00:50:54.568
Where is the best place to invest resources, you know, sort of like stuff like that and, and accelerating or, uh, capturing value, et cetera. Awesome.

386
00:50:54.748 --> 00:51:04.888
And to wrap, if you had a magic genie that could answer any question that would change your worldview, uh, what question would you ask? That's a tough one. Um...

387
00:51:06.328 --> 00:51:09.528
Like you, you operate based on a set of facts about the world right now.

388
00:51:09.908 --> 00:51:17.208
Then there's maybe some unknowns where you're like, "Man, if I really knew the answer to this one, I would do so many things differently, or I would think about things very differently."

389
00:51:18.068 --> 00:51:27.228
So I'm of the view, at least everything that we've seen so far, is that large scale training has to happen in an individual data center with very high-speed networking.

390
00:51:27.268 --> 00:51:35.048
Now, everything doesn't need to be all to all connected, but you need very high-speed networking between all of your, uh, your chips, right? I would love to know,

391
00:51:36.048 --> 00:51:48.828
you know, "Hey, magic genie, how can we build artificial intelligence in a way that it can use multiple data centers of resources where there is a significantly lower bandwidth between pools of resources, right?"

392
00:51:49.408 --> 00:51:54.428
Because that would instantly... Like, one of the big bottlenecks is how much power and how many chips you can get into a single data center.

393
00:51:54.828 --> 00:52:03.128
So like, A, Google and OpenAI and Anthropic are working on this, and I don't know if they've solved it yet. But if they haven't solved it yet, then what is the solution?

394
00:52:03.368 --> 00:52:10.988
Because that will, like, accelerate the scaling that can be done by not just like a factor of 10, but like orders of magnitude because there's so many different data centers, right? Like if you...

395
00:52:11.028 --> 00:52:18.708
You know, across the world and, you know, uh... Oh, if I could pick up, you know... If I could effectively use 256 GPUs in this little data center here and then with this big cluster here.

396
00:52:18.748 --> 00:52:26.908
You know, how can you r- make an algorithm that can do that? Like, I think that would be like the number one thing I'd be curious to know if, how, what...

397
00:52:27.288 --> 00:52:36.808
Because that changes the world significantly in terms of how con- we continue to scale this amazing technology that people have invented, uh, over the last, you know, five years. Awesome.

398
00:52:36.828 --> 00:52:38.408
Well, thank you so much for coming on, Dylan.

399
00:52:38.928 --> 00:53:01.148
[upbeat music]
