WEBVTT

1
00:00:00.080 --> 00:00:12.340
[on hold music] Okay. Hi, we're here in the OpenAI studio with Mia and Olivia from the Frontier Eval team, or however you want to introduce yourself.

2
00:00:12.560 --> 00:00:21.340
Maybe, Mia, you want to introduce-- name what you do at OpenAI, and we can get it started. Sure. Hi, I'm Olivia. I'm on the Frontier Eval team. Great. Short and sweet. Hi, I'm Mia.

3
00:00:21.540 --> 00:00:34.580
I am a VP of research at OpenAI, and my teams are the Codex team, the Human Data team, and the Alignment team, and we work a lot with Olivia's team on Frontier Eval. Yeah. Very exciting.

4
00:00:34.640 --> 00:00:39.400
And as by my understanding, you were part of the original team that worked on Suitebench Verified as well. Yeah.

5
00:00:39.460 --> 00:00:45.240
Olivia's team, the Frontier Eval team, and the Human Data team collaborated on creating Suitebench Verified.

6
00:00:45.560 --> 00:00:54.300
So you've, you've seen the evolution of coding benchmarks over time, and, uh, I think it was roundabout sort of mid to late twenty twenty-four when you first published Suitebench Verified.

7
00:00:54.820 --> 00:01:03.840
Things have evolved a lot since then. What's the blog post that you have worked on that you're, that we're releasing today? Like, what, what is the sort of con-- what is the main thesis that you're pushing out?

8
00:01:04.420 --> 00:01:11.760
So the main thesis is that Suitebench Verified has been one of the North Star coding benchmarks that the field has looked at to measure coding progress.

9
00:01:12.160 --> 00:01:20.340
But recently we've seen that progress has kind of stalled, and this we realized that this is because the eval is effectively saturated and also highly contaminated.

10
00:01:20.780 --> 00:01:29.300
So at this point, we think that it's not really measuring coding performance improvements well anymore, and we think that the field should move away from this towards other benchmarks. Like Suitebench-Pro.

11
00:01:29.340 --> 00:01:39.740
Like Suitebench-Pro, yeah. Amazing. Yeah. I, um, uh, one of the jokes I always have is, like, there's a group chat with all the labs, and everyone just takes turns to increment like zero point one on [laughs] charts.

12
00:01:39.780 --> 00:01:47.040
And then it's like, okay, well, you have the best coding model, I guess, because you're zero point one higher, but it's not super convincing at this point. No. Yeah. No. So cool.

13
00:01:47.280 --> 00:01:54.820
I think the-- let's, let's sort of reset on, like, what was your orig-original work that you guys did for Suitebench Verified, which I think was pretty substantial.

14
00:01:54.880 --> 00:02:03.680
Like, it was, like, a very significant investment for an OpenAI, which, like, people still don't appreciate. And then what were the dissatisfactions that we, that we found over time, right?

15
00:02:03.740 --> 00:02:07.660
So l-like, what, what was Suitebench Verified that should, should, that people should know about?

16
00:02:08.240 --> 00:02:25.220
Suitebench Verified was a kind of a cleanup of original bench- academic benchmark from a lab at Princeton called Suitebench, and the agent is basically given a code base and a task that was sourced from a real-world repository and GitHub issue and was asked to solve a task and is graded on whether some tests pass.

17
00:02:25.820 --> 00:02:32.260
And at the time, this was quickly became a popular benchmark because at the time the field didn't really have good real-world coding benchmarks.

18
00:02:32.740 --> 00:02:48.740
But then when OpenAI took a look at the benchmark as part of one of the evals we wanted to track in our preparedness framework, folks started realizing that some of the cases where agents were failing were due to bad problem setups rather than just to models being dumb.

19
00:02:49.340 --> 00:03:06.220
So folks at OpenAI did a pretty extensive human data campaign, hiring like almost a hundred real-world software engineers to go through the problems and figure out, like, are the tasks well specified, are the tests actually fair, and kind of created a curated set of, like, five hundred tasks that we thought were much better.

20
00:03:06.580 --> 00:03:13.220
It's just, it's maybe, it's hard to, to overstate like the amount of effort that it took to, like, create that benchmark.

21
00:03:13.240 --> 00:03:29.660
It was literally like many expert software engineers reviewing the problems, like sequentially multiple times and to, you know, basically like three different experts independently decided it. The, the- Yeah.

22
00:03:29.840 --> 00:03:33.600
You didn't have to do that. You just tripled your cost for just- I mean, we had to do it.

23
00:03:33.700 --> 00:03:42.440
We had to do it actually because it's quite a hard task to like look at something like a, a problem and, and the patch and then like it's not just the problem and the patch, right?

24
00:03:42.460 --> 00:03:47.860
You have to like understand it in the context of the code base that the human or, or the models and to solve the task.

25
00:03:48.160 --> 00:04:01.300
So it's a very complex problem, and it was definitely needed to have three reviews, and I think like maybe we should have done more [laughs], but it was definitely a lot of effort to get there. Yeah.

26
00:04:01.360 --> 00:04:04.200
And there's, there's more, but people can read the, the blog posts for that.

27
00:04:04.420 --> 00:04:13.500
I will note that you guys set a trend in verifying benchmarks because I just recently saw, I think Qwen had a HLE verified for Humanity+Llama verified. Yeah.

28
00:04:13.540 --> 00:04:18.279
Which like, so now everyone's verifying everything, which is nice and good and like extra quality there. Okay.

29
00:04:18.420 --> 00:04:29.440
So but the, I think the, the meat of it is the, the, this, this was a lot of like, well, here's the issue or problem statements and then here's the, here's the diffs, here's the golden tests, and here's some regression tests, right?

30
00:04:29.500 --> 00:04:36.900
That's, that's like the rough set about these five hundred problems. And there's some contamination always happens because all the Suitebench Verify was fully open, I think.

31
00:04:37.320 --> 00:04:47.100
You, you did have canaries, but like, you know, stuff, stuff leaks. There's like multiple avenues that like the problems are sourced from open source repos. Yes.

32
00:04:47.480 --> 00:04:58.460
So it's not just like when we usually publish evaluations, we publish evaluations, and then we, we add canary strings to ensure that, you know, they are easily filtered out at training time.

33
00:04:58.880 --> 00:05:09.400
Obviously, if you use sort of like data from like open source- Just GitHub. Yeah. You don't have actually like a ca-canary string in there. Yeah.

34
00:05:09.440 --> 00:05:17.900
And you- And these are also, like some of these are very popular repos, like the Django repository. So you're gonna see like many instances being used kind of throughout GitHub. Yeah, yeah.

35
00:05:18.080 --> 00:05:26.720
You, just before recording, you were telling me that you found this in your own chain of thought with like GPT 5.2 also seeing that like they had extra knowledge or something. Yes.

36
00:05:26.780 --> 00:05:36.060
So this was an example where the task asked the agent to influence something, but it wasn't told that there was this specific argument that the test was going to be looking for it using.

37
00:05:36.460 --> 00:05:45.440
But in the GPT 5.2 chain of thought, we actually saw instances of the model reasoning like, "Hey, I think that at some later version of this repository, they implemented this particular argument.

38
00:05:45.480 --> 00:05:52.040
Maybe I should add it in." Yeah. So this is an example of a test that like would be pretty impossible to pass without this contamination knowledge. Yeah.

39
00:05:52.080 --> 00:05:58.060
And I think you found that sort of first, right, and it triggered like a whole investigation both like in our own models- Oh...

40
00:05:58.080 --> 00:06:11.060
and also in other frontier models like in the market and like understanding how contaminated the benchmark is like across the industry. What else did you find? I mean, [laughs] that's... I have to double-click on this.

41
00:06:11.420 --> 00:06:16.140
So we-- And when I say we, this is mostly- Yeah... work from other folks at our team, not, I don't want to take credit but- Not the Mario we- Yes.

42
00:06:16.560 --> 00:06:22.560
But so we did some analysis on, first of all, are the tests actually fair?

43
00:06:23.080 --> 00:06:34.200
And so this happened by first taking all the problems that o3 couldn't solve reliably, and then again, getting a lot of humans to do basically another pass of kind of digging into, you know, what's wrong.

44
00:06:34.380 --> 00:06:45.320
Is it the same exact analysis, or were they reading o3's output and going, "Here's where o3 went wrong"? I think it was... I mean, it was definitely like a scope to the set of problems that models failed.

45
00:06:45.340 --> 00:06:54.080
And I believe they were able to look at like what the model solutions look like versus what the- This is, so this isn't the same work as the original- It's- It's not exactly the same work. It was like a, a deeper dive.

46
00:06:54.140 --> 00:06:58.480
It's like, okay, which are the problems that we don't see any model solving?

47
00:06:58.820 --> 00:07:07.100
It's like, is there something fundamentally wrong with those problems or is there something, you know, wrong with, like, are the models just not smart enough to solve the problems?

48
00:07:07.480 --> 00:07:17.220
So that's kind of like what we, what we dug into. Yeah. And you found some? Oh, yes. Like, in over half of the problems that were investigated in that deep dive, there was one problem or the other.

49
00:07:17.580 --> 00:07:31.580
I think the most common problem are, like, overly narrow tests where there's some particular implementation detail that the tests were looking for but wasn't specified in the problem description, so it wasn't fair to expect that model to make that particular design choice.

50
00:07:31.900 --> 00:07:41.260
Like, one pretty blatant example are cases where the task asks you to implement some feature, but the tests are looking for you naming that argument or that function with a particular name.

51
00:07:41.680 --> 00:07:51.120
But if you na- chose another reasonable name, the test would fail. Yeah. And another set of types of bad tests are tests that are just looking for additional features that were never mentioned in the problem description.

52
00:07:51.840 --> 00:07:57.660
But, but that is significant. It's like, that means that if you pass a test, actually, like, you probably did, like, a really good job.

53
00:07:57.980 --> 00:08:06.220
But just because you didn't pass a failed test doesn't mean that your implementation wasn't, like, a good one, right? So it was just like we only accept, like, very narrow

54
00:08:07.140 --> 00:08:16.620
versions of solutions and, like, not the whole space- Yeah... of, of, like, viable and sort of like good solutions to the problem. Yeah.

55
00:08:17.000 --> 00:08:24.560
I think it's important that you're doing this because it, in some way it is you in twenty twenty five, six, going back in time and correcting your own work, right?

56
00:08:24.600 --> 00:08:28.460
Because you could have caught all this in v- in the original Verified work. I think so.

57
00:08:28.740 --> 00:08:38.500
It's definitely much harder to find a problem in the abstract than when you're looking at a very smart agent's best effort solution and trying to compare it. It is harder or harder or- Sorry.

58
00:08:38.560 --> 00:08:47.000
It's much easier when you have those- Much e- exactly. Yeah. Sorry. I think, I think also, like, at the time when Suitebench Verified was published, I think it was, like, a very strong benchmark.

59
00:08:47.060 --> 00:08:56.680
It's not like we're, we're like, "Oh, this is not... This wasn't, like, a strong benchmark at the time." I think this is something that a lot of benchmarks go through, like, as an evolution, right?

60
00:08:56.720 --> 00:09:06.300
Like, when they start to become, like, popular and, like, viable, it's because they measure something, like, important, and models maybe do, like, twenty percent correct on them, sometimes even less.

61
00:09:06.700 --> 00:09:11.340
And sort of, like, people have something to hold on and, and improve models on, on these benchmarks.

62
00:09:11.400 --> 00:09:20.640
And by the time that you hit, like, very high performance on the benchmarks, like, additional, like, point one percent improvements become sort of, like, meaningless.

63
00:09:20.900 --> 00:09:31.900
And so, like, at the time, I think, you know, that benchmark was, like, super valuable, and it, it taught, like, us and, like, the industry a lot. It's just, like, now, at the point that we are at now- Yeah...

64
00:09:31.920 --> 00:09:46.220
where models are as strong as they are now, we're kind of starting to measure not necessarily like what we want to measure, which is, like, coding capability of our agents, but, like, the agent's ability to, like, correctly guess how to name a specific function.

65
00:09:46.400 --> 00:09:53.840
Yeah. And, and that isn't really what we are, like, want to measure at this point. Yeah. I, I think that's fair. Is, is there...

66
00:09:54.380 --> 00:10:00.940
I mean, if I, if I asked you to ballpark it, like, uh, most models are, most frontier models are now, like, eighty something. Is there, like...

67
00:10:01.380 --> 00:10:10.220
What's the actual, like, number on Suitebench Verified that you [laughs] did you guess as, like, the ceiling? Or... [laughs] I guess it's really hard to say. Like, I...

68
00:10:10.920 --> 00:10:20.100
When GPT 5.2 came out, folks took a look and found that it was solving, like, thirty-one problems that were in the set of should be very hard to solve without contamination problems.

69
00:10:20.140 --> 00:10:28.299
So I think it's quite possible that that number is already something that we've hit if you didn't have contamination at all. Fair enough. Hard to say, though. Yeah. Uh, cool.

70
00:10:28.520 --> 00:10:35.860
We're gonna stop reporting Suitebench Verified, right? And then Suitebench Pro will, will be sort of the next one, which is an effort from Scale. What's your sort of comparison analysis?

71
00:10:35.900 --> 00:10:39.340
What's, what attracts you to Suitebench Pro? The first one I think is just that it's harder.

72
00:10:39.800 --> 00:10:46.920
For Suitebench Verified, I think a, I mean, like, ninety percent of the problems are things that were estimated to take, like, an expert software engineer, like, less than an hour.

73
00:10:47.160 --> 00:10:56.160
They're, like, very well specified, very self-contained. And the Suitebench Pro problems are just bigger and harder, and there's much more headroom on the eval because it's not saturated.

74
00:10:56.380 --> 00:11:06.200
Uh- Yeah, like categories of, like, one to four hours and four plus. So that kind of thing. Yeah. And it's more diverse. Lots of r- r- repositories, multiple languages, qualitatively more different types of problems.

75
00:11:06.500 --> 00:11:09.580
So all that's great. On the contamination side, we also think it's better there.

76
00:11:10.320 --> 00:11:31.520
So the way we were measuring for contamination for Suitebench Verified was with this little, like, contamination auditor agent, which is given the description of the task and the patch and the task ID and told to go take this target model and kind of as an open-ended, like, set of questions, try to find questions that will manage to kind of reveal what contamination might be lurking in that model.

77
00:11:32.120 --> 00:11:42.160
And in Suitebench Verified, we found many instances of contamination across, uh, across OpenAI models, across, like, Claude Opus 4.5, Gemini Flash.

78
00:11:42.680 --> 00:11:55.580
And in all of these, we saw things like regurgitating the ground true solutions, things like, in some cases, giving, like, the task IDs and other things that are pretty clear evidence of at minimum familiarity with the repositories.

79
00:11:55.680 --> 00:12:01.920
Remember. Yeah. Suitebench- I mean, know a task ID, that's- [laughs]... that- Yeah. Yeah. In Suitebench Pro, on the other hand, we don't see this.

80
00:12:01.980 --> 00:12:13.600
I think there, the auditor agent found some, like, very light evidence that maybe a couple models might be very lightly familiar with, like, one or two of the source repositories, but it's very different than Suitebench Verified.

81
00:12:14.140 --> 00:12:15.400
So yeah, less contamination is good.

82
00:12:15.968 --> 00:12:29.368
I think there also, like we should expect that at some point, like that-that's not going to be like the right benchmark anymore, and like it's a field we kind of have to continue to like move on and like find harder and more representative problems- Yeah...

83
00:12:29.388 --> 00:12:34.868
that we can measure our capabilities on. Awesome. So let's go into that. I think that there are a lot of...

84
00:12:34.908 --> 00:12:47.428
I think we also prepped this in the, in the pre-chat was, w- people feel a qualitative difference when they're using 5.1 to 5.2 to 5.3, and it's not super expressed in th- these benchmarks because they're, they are on a number of these things.

85
00:12:47.648 --> 00:12:55.848
What capabilities do you really want to benchmark in a, in a ideal coding benchmark? You know, I guess like agentic coding benchmark, whatever you call it, uh.

86
00:12:56.268 --> 00:13:07.128
I mean, one thing is kind of open-ended design decisions, places where the problem maybe is a little bit underspecified and seeing if the model can make reasonable design decisions. What's a reasonable prompt for that?

87
00:13:07.168 --> 00:13:17.768
Like just five code me a B2B SaaS and make no mistakes or, you know, like [laughs] That's, that's the meme, but like, okay, what's like, what's like an actual usable open-ended problem like that? Like [laughs] Sure.

88
00:13:17.808 --> 00:13:27.008
I mean, maybe an example could be finding a way to speed up a particular part of a code base, but there might be m- multiple different ways to speed it up. Yeah. There are dedicated performance benchmarks.

89
00:13:27.068 --> 00:13:33.888
I think you guys have one, Sufficiency, or is that, is that... Oh, no, I think that's, that's Sophia Harris's group. Mm-hmm. But yeah. Yeah. I mean, that, that is a good one.

90
00:13:33.968 --> 00:13:48.148
I think there's just many, many things that people like value about working with, with software engineering agents. I think Sweet Ven- Sweet Bench Verified obviously measure like some...

91
00:13:48.828 --> 00:14:01.308
It measures like some important capability, which is like given like a description of a GitHub issue, can you produce like a patch that solves that issue, you know, satisfactorily?

92
00:14:01.808 --> 00:14:09.588
And like obviously there's like some issues with the, with the benchmark that means that now that we're at like eighty percent we don't really trust like further improvements on it.

93
00:14:09.828 --> 00:14:14.748
But like it does measure something that is like a real, real, like capability of models.

94
00:14:15.168 --> 00:14:24.008
But I think as a field, we're like moving beyond sort of, you know, can my coding agent like solve a small like GitHub issue for me, right?

95
00:14:24.088 --> 00:14:34.568
And so we are starting to look at like much more lo-longer term tasks, right, like that don't take like fifteen minutes, but maybe like hour, sometimes days.

96
00:14:35.068 --> 00:14:48.788
And then beyond sort of like what kind of tasks can my a-agent solve, like there might be things that are kind of a bit harder to grasp, right? Like Olivia talked about sort of like does it have like design taste, right?

97
00:14:48.828 --> 00:15:02.628
Like, does it solve the problem the way that, you know, my team likes to solve problems? Got it. The... Is the code nice, right? Like is it, is it well-written? Like th- is it sort of like clean code, right?

98
00:15:02.668 --> 00:15:07.528
Like people care about this. Is it maintainable in the future? People care about a lot of these

99
00:15:09.008 --> 00:15:19.428
l- maybe less tangible, less tangible and like harder to measure, frankly, things that, that are still like super meaningful for people that are working with coding agents. Yeah.

100
00:15:19.708 --> 00:15:26.808
So I mean, these are all qualities that are obviously the no longer the low-hanging fruit. Like w-we have no idea how to eval this.

101
00:15:27.168 --> 00:15:30.808
I think the, the, the simple question maybe the, the, the there's sort of two forks in the road.

102
00:15:30.948 --> 00:15:36.928
One is the sort of very human intensive, money intensive path, which is hire a bunch of contractors and try to annotate this.

103
00:15:37.208 --> 00:15:46.888
The other is use an LLM to, to proxy it and try to align the LLM so that it can give you a reasonable proxy. Which of those would you want? I... Want, you want to do both?

104
00:15:47.508 --> 00:15:52.088
I think like maybe you should talk about GDPVal as like an example. Um, sure.

105
00:15:52.408 --> 00:16:07.148
So GDPVal is an eval that was a- again, produced by a collaboration between Human Data team and the friend of Eval's team, and it's trying to measure whether agents can do kind of a variety of like real world white collar work.

106
00:16:07.628 --> 00:16:16.668
That was an eval where grading is very hard, r- requires kind of a lot of kind of like domain knowledge on exactly what are you looking for in each different context. Yeah.

107
00:16:16.728 --> 00:16:23.788
Across like fifteen, sixteen white collar jobs, uh, professions like I... That take up a significant part of GDP, which is great. Yeah.

108
00:16:23.808 --> 00:16:31.768
Kind of high level professions and then a lot of like different granular sub-professions. Listen, I have said like I, I'm a big fan. It's- Cool... it, it is, this is the eval for, for AGI, basically.

109
00:16:31.928 --> 00:16:36.227
[laughs] But part of because it was so hard to...

110
00:16:36.667 --> 00:16:49.648
It required so much kind of like domain knowledge that the Human Data team hired like a lot of people from these professions to be very involved in creating tasks and creating the gold solutions and trying to help create rubrics and so forth so we can grade it reliably.

111
00:16:50.188 --> 00:17:00.648
So basically take the GDPVal, which is a general listing, take that same approach to apply it to code, and you roughly have like a, a rough roadmap. I think it's an interesting- Yeah... solution, I think.

112
00:17:00.668 --> 00:17:10.848
What you're pointing out is an important problem, which is sort of this, this like how realistic is it and like do you know what, what we want to do is like

113
00:17:11.868 --> 00:17:23.328
coding agents should write code that, you know, we think is good. And so it's like asking human, it's actually like a, a good way to ensure that. It's also kind of a slower, like complex way to do that.

114
00:17:23.668 --> 00:17:33.128
And so part of why I think, you know, Sweet Bench Verified ended up being super popular and where we are seeing like ALBA benchmarks like this being super popular is like it's very easy.

115
00:17:33.668 --> 00:17:45.408
It could even be easier, but like validating that a solution passes all the tests, it's like pretty trivial once you can like run the tests in, in like your, on your computer or wherever you're running them.

116
00:17:45.888 --> 00:17:51.528
And you can kind of like, okay, is it correct or is it not correct? And you can kind of aggregate that and that is super simple.

117
00:17:51.588 --> 00:18:04.428
But it doesn't tell you, it's like, you know, did the model like solve the problem like well, like, you know, ugly with like what if actually like an open source maintainer of that project have like merged that PR, like that it doesn't tell you.

118
00:18:04.848 --> 00:18:17.064
But there is a lot of value in having benchmarks that are both like easy to compare across the industry and also that can be sort of run really fast without human involvement. Yeah. Amazing.

119
00:18:17.344 --> 00:18:30.084
Your teams also put out other kinds of evals that are related, like the... I think there's an RL, a paper bench, and then they- they sort of like the more sort of re- recursive im- self-improvement type evals.

120
00:18:30.464 --> 00:18:37.744
How much should that figure into mainstream coding evals? You know, like is there, is there some way in which those things join together?

121
00:18:38.604 --> 00:18:45.684
So we were asking like, should we build e- should also be building evals for the self-improvement evals? Are you saying do coding evals currently cover that?

122
00:18:45.864 --> 00:18:54.144
I mean- I think, uh, I, I, I just think like those are some of the most advanced evals that we have, and we're not using them in the normal path, and it's just...

123
00:18:54.404 --> 00:19:05.064
It, it's an interesting split between, well, here's evals f-for coding normal things, and then here's the one for machine learning, [laughs] that is like completely different, right? Yeah, I think you get what I mean.

124
00:19:05.484 --> 00:19:15.904
And that's mostly like a safety argument, I guess. But also, like it's actually really useful for people to understand if the model is really good at, like AI code, basically. Yeah. Oh, yeah.

125
00:19:15.944 --> 00:19:24.404
Like my guess is that part of the reason that a lot of benchmarks so far haven't focused as much on the AI coding is just a question of like what datasets are easy to gather. Yeah.

126
00:19:24.444 --> 00:19:33.184
Because a lot of the like, you know, state-of-the-art AI code bases are proprietary. So if we make evals for that, like we're probably not gonna release them. Yeah.

127
00:19:33.244 --> 00:19:39.564
And it's a harder for people in the field to make evals that kind of measure like, is this a realistic research coding workflow? Yeah.

128
00:19:39.604 --> 00:19:45.264
I do think that it's good for the field to try to measure these skills in a public way. I think it's just harder to make it realistic. Sure.

129
00:19:45.304 --> 00:19:54.504
And then one more thing that a lot of people are trying to do, which is like sort of, well, in- instead of like a percentage of zero to one hundred, maybe we re-do-redenominate in dollars, right?

130
00:19:54.624 --> 00:20:04.704
So you have Sweet Lancer and all that. Other people are doing like vending bench, whatever. Any, any alpha in those or are they, are they... You, you still want like a traditional academic benchmark? I think

131
00:20:05.724 --> 00:20:12.524
in a way, like there's like different ways to measure the same thing, right? If we're like, "Oh, this is like how much money it produces." Yeah.

132
00:20:12.564 --> 00:20:20.464
It's a fairly similar thing to saying like, "Oh, this problem would take like a human, you know, two hours to solve," or something like that. Usually they, they're like fairly like correlated, right?

133
00:20:20.504 --> 00:20:30.204
Like however, you know, much it would take like a human to solve that problem kind of determines the value that we ascribe like a solution.

134
00:20:30.844 --> 00:20:45.084
And so I, I do think that is like an important thing is like h-how complex and how sort of long running are the tasks that we are like able to entrust our agents with. Yeah.

135
00:20:45.124 --> 00:20:57.304
And so I think that that's like an important piece, but I, I think here sort of monetary value, time or complexity, they all kind of like try to capture like a similar thing. Yeah. Okay.

136
00:20:57.464 --> 00:21:02.804
So they're, they're all proxies for some amount of increasing ca-capacity that we wanna measure. I think that's a good thing.

137
00:21:02.864 --> 00:21:11.044
I think the only other sort of major player in this field is Meter, which has done the sort of long grass, and congrats, you guys have completely destroyed the curve for that. Any takes on that?

138
00:21:11.244 --> 00:21:18.424
Obviously you, you've come up really well, so like, it, it looks good, but I don't know if like that approach is something that you wanna incorporate in your own work making evals.

139
00:21:19.004 --> 00:21:29.824
Uh, this is the long autonomy task if you're- Yeah. Yeah, no, we're very familiar and, and, and we, we, we work with, with Meter on these evaluations. So like we, we do appreciate them.

140
00:21:30.284 --> 00:21:34.524
I think then they're using time, right? They're not using money. So I think like that was your question.

141
00:21:34.624 --> 00:21:42.564
I think like complexity, however we can sort of like quantify it, is really important to understand like where our models are, are, are getting to. Okay.

142
00:21:42.684 --> 00:21:49.144
Com-complexity is the abstract thing, and then it projects down to time, projects down to story points, whatever, dollars. Great.

143
00:21:49.384 --> 00:21:54.884
One last question on just like, just the overall preparedness framework is, you know, I was actually kind of looking at... People mention the preparedness framework a lot.

144
00:21:55.044 --> 00:22:02.884
I don't think it's well explained to a lot of people, and you actually have a nice website where it's like, I think it's like test and like inform and teach something.

145
00:22:02.924 --> 00:22:08.464
And I, I feel like you, you actually do a lot of work there and, and I don't know if you want to talk about how the preparedness framework applies.

146
00:22:09.024 --> 00:22:14.864
So the preparedness framework is OpenAI's kind of like public framework for how we track frontier risk.

147
00:22:14.924 --> 00:22:28.104
So these are kind of capabilities that are typically dual use, like you can use them for good things or bad things, but we wanna at least keep an eye out for the bad things to make sure that we ha- both we as a company and like the broader society are kind of prepared to handle the potential downsides.

148
00:22:28.704 --> 00:22:32.023
And so at the moment, we kind of track three different categories.

149
00:22:32.484 --> 00:22:42.884
One is kind of bio risk, another is cybersecurity, and a third is kind of research automation and model autonomy, and that's kind of what ties most into the Sweet Bench- Research auto, got it...

150
00:22:42.964 --> 00:22:54.384
where coding is not all of automating research, but it is one very important key component. And so we initially created Sweet Bench Verified as part of like building out evals for that model autonomy work stream.

151
00:22:54.684 --> 00:23:03.624
And now, uh, I think for like we have to move beyond that towards looking more at like can models actually start to actually automate research workflows. Yeah. Amazing. Great.

152
00:23:03.704 --> 00:23:12.084
Amie, anything else to add on just the general what people should know about preparedness and how evals and human data alignment all work together in that?

153
00:23:12.704 --> 00:23:18.444
I think maybe the thing that I would say is that we really appreciate...

154
00:23:19.344 --> 00:23:27.464
We, we work really hard to build these evals and, and so we- that's why we published Sweet Bench Ver-Verified, and that's where we're like sharing GDP eval, these sorts of things.

155
00:23:27.824 --> 00:23:37.944
We also deeply appreciate like other people and the entire field to kind of build evals and, and share them, and we use them. Like Sweet Bench Pro, we're like, "Yes, that, that's a better eval now. We should use them."

156
00:23:38.344 --> 00:23:50.484
So would really encourage people to find more ways to create and share evals that we can, we and the entire field can use to measure like progress- Yeah...

157
00:23:50.564 --> 00:23:57.544
on, on, on like a variety of capabilities, including, including coding, because it's important to understand sort of where we are.

158
00:23:57.964 --> 00:24:07.764
Amie had to leave, but we're, we're just kind of talking a little bit about like the, the future directions that we want evals to go. Mm-hmm. And I, I think here, here we can dive in on like give us

159
00:24:08.704 --> 00:24:15.284
work, good work on these, these, these things. We'll talk to you. You know, like here's your platform to make a call for what you're looking for.

160
00:24:15.684 --> 00:24:19.884
I think a few things that would be useful, I'd say, first of all, really, really hard tasks.

161
00:24:19.944 --> 00:24:34.384
Like the kinds of things that would take top-notch engineers months or teams weeks would be quite good, especially if grading is reliable and grading is like, you know, you have, for example, like rubrics that have been sourced and validated by many people in the field.

162
00:24:34.564 --> 00:24:41.884
I think that'd be quite valuable. I think also benchmarks on kind of creating products end to end. I think if people are like coding more, that would be quite useful.

163
00:24:42.384 --> 00:24:54.104
I think a third thing that I'd say that is maybe not quite an eval, but I think is still relevant to the kind of overall mission of like we as a field and as a world should be tracking like where are these capabilities going.

164
00:24:54.404 --> 00:25:03.844
I'd like to see more metrics tracking like real world usage. Like how much is AI actually being used in the field? Uh, how much is it, you know, replacing people's jobs?

165
00:25:04.044 --> 00:25:08.344
How much is it, you know, augmenting people, speeding people up? Just like real world metrics. Yeah. Yeah.

166
00:25:08.364 --> 00:25:17.264
The, the, the replacement thing is always like a sensitive one on, on a sort of PR side of things, but, you know, we create new jobs that, that manage the old jobs and [laughs] that's how it is.

167
00:25:17.484 --> 00:25:25.544
Yourself, like, you know, I think i-in terms of the frontier evals that, that OpenAI is really going to excited to push, like you, you put out really good work every single time.

168
00:25:25.984 --> 00:25:31.724
What should people expect from, from OpenAI itself? I'm not sure I can say what we're gonna- General directions... soon.

169
00:25:31.764 --> 00:25:40.884
I mean, general directions, I think looking at real world impact, like re-real world, real- GPT eval two. Yeah, whatever... that kind of stuff. Yeah. Yeah. Yeah. Amazing. Okay.

170
00:25:40.924 --> 00:25:42.564
Well, I'm excited for more real world impact.

171
00:25:42.644 --> 00:25:52.304
I, I think you guys have, you know, really made a lot of progress and I think taken a lot of industry leadership for Sweet Bench Verified and, and now moving on to Sweet Bench Pro. So thank you for doing this.

172
00:25:52.344 --> 00:25:58.824
Thank you for being so transparent, and I think people will respond in kind. Yeah. Yeah. Thanks for your time. Thank you.

173
00:25:58.824 --> 00:26:11.824
[outro music]
