[upbeat music] Welcome to the Rebooting show. I am Brian Morrissey. This week's conversation is with Cloudflare Chief Strategy Officer, Stephanie Cohen.
In an era when most of big tech has turned its back on publishers, I'm not gonna name anyone, but you know who we're, we're talking about, you know, one company has quietly become something of a trusted ally, and that's Cloudflare.
Cloudflare serves a vital role in the open web. It is a content delivery network, and that basically means that it is infrastructure for the open web.
Its tools underpin everything from security to performance to how modern media companies deliver content at scale without relying on the big tech platforms.
And on this week's episode, I speak to Stephanie, who leads Cloudflare's media and entertainment business, about how the company's network has become the connective tissue of the modern internet and why it's found traction with publishers when others in tech have not, and what it means to be a neutral partner at a time when the neutrality of the information ecosystem is in short supply.
Now, Cloudflare is making aggressive moves to stand on the side of publishers when it comes to AI slurping up their content. This is...
I don't think it's exaggerated to say it's an existential threat to many publishers, and so Cloudflare as an ally is noteworthy.
But before we get to that conversation, I wanna tell you about something that I think you will find useful from the Rebooting's partner, Beehiiv.
When you're running a media business, email is usually the backbone of it, and getting it right is unfortunately usually harder than it should be, and that's where Beehiiv comes in.
This is a platform that was built by the people who scaled Morning Brew to millions of readers, and they created the platform they wish they had when they were in your shoes.
So it is no surprise that companies like Time and the Texas Tribune are already running their newsletters on Beehiiv. It is designed for serious media operators, not marketers bolting on a newsletter as an afterthought.
If you are running a publication with more than a hundred thousand subscribers, Beehiiv is built for you. Their team has scaled newsletters at places like The Information, Puck, and Morning Brew.
They understand the challenges big publishers face, from seamless migration to enterprise-grade deliverability to monetization. Beehiiv makes it easy to grow, retain, and engage audiences at scale.
So if you want to see what your next stage of growth could look like, go to beehiiv.com/trb. That is spelled B-E-E-H-I-I-V.com/trb.
You can meet with their team of growth and newsletters experts today. Thank you so much to Beehiiv.
Be sure to check out actually last week's conversation I had with Tyler Deng, Beehiiv's CEO, for more on how Beehiiv is helping publishers, large and small, put the audience at the center of their strategies.
Now, on to my conversation with Stephanie. [upbeat music] All right, Stephanie, thank you so much for joining me. We're, like, across the World Trade Center plaza from each other.
[chuckles] You're in World Trade One, I'm in World Trade Three, so I can't see you across, but you're across the plaza. Across the internet and across the yard, basically. Yeah.
[chuckles] And probably this data is, is flowing through Cloudflare.
I wanna get into how Cloudflare is, is doing pay per crawl with every single publisher obviously is very focused on the changes to distribution patterns and therefore their, their monetization with AI in particular.
There's tons of AI crawling going out there. But I just wanna sort of, you know, level set with the role Cloudflare plays, which is unique.
I think a lot of times we think about the, the sort of giants of, of technology on the distribution side because they control the distribution, but Cloudflare operates on the infrastructure layer.
So give us a Cloudflare 101. Yeah. So infrastructure is, is really cool, by the, by the way. Yeah. I did a lot of things before I- I like roads, I like plumbing, I like- Yeah, so I- I love it.
[chuckles] I did a bunch of things before I came to Cloudflare, all at Goldman Sachs, and one of them was that I ran the general industrial business, which was, you know, physical infrastructure I've always found incredibly interesting.
So Cloudflare is just the internet version of infrastructure. So today, Cloudflare works with millions of customers to help them build and run their applications so that they're fast and secure.
Our mission is that we help build a better internet. Two most important words there are help and internet, which I'm sure we'll get into as part of this conversation. Yeah.
But, but your question is we do that by having this very large interconnected network.
So the Cloudflare network today sits in three hundred and twenty-five locations in more than a hundred and twenty-five countries where it's our server running our software. And like- Yeah...
the magic of the Cloudflare network is that we're capable of running every piece of software in every single server, and so that makes us incredibly reliable, fast, and secure.
And so what do we do on top of that, like, layer of infrastructure, is that we help your application so that traffic gets to you fast and that the traffic you don't want gets deflected. Yeah.
We also protect your data, your employees, and your network when they're on the internet, and then we help you... You can build applications on that. So- Right...
so we are this incredible network that help other people then build their businesses- Yeah... on top of it. So you're s- you're CDN, right? Content delivery network.
And, and a lot of that is basically ca- I mean, there's a lot of data moving back and forth. I'm gonna sort of like... [chuckles] But there's, there's a lot of data moving back and...
That's what the internet is, it's data moving back and forth, right? And- Yeah...
and you, you guys sit on the side to make sure that whether it's through caching, that the data is very fast when you go on a website and, like, the, it renders very quickly.
That is thanks to, to CDNs, correct?Absolutely.
And, you know, one of the amazing things about the internet is that there's this great diversity was developed on the internet, but you have to be able to be on the internet in order- Yeah... to do that.
One of the problems is if you put a website on the internet, it's gonna get attacked. Like, that's just the world we live in.
Cloudflare showed up and said every one of those websites, legal websites, has a right to be on the internet and free from attack. So it means that all of these millions and millions- Right...
of individual businesses- So it's not just speed, it's security, 'cause there's all kinds of malicious characters out there, whether they're part of a foreign government or just someone who's trying to grab a credit card information.
And so that's a critical role that you play in defending, like, the sort of internet. But it also, I think... And, and the reason I wanna get into this is because there's, like, a quasi
r- regulatory role almost, like, a policy role because you have to make decisions, and some of them are, they're, they're so... You know, it... Phishers. You know, a phish it's pretty easy.
It's not a very controversial decision to make, though I don't think... Well, I guess the phishers are very pro-phishing, but most people wouldn't regret. You're not gonna get a lot of people protesting that.
But, you know, because you have this, I think it's, like, 20% of the internet end up going through Cloudflare? Like- More than 20% of the internet sits behind- More than 20%... Cloud- Yeah... breadth of the internet.
So I mean, you, you make decisions that affect, you know... It has, like, a... I don't know. How would you put it? Like, a policy. Yeah. It's kind of like- And we-...
to me, it's like, it's like when California puts emission standards out, like, they become...
It's more than just California because, like, you can't serve all of the country a- and without, like, California, so the, the emission standards become the de facto.
So, like, when you guys make a decision, it has a greater impact even than just, quote, unquote, "just 20-odd percent of the, the internet traffic." It... So I think it's important to now dig into- Sure...
some of the complexity around this, which will make not complicated when we talk about it, which is that, like, every piece of traffic that comes through the Cloudflare network, we're, we're scoring to figure out, like, is this a re- a really bad guy, gal?
Yeah. Is this a bot? Is this a human?
Like, we're trying to figure out what's coming at the Cloudflare network, and the real, like, benefit, amazing part of the Cloudflare network is because we have millions of free customers in all of those locations, we have really, really great information about what's going on on the internet.
And so that allows us to be able to tell what's going on.
And the reality is that I, I would say the vast majority of the decisions that are being made, one, they're being made by machines, but two, they're, they are pretty obvious. Like, DDoS attack is deflected. Yeah. Right?
Like, bad actors, deflected. Everything else that is, quote, unquote, "a judgment call" is really made at the individual domain level. So the website owners are making decisions about what gets through and what- Okay...
doesn't get through. And so they're making that decision, and so some of them will say, "I want nothing but a human on my site." You can imagine the types of sites that say nothing but a human.
There's others that really want that type of traffic, and so they're letting more of that through. And so we're giving them the tools, and we're giving them the ability to do that, but they make those choices. Right.
So the ultimate decision is made at that, at the endpoint. And, and we'll get into the, the, the devil being in the defaults and, and whatnot. Absolutely. So let- let's talk about Paper Crawl. I was across the plaza.
You guys had a, had a very late party for a Monday night. I'm still shaken by it. But it was, it was to herald a new [laughs] um, a new internet, and obviously, look, AI is...
It's inescapable at any conversation, particularly with publishers, and usually it can be a little bit of a depressing conversation because there's a lot of fear of what...
First of all, it's bigger than AI, but the, the changes in distribution patterns of the internet are just shrinking a lot of a publisher's, you know, the traffic to them, and they built business models around that, that traffic.
And so they're trying to rapidly diversify their models. They always wanna talk to me about their events and, and various things that have nothing to do with putting ads on web pages.
They still have a, a, a large business in putting ads on web pages, and they need to get people to those web pages. And, you know, AI is... It's an existential threat.
I mean, you put in, like, Armageddon and media, you get, like, many results. It's very disturbing. Like, I think we, we need to stop using [laughs] that. [laughs] It's not very good vibe.
And so the question is, like, how is there going to be a new economic bargain for the internet? Google, to me, was always the linchpin of the internet.
Before we started, I was talking about how, you know, I'm covering them since before they were public. And, you know, they've played this arbiter role on the internet.
Uh, also a de facto regulator really because, because they just had such an outsized role in distribution.
And broadly, I always felt like they almost got too much grief because, like, they were balancing a bunch of different constituencies, and they did it imperfectly, and obviously in a self-interested manner.
But, you know, it's a tough job. It's a good job. But it's a very lucrative job, by the way.
But i- it's kind of tough to, to please everyone all the time, and at least their interests were broadly aligned with a, a healthy open internet.
Their, their f- They, they were making all that money off of the open internet. You know? That, that's been changing, and AI to me is just an accelerant on a lot of that stuff.
So what did, what were you guys seeing on... And it's beyond, obviously, Google and OpenAI, but what were you, you seeing on the Cloudflare side, because you see the data, right?
That was a little bit worrying and, and made you want to, you know, devote resources to even get involved in this? So you spend a lot of time with the publishers. You know? They're our customers.
And so we started hearing from them a, a while ago now. Actually, the first thing we did around this happened about a year ago. We started to hear exactly what you were hearing, which was that traffic...
There was less traffic coming to their sites. And- Human traffic... less human traffic- Yeah... coming to their sites, so less eyeballs on- Yeah... on their pages. And we heard lots and lots of complaints.
By the way, I do at some point wanna get into this. We're now hearing that from lots of other industries. It's not just-The, the publisher space.
But, but it was- but hit them first, which is unsurprising given the business model. And so then we went back and we looked at the data.
And so we actually on the Cloudflare site, this great site called Radar, which has tons and tons of information on it, which will allow you to see this real time. Mm-hmm.
But we looked at the data, and the data was, was more than we would've expected and was changing faster than we would've expected.
And so we can see how often bots are crawling a site, and we can see how often they're referring traffic.
And those numbers, so the rate you want more traffic, human traffic to your site, like per crawl, and those numbers were at least 10 times worse than they were a decade ago, and for some bots they, you know, versus where Google had been, they were tens of thousands of times worse, which is again, kind of unsurprising when you take a step back, which is that they're crawling a lot and there's very little traffic going back to the site.
Yeah. And we said, "You know what? This is like a real problem, and so we need to do something about it." And this is beyond... I'm sorry. The, the- but this is beyond just, quote-unquote, "training data." Absolutely.
This is... So- Okay... interestingly, despite, you know, what I, I think s- you know, some people report, on our network, we still see the vast majority of crawling comes from training. Okay.
It doesn't mean there isn't other types of crawling, but the vast majority of the crawls that are happening are still training-related crawls, but doesn't mean that that's not gonna change over time, and it doesn't mean we're not seeing things for RAG and operators and, you know, agents.
But the vast majority of the crawling is still training related today. Wait. Can you... 'Cause you brought up RAG. Can you explain RAG to the people who aren't again into it? Yeah. Yeah. No, no, yeah.
[laughs] The, the idea really is that if you're asking a question to an AI agent, they're...
or a chatbot, they're gonna use the model to come up with the answer, but they're also gonna check to make- see what the most recent information is.
And so to make sure that they're getting the most recent information in answering your question, which I do think is- Okay.
So that's when, like, when they say, like, checking whatever, like when you're on ChatGPT, is that, like then it's going out with like RAG or no? When they're, they're processing everything through their, their...
I mean, by the way, all the models don't run exactly the same way, but they're- Right...
they're processing the data based on the actual model and what's been trained, and then what we have seen, and we've seen this since the announcement, that they are heavily dependent on recent information.
And it obviously depends on the, what question you're asking. But you can imagine- Sure...
that if you got asked a question and you didn't know what's happened over the last two weeks, like you don't have access to recent information, your answer is gonna be stupider than if you- Yeah...
have access to the most recent information, and so it's pulling- Right... information from the open internet every time it's answering a question.
So the, the, the, the, the bargain used to be you can crawl our site, particularly with, like, Google, you know, Search was like their, you know, crawler, but like you can crawl our site, you're gonna basically scrape...
I mean, I remember back, you know, covering this, like, they're scraping the data. It's like, okay, you're gonna scrape some data. They're gonna claim it's not scraping or whatever. You're...
But, you know, the, the trade, traffic was the trade, right? Like a- at the end of the day, it was like Google had this imperfect system that was like, "Look, we're gonna, we're gonna basically scrape your sites.
We're gonna take little snippets on there, and if you sue, we're, we have way more and expensive and better lawyers, so it doesn't really matter. Uh, but in return, we're gonna send you traffic.
You're gonna monetize that traffic. Oh, by the way, we, we have the ad tech stack too, [laughs] so, you know, we're gonna get a piece of that."
But, like, it's better to have customers, you know, to be in business together 'cause your interests are more likely aligned.
Little, little bit of a different bargain I feel like now because the, the, the amount of crawling that's be- that's taking place just overall does not match up with the amount of traffic that's coming back.
That currency has sort of been devalued. Yeah. I think the underlying thing you're talking about is that there's a view that the internet was free, right? Yeah.
So we've kind of grown up with the idea that the internet was free, but the reality is the internet was not free. Mm-hmm. It was being paid for by, by advertisers. And Google, to your point, like facilitated that model.
There are others, but Google is facilitating- Sure... that model, and so when that model breaks down, then that economic model breaks down.
So yes, there's a bunch of crawling that was going on, has gone on, continues to go on. Previously, that would...
The amount of crawling would be better linked to the amount of traffic that was being sent back to your site.
But now when people are reading the derivative rather than the original, they can get the answer, so they get the answer from the crawling, but they never go back to the original site- Right...
which means you cannot monetize subscriptions, you cannot monetize ads, and by the way, we think you should not underestimate ego in all of this, which is just the idea that you, someone is reading what you wrote. Yeah.
For sure. And so explain the... Obviously, this is, this is an issue. It, it's an existential issue for many publishers, and Cloudflare is uniquely positioned to address this, I guess. So why...
Explain, explain the system of, like, paper crawl and how- Right... it works.
So I think what I'd like to do is talk about what we ki- all the changes we made, which w- are linked to paper crawl, but paper crawl in isolation doesn't really make- Okay... a ton of sense without everything else.
Okay. One, what we announced on July 1st and made you show up at, at midnight was because it was Content Independence Day.
So instead of ringing in the new year, we rang in Content Independence Day and pressed the big button of, uh, we changed the defaults on our network. Okay. So what did we do?
One, we changed the defaults for new customers, so new customers, new domains on the Cloudflare network, blocking training by default.
We blocked that either at the network level, if that works with a specific crawler, or we instruct the crawler to not train using a robots.txt message, which we can come back to. That's the first thing we did. Yeah.
The second thing we did was that we made it significantly easier to see who was scraping your site for all different types of reasons, and we gave you the ability to turn each one of those on or off.
And I say all that because it mattersCreating a market without having the conditions for a market doesn't work. If you don't have scarcity, there's no market.
And so the changing of the defaults, but really this idea that you would have more transparency and more control over who was on your site was really the most important thing that we did, because that's the only way for a market to then develop.
We then launched a pay-per-crawl beta with the simple idea being that you can block, allow, or charge. So you can say, "I don't want you on my site," block.
"Fine, come to my site because I already have a deal with you," or, "Because I want you on my site," or, "I feel like I'm getting traffic."
Or, "Neither of those really are great solutions for me, so what I really like to do is have you pay me to be on my site." The simplest way to do that was pay-per-crawl.
I think we can all agree that, like, we're at the very beginning of this, and so- Mm...
we'll figure out how the market needs to develop and what the right business model is, and is it one business model or many business models, but that is the, the most simplistic thing that you can do because it's...
You can tell when it's happening. Like, you can calculate a crawl. And so we launched pay-per-crawl and, and I'm happy to talk about- Yeah... kind of where we are today. So I guess the...
I mean, the first thing is, like, I...
'Cause I get this, 'cause like I, I, I did a podcast with, with Neil Vogel, the, the CEO of now People Inc., back in, in Cannes, and, and we talked about this issue, and he was like... I was talking...
I was, you know, skipping ahead to the money, to the marketplaces. He said, [chuckles] "We gotta develop scarcity first.
We have to, like, you have to understand, like, what is going on and whether you, you can block them." So I mean, do you see evidence that you can actually block them? 'Cause I mean, you mentioned, like, robots.txt.
And tell me I'm wrong, there's a lot of nuances here, but, like, I always thought that the whole point of robots.txt was that publishers could say, "Don't crawl this. Don't crawl these pages. They're transaction pages.
We don't want that indexed or anything."
And then it seemed like, obviously there are no, like, government, governing bodies seemingly on the internet, so, like, then it was just like, oh, some people were just like, "Yeah, we're, we, we, we don't, we're not...
We're ignoring robots.txt. This is not, like, some, what, is Interpol gonna arrest us?" It's, it's not some law or something. But explain, like, first of all, like, does the... Is this effective?
Can you, can, can you actually block this crawling? And, and because I, I just assumed that they would just go around it- So yes... in some ways. Yes, we can block.
It's actually the core business, the thing we started out at the beginning- Okay... of what Cloudflare does, we've been doing for 15 years. So yes, we can block is one.
Two, the vast majority of actors on the internet are good actors.
We actually launched a site that's goodaibots.com which actually tells you which bots obey instructions, and the vast majority of bots obey instructions on the internet.
What if you are using a provider, someone like Cloudflare, but there are others, if you are using...
If you have bot protection and you want to block a bot that then disobeys your robots message, you h- you have ways to, to then go after them and to stop them. And so the... Our...
We feel quite strongly that, yes, you can block. I would tell you that we have lots of evidence that blocks are effective. Like, what is the evidence?
One, large publishers that are blocking, that are, like, who have deals with some and blocking everyone else, would tell you that the complexion of their conversation has changed.
Two, we, we have evidence, we've written blog posts about it, of actors that are blocked and then try to evade the blocks, that we can stop them, and that when they don't get access to the information, they start making up w- the answers.
So we've made them less intelligent in terms of what they're answering. Yeah.
And three, we anecdotally hear that there are customer complaints with large models because they no longer have access to the same level of up-to-date information.
And so many of these companies are running consumer businesses that require that they provide high-quality product to their end customers.
And so as they have access to less information, as you create scarcity, there will and should be demand for, for that information. For content that is easier to make scarce, we already see the benefit.
For content that's harder to make scarce, meaning it's out there, it's more difficult.
We do believe that the world will move towards high-quality, original content because that's, in, in terms of that's where most of the value will be, which feels like a- Yeah...
a good thing for us to all have an incentive to create. I hope so. I hope so. That was my utopian- Yeah. No, I love it. I love it... vision for life. [chuckles] I wanna... It's Friday, it's... The weekend is coming.
I li- I like it. I think the question ends up being for a lot of publishers, they're downstream of technology giants fighting intergalactic battles, like sometimes literally, they're in space.
And the publishing industry, while those in it, you know, it's, it's, it's... and it is, like, societally very valuable, it's a very small, it's a very small industry. There's not a lot of money in it.
It's very easily squashed. Publishers typically are pretty easy to push around, and I...
there's a long history, I don't need to go through, that I think publishers against technology companies, they're la- it's like the Washington Generals against the Globetrotters, [chuckles] like as far as the record goes at this point.
And so, you know, ultimately the question is, like, do they get some measure of leverage?
And I know Matthew Prince has said, I think he said it at, at that event, that, like, that these, that these companies want to do deals, right? That they want to. They, they want someone else to go first.
And so I think there, there has to be... There's no, like, silver bullet, like, solution to any of these issues.
Cloudflare is very influential, obviously, like we talked about, it's still 20 odd percent, like, and it seems like these companies don't want to cut the check except for in a few s- small cases, right?
And, like, what do you see? Because I, I was asking, like, John Roberts actually, right, one of, on, on Neil's team, at that event, I was like-Are, am I gonna see their products get worse? Like, am I... Like, is that...
'Cause the only thing that I would see, I don't believe in benevolence when it comes to these companies, and I don't think we should.
Like, i- the only thing would be is if their products get worse, then it's like, okay, we'll, we will pay to make the products better.
So we see evidence, and obviously we have good information about who's blocking- Yeah... and therefore which, which models, and therefore where is there gonna be an information gap, right? And so the rest of the...
It'll take the rest of the world longer to see where there's, where the models start to have issues. But we see evidence of issues both.
We can see it in the models, we can tell it from the conversations, and we see how the interaction works between large publishers interacting with large model companies.
Interestingly, like, and I know you know this, that's not the only place where the market is. There's a long tail on both sides. There's a long tail of content, and there's a long tail of models. Yeah.
And so that is, like, in many ways that's where a lot of the diversity and innovation is gonna come, right? If this only...
If the only thing we solve is we solve this for really big publishers and really big model companies, that will be better than the current situation, but that's actually doesn't get you to high quality diverse content on the internet.
But the first- No, the opposite actually. [laughs] So the first- Yeah... place you'll see it is with large publishers, large models, but there is real benefit across the ecosystem.
And interestingly, there's the value of the content, which is valuable, but there's also, like, how do I make sure I get the information quickly? How do I make sure that I get it efficiently?
Meaning, do I have to crawl the same site over and over and over again, or can I tell that the content is fresh?
Like, there's, in many ways, the current situation of a bunch of machines crawling an internet that was meant for humans doesn't make a lot of sense. There's a much more efficient way to do this.
And so one of the benefits of us being a technology company is there's a lot of technical solutions to, to making all the crawling more efficient, which has tons of value for- Right. Yeah...
these big tech companies that spend lots and lots of money on CPUs, GPUs, and capital expenditures and everything. Yeah, they love spending on compute. Um- It's another line item. Let's go. Raise another- It's-...
four trillion. Yes, but it, it's a business decision ultimately. What, what is the, what is the hold up on the business decision? 'Cause this is all technologic. It, it seems like we do a- amazing things with technology.
This doesn't seem like a technology problem per se, or not a... I- it's a solvable problem, right? So- It's a b- it's a business issue. Yeah, so some of it is scarcity, meaning in the place, like you have...
You talked about GPUs. We can talk about CPUs. We could talk about talent. In those places there is scarcity today. Mm-hmm. Right?
Like, some of that, like, the history tells you that the cost of compute will continue to go down. Yeah.
The cost of talent, now that everyone realizes that this is a place to make a lot of money, there'll be more talent in those areas.
And so they're spending lots and lots of money on those two things, but you would argue that the cost for that will go down.
And so there will be room for, for content, but on top of that you have to have scarcity in order for them to just... Like, they have to have a reason to do it.
Uh, you know, I think we've been quoted many times as saying that no one wants to be the sucker. No one wants to be the, the only person that's paying.
And so we're right now two-ish months in from kind of the original, the big announcement around what we're doing, and so that's time to demonstrate that it's working.
You know, one of the, the questions of, like, is there demand? I know you know there's demand to be paid, but you want a stat- Yeah... that tells you there is? There's been a billion 402.
There are a billion 402 requests a day on the Cloudflare network. And for those wondering what a 402 request is, that is a site saying, "Please pay me." So instead of saying- Oh, yeah.
[laughs] So instead of saying, "Just block," it's, it's saying, instead of it just saying a block, it says, "Actually, what I really want you to do is to pay me," and we recently- How many, how many okays came back?
[laughs] The, I mean, the... You need a market in order for that, and it's gonna take a little while. Okay, so let's talk about the market, because it's like, okay, you gotta, uh, you know, Neil's two step.
It's like you got to establish the scarcity. You have to block and get some kind of leverage.
Let's assume that these models, there are signs that they're, they're sort of reaching the limits, right, of, you know, the, the GPT-5 was, was broadly seen as a disappointment.
I can see having unique access to fresh, high quality data sources when you're making a consumer product, that, I would think that would be good at least in marketing and, and probably on the product level, right?
But then you have to figure out what is the bid, what is the ask. And my experience with publishers is they always view, you know, their, like, journalism or, or, you know...
They don't like it being called content, and then it's even worse when it's called data. 'Cause then when, once you're, once it becomes, like, ones and zeros, like, you're completely commoditized.
And so I'm always, like, wondering what, what is the marketplace gonna be like for this, and whether it makes up for what is lost. Yeah, like, uh, right.
So I think those are two different questions, but let's start with the, like, what could the, what could the market look like? Like, how could this actually work? And I think, you know, world tends to think in analogies.
Yeah. Let's talk about Spotify- Right... for a second, right? And so I think there are...
Everyone could have their issues with the, with the Spotify business model, but there is lots and lots of money going to creators in the Spotify business model.
Took a little while for that market to develop, but to us that's a reasonably good...
It's not exactly a perfect analogy, but it's a reasonably good analogy for, for what happens, which is there's a whole host of content.
In this case, instead of multiple subscribers paying for it, individual models are paying for it, and then you figure out how you're gonna distribute that, that amount of money to the creators on inside a certain- Mm-hmm...
ecosystem. So that to us doesn't feel like a crazy model and feels like a way-For the large content creators, they can do their own version of that negotiation, and they should.
But for kinda the long tail, any one of them individually is not going to have the ability to have that level of negotiation. And so this is a rational way to do it, and there are a bunch of data, data...
There's a lot of data that shows that's encouraging high-quality original content.
The, the second thing is that if you look at Netflix and YouTube, to your point, those models, they are competing for the best content, right? They...
You want the, the best, newest thing so that people sign up for your system. And I do think that that is an interesting world for particularly for the AI companies that are more consumer-oriented.
And so interestingly, a s- like local press, and we've had lots and lots of issues with local press over the last couple of decades. It, it's a business model that's had lots of problems.
But in many ways, they do have very unique access to information that a cohort of people in that geography really care about.
And so this, this idea that you can create content that's specific and high quality that people want, and that an AI company will ultimately have a pretty good view on their need for it and willing to pay for it, doesn't feel crazy to us.
Mm-hmm. And then how that negotiation happens, there's a Spotify version where there's kind of one negotiation and then it's programmatically distributed.
There's an auction version of how this all develops, which maybe feels more like the ad networks, and then there's one-to-one negotiations. I think we have no idea how this will develop, for sure.
For now, it feels like we're gonna test out as a ecosystem all of those models and see which ones work. Okay, so it's not... Like, so that is still, like, up in the air, right?
Because I think one of the questions ends up being is, like, how do you value... It's very qualitative. Like, I mean, how do you value a particular piece of content?
Like, just saying like, okay, y- you know, when you break it down into ones and zeros, everything's the same, right? And it's, like, access to... There's a lot of different types of publishers, and the...
And, you know, words are not all equal, right? And some are, are more valuable than others. And the question is, how do you arrive at what the value is?
You can imagine a world where the AI companies themselves have a view on that. Meaning, they have a view for their own business and their own- Yeah... model, what that, what that information is worth.
By the way, is it worth more immediately than it is worth five minutes? Like, there's certain content that has real temporal value, right? Sure.
So if you have access to it immediately versus five to ten minutes later, that fundamentally changes the value of that content.
And so you can imagine a world where you don't have to have a central place that makes that determination. You have...
Every model has their own view, and that gets inputted into how they think about what they bid and what they pay, which to us feels like a rational system to develop. Yeah.
I mean, because, like, they ha- You know, it's like if you look at the programmatic world, you know, it's always been this...
You know, the, the buy side has insights into how valuable a particular person or cookie is to them, and so therefore, they're gonna bid, you know, specifically to that.
I mean, the AI companies, I would think, like, ha- have unique insights into the value to them for particular types of content, when, recency, how fresh they are, et cetera.
It might be different than, than what the, the, the sort of publisher. I mean, I'm sure the publisher will say, "No, it's more valuable," [chuckles] but...
Most markets, having spent a lot of my career in markets- Yeah... I mean, financial markets, you know, most markets, like, there's always that... There'll b- always be that tension between the buyer and seller.
Today, the tension is greater because there isn't, in most markets, there isn't scarcity, right? Yeah.
So in other markets where there's tension between buyer and seller, it's an opinion, but the reality is the option is you just don't buy it.
Unfortunately, today in a lot of places, the answer is you'll just get it from somewhere else, right? That's like the problem we have today is that you...
there is an alternative to paying the price that, in this case, the seller wants to sell it to you at.
Okay, so the pay per crawl, this, this initial imple- implementation is not necessarily the only implementation or the only way to create, like, a marketplace. Absolutely not. The- Right...
the idea with pay per crawl is that, one, we, we wanted to learn, meaning as a ecosystem, to learn what the right market development was. Mm-hmm.
What we set up was very simply, if you're, if you have content, you can, instead of saying block, you can say, "I would like to charge you." You do that through a 402 request.
It's today a private request, which means that the world doesn't see what you're asking for, just that one crawler will see what you're asking for. They can say yes or no. If they say yes, they get to crawl.
If they say no, they're blocked. So today, simple and not particularly programmatic, meaning, like, it's, like, ki- it's a human-to-human, one-to-one thing. Yeah, and it's blunt. It's like, it's yes- It's totally...
no, it's not. Okay. But...
And the idea behind that was if you add a lot of complexity into a market that has not yet been developed, you can just have lots and lots of fun with the complexity, but then you haven't really learned what the market demands.
Yeah. And- I mean, search started with a fairly basic formula. Like, it's like click-through rate, you know, times, like, what was bi-- I mean, it's like, became much more sophisticated over time. Right. For sure.
And, and then I think you can decide how much complexity do you want on marketplace versus how much complexity happens off marketplace and just turns into a reasonably simple back and forth between, you know, two, two parties.
And so we're, we would say, exceptionally early in the development of this.Okay, so if you, if you think of the, the demand side in this- Yeah...
as being the AI companies, and then the supply side are the, the publishers. You, you start... I mean, it's in beta, right? Like, it's in testing. Yep. So it's like it's clo- it's private, right?
Like, it's not like anyone can, can go into this. Like, it's, you know... I, I saw a lot of publishers I know there. They're, they, they're big businesses. Yeah. The...
We launched in private beta because that's the right way to launch a new product, which means you can request to be a part of it, whether you're on the supply side or the demand side.
And so, you know, we do a bunch of things to, to make sure it makes sense, and then we, we put you on the system with the expectation that you will...
that it's not a set it and forget it, meaning that you're gonna have to actively manage it today because it's not overly programmatic today. And the- How many publishers now are, are participants?
The, the answer is I don't know the answer to your question. Okay. I know, I know how many pu- I don't know the answer to your question.
Um, we have been very careful about it doesn't make sense to send bad deals through our marketplace- Yeah... if that makes sense. And so we're...
most of it today is still, even if it's gonna run through the marketplace, is bilateral- Yeah... conversations. This is a different... I mean, how are you gonna make money off this?
Don't tell me you're not gonna make money. [laughs] That would, that would worry me. The... A couple of things. One- [laughs]...
Cloudflare is, like, in the business of having people on our network, and us, you know, doing content delivery network- Yeah...
and us doing bot management, and so all of those are good- Websites pay you, app- Like, they pay you to provide these services to them. This is a different model.
Yes, but to be clear, that is part, like, meaning in order to be on the system, you're running your, you're running your traffic- Yeah, your customers...
through, through, through Cloudflare, which by the way, we, both sides are customers kind of unrelated to, to this product, but, but also creates a better experience for people on our network. So that's one.
But it's, like, a non-irrelevant part of, of Cloudflare. The second thing is, you know, we'll work through what the right economics are over time.
You should think about it as a marketplace in terms of us, like, s- charging a fee for the facilitation of- Yeah... transactions on marketplace. We don't...
To be clear, we will have plenty of publishers and content creators who ask us to do the blocking, so to do the enforcement, but we'll, we'll do deals, like separate, large, privately negotiated deals that won't run through the marketplace.
Okay. I mean, would it be like Stripe, how Stripe acts? The... I think the right way to think about it is more Spotify, like those types of...
Spotify's a better analogy in terms of how to think about a marketplace or any other place where you're getting paid a marketplace fee for creating a marketplace, doing the facilitation. Yeah.
The Stripe part is only the payments, and so yes, we will handle the, the payment side of that, but that's not the- Yeah, I think I meant with the Stripe in that, like, the, you know, the, the merchant controls the relationship with, you know, the person buying, right?
Yeah, the merchant and the- Like, so I'm just wondering- Yes. Yeah. Like, I just wanted to make sure because, like, a lot of...
You know, like, Ronald Reagan had a thing, like, you know, when, when he said the seven scariest words are, like, "I'm from the government, I'm here to help."
Like, I feel like publishers might have that, but it was just with technology. [laughs] And, like, 'cause it's like, uh-oh, they're gonna get between us and, and maybe these are the next, like, customers.
And I'm just wondering how that, like, would work, whether it's, like, y- the, the sort of AI companies are doing business with you, and then you are, like, passing along, or if it's, like, the directly, the publisher's working directly or, or it's just a, a, a variety of models.
I think the answer is yes. Okay. Meaning all of it. Maybe the answer is all of it.
And so we, we, we expect, see, believe that it makes sense for large publishers with large model companies to do deals directly, and those don't...
None of that passes through a marketplace, but many of those publishers will use us to enforce those deals, meaning they will make sure that the people they have deals with get through, and that they're using their data, meaning they're scraping, they're using the right crawlers in the right way.
That's one. And that they will block everyone else they don't have a deal with, which is a very important part of making this all work. But doesn't mean that payments run through in any way, shape, or form.
Cloudflare, that is Cloudflare really behaving in the traditional Cloudflare way in terms of what we do every single day, which is letting good traffic through, blocking traffic they don't want. Yeah.
So- No, I, no, I was gonna say, 'cause I wanted to get to Google for a minute, 'cause they're, they're very unique- Yeah...
in this, in that, like, it seems like there's an asterisk with Google, because, like, there's one crawl, the way I understand it. Like, you can't turn off Google.
It's like the m- the person who's, who's, you know, they could, like, be banging on the drum of, like, "We must stand up," but it's like, well, Google's a little different because, you know, they're- We were very careful with what we did on July 1st to treat everyone exactly the same.
So, so Google has one bot, but they have Google Extended. Google Extended does both training for Gemini and grounding for Gemini. And so we and anyone else- What's gr- I'm sorry, what's grounding?
Grounding basically is indexing. Okay. And so the organization of the, the information. Think about it in some ways as, like, search for, for Gemini. Mm-hmm.
So what we did was we block anyone who has a training-only crawler in our defaults. We block them at the network level. Mm. For Google, we inject a robots.txt message which instructs Google Extended not to train.
We have every evidence on the Cloudflare network is that they obey that instruction.
We have no evidence that they do anything other than if you block, if you tell them to not train, therefore basically doing the equivalent of blocking Google Extended, they, they do what you have asked them to do.
Which means in our default block, we treat them exactly the same way as we treat everyone else.
Like, interestingly-You know, Perplexity, which there's been a bunch of back and forth, we don't, in our default block, Perplexity doesn't train, so they're not part of the default block. Okay.
Then when an individual site owner is trying to figure out what to do with each bot, you're absolutely right that if they want to be in Google Search, they, they also are in AI Overviews.
So they don't have the ability to say, "No Search," but yes, "Yes, Search, no AI Overviews." Yeah. And that's where the complexity is in terms of, like, figuring out how to deal.
With Google, we've been very vocal around our request for all bots that they tell us what IP addresses they're announcing from, so they verify who they are, they cryptographically verify so that we can confirm they are who they say they are when they are crawling, and that they're very clear on their intent, and they listen to their instructions, or the instructions of kind of the network.
And on being very clear on their intent means that separating things out between search, between things like RAG, AI Overviews, training, and so that's the thing that hasn't been changed yet, meaning some bots already do that, meaning if you look at how OpenAI runs, like, things are happening with a different crawler, so you could easily pick one versus the other.
Google, you're stuck with, other than Gemini training, you're stuck with everything else if you want Search. Mm-hmm. What...
Give me the optimistic take for, for what, what happens next, 'cause, I mean, I, I think when we talk about...
I almost think that there needs to be like, they talked about that like Mar-a-Lago accords, like with current...
Like, there needs to be some kind of new economic bargain, and to me it starts with Google because that they're the, the center of, of, of the internet. Maybe, maybe I'm wrong. Yes. But what, what is it...
What, what happens, like, that, that, that changes this dynamic?
Is it, like, Google moving and saying, "Look, we under- like, we're, we're going to cut a new bargain," because we see, they, they see what is happening, and I know they, they gen up some studies that, that it's like telling [laughs] people that it's not raining.
It's like, it's raining. Like, every single publisher is dealing with this. You're not going to convince them otherwise.
Like, what, what actually, like, I don't know, what's a tangible thing that will, that will happen that will show that there is progress being made?
The single most important tangible progress that will be made is that they split out their crawlers.
And that doesn't mean they actually have to have separate crawlers, but they will allow you to be in search and indexing, but they will allow you to opt out of things like AI Overviews.
And that one, that creates a level playing field across the various AI companies- Mm-hmm... and gives publishers the contr- the control that they need to be able to say, "Search was the grand bargain of the internet.
Like, that works for me. You get to scrape my site, and you send me back traffic."
But for things where you're producing a derivative, where the likelihood of someone coming back to my site is meaningfully less, that's a different bargain, and so I can choose to block you.
Also, by the way, we can negotiate a deal, whether that's directly or programmatically, where you just pay me for that. But once you get to that world being different, the, a business model can develop. Yeah.
So in, like, five years in this optimistic take, 'cause I, I talked with Neil about this, I'm like, is, is, is the end goal that your, that your revenue line for licensing is far bigger in five years?
Maybe your advertising line is smaller, but that maybe you're more of a wholesaler than a retailer in some ways.
And I'm meaning, like, you're wholesaling content for training to, you know, and you're not, you're not having that direct rela- You're, you're not as, as reliant on, say, programmatic advertising, on all the affiliate and all, you know.
Y- yes, you're gonna be doing that, but that your revenue mix is gonna change. Like, is, is the, is the fundamental economics of the internet shift away from advertising then in some ways?
Because if attention is finite, and if, if people are increasingly going for this information, and they're getting it from this, these centralized sources and all the AI apps that are gonna be built off this, and I don't know if publishers are gonna do that, then it seems like the, the, the fundamental nature of these businesses will change.
Yeah. Uh, projecting five years in the future is petrifying- All right, great... but, [laughs] the, um, you know, it's, it's- I can't get any publisher...
I asked them, I'm like, "What does your business look like in three years?" Nobody has taken me up on the offer to, [laughs] to give it like a hazard, a guess.
I, I do think that, you know, subscriptions, o- one, you've seen a bunch of content creators with subscription models have a reasonably attractive business model.
So you talked about ads, but you didn't say subscriptions, which we think is relevant. Two, today the AI companies actually themselves have reasonable subscription models.
So if you imagine a world where you're paying AI companies X, and so they earn X per monthly active user, and there's some amount- Yeah...
of that that's going out to the content creators based on monthly active users, there is a little bit of this world of you win, I win, like you're growing monthly active users, and that's going to the AI company.
So if you wanna call that, I mean, that's, you can call that licensing if you, if you would like. And so it feels like that is a, that is a world that, that could exist.
Whether people will figure out advertising, you know, we will, we will see in terms of what, what that, what that looks like.
The idea that we could get to a world that has less clickbait and, you know, is something that's, you know, better, higher quality from a content perspective maybe means- Yeah...
that you have some advertising but not as much. But I think it's hard to predict exactly what it looks like. I think it's clear that the current, the status quo is unsustainable.
Every content creator, like, ceasing to exist does not work for the world.A few companies deciding what content gets created sounds scary, and so figuring out a model that's more about, like, what is the populace and asking for feels like- Yeah...
where we're trying to head, and that would mean, you know, publishers have a business model- Yeah... that, that grows with the number of people who wanna read or wanna learn from the things that you've created. Yeah.
I mean, that is the goal, I guess. Like, I mean, 'cause ultimately, I mean, I guess- Yeah...
the sort of, like, the biggest, like, optimistic take is, you know, I feel like publishers, a lot of times, and, and, you know, the incentives have been set to do lots of things that are outside of the core mission that will sell subscriptions.
Like, everyone has to chin up all kinds of different...
You know, it's like I used to make fun of all the Game of Thrones recaps, and you had to do everything for these algorithms, and it was distasteful to some degree [laughs] depending on your taste level, I guess.
But it was like, eh, you gotta do what you gotta do.
And, you know, I think the, the ideal would be that those kind of, like, hacks, you know, become less integral to a lot of these models, and yeah, there's more of a strong subscription element to, to these businesses.
I think the incentives of the, the ad-supported internet a lot of times, you know, got, got publishers going in, in directions that were not great, to be honest with you. Cool.
Well, thank you so much for, uh, taking time and, and, and walking through this. I don't know. Any final thoughts, like, for publishers as, as they contemplate their future in an AI world? One, thank you for having me.
It's fun to, to connect. We can do it in person next time across. Yeah. [laughs] The walkway. No, just that we know that the things are changing rapidly, that lots is going on, but there...
But one, like, you can have transparency, you can have agency, you can have control, and it will take a while for the business model to develop, but, but it starts with making sure you know what's going on on your site, and you, you take control over who's, who's on it.
We're happy to help. Okay. Cool. Stephanie, thank you so much. Really appreciate you, uh, taking the time. Awesome. Good to see you. [outro music]