In this episode of Leaders of Code, Stack Overflow’s Janice Manningham and Josh Zhang sit down with Cloudflare VP Will Allen to discuss the innovative pay-per-crawl model co-launched by their organizations. They explore how the rise of AI has disrupted the traditional “open versus block” internet model, creating a need for platforms to protect their content and data from commercial exploitation while maintaining community access.The discussion also:
- Explores the future of the bot ecosystem, emphasizing the importance of putting publishers back in the driver’s seat to decide how their content is accessed and monetized.
- Explains the technical implementation of the pay-per-crawl system, which uses Cloudflare’s bot categorization and WAF rules to serve a 402 “Payment Required” message to specific crawlers.
- Highlights the strategic value of data licensing, comparing comprehensive enterprise contracts with the more flexible, programmatic pay-per-use access enabled by the new model.
Notes
- Connect with Will Allen, Janice Manningham and Josh Zhang on LinkedIn.
- Learn more about Stack Overflow Data Licensing here.
TRANSCRIPT
Eira May:
Hello and welcome to another episode of Leaders of Code on the Stack Overflow Podcast. If this is your first time listening, this is a segment where we get senior engineering leaders in the same room, same virtual room anyway, and we let them ask each other questions about the work they're doing, how they build teams, and the biggest challenges they're facing right now. My name is Eira May. I am the B2B editor at Stack Overflow, and today I'm joined by two of my colleagues at Stack, strategic product leader, Janice Manningham and site reliability engineer, Josh Zhang. How are you guys doing today?
Janice Manningham:
Hey, Eira.
Josh Zhang:
Good.
Janice Manningham:
Nice to see you.
Eira May:
Thanks for joining us. We also, in addition to Janice and Josh, we have a guest today, that's Will Allen. He's a VP at Cloudflare. Will, welcome to the show.
Will Allen:
Thanks so much for having me. Great to be here.
Eira May:
Great to have you. So we wanted to get you in the virtual room today because we wanted to talk about a new Pay Per Crawl model that Cloudflare and Stack Overflow are co-launching. So this is a model that is an effort to monetize public data and get bots to pay for crawls. So I'll turn it over from there to Janice and let her tell you all a little bit about this project.
Janice Manningham:
Thanks, Eira. So let's talk about Pay Per Crawl. First, I want to emphasize that it's not to be confused with what others might have heard, which is paper crawl, as in a piece of paper and crawl. It is in a more correct form, paying for each type of crawl or bot who is attempting to access a site. Why we are talking Pay Per Crawl is because with the rise of AI crawlers, they've fundamentally broken what I believe is the old internet, which is open versus block models.
Traditionally, we've had a lot of content platforms like Stack Overflow operate in a way where we have open access generally for bots to be able to take advantage of our public content. We block for any malicious activity, but with the rise of AI products looking to take data from model's training, we have found ourselves in a position in the last year or so to revisit that approach and look at ways for us to protect our data against commercial usage for model training, but also still allowing access to our community. That's why companies like Stack and Cloudflare have started to talk openly about this pay access to our content and other content providers as well.
I want to actually bring in Josh to explain a little bit about historically what Stack has done with our bot traffic and the work that he has done to manage this set of AI crawlers up to date and before we really worked closely with Will and Cloudflare.
Josh Zhang:
Sure. So historically, before the advent of AI, a lot of bots would actually just try to bring the website down, right? So one of the reasons we switched to Cloudflare was for their DDoS mitigation. And I basically specialized in protecting the site from bots that brought the site down. But with the advent of AI, bots evolved because now there's money in scraping and sending as much traffic as you can to a website, but masking it as normal traffic. So on one hand, they're not taking your site down, but they are just sending you a ton of extra traffic that you ultimately have to pay for that isn't bringing any value. In fact, they could take away your value because if they're taking the data but not kind of sending traffic back to your site, it's not a good virtuous cycle, right? So with bots getting more and more sophisticated, it's getting really complicated to try to basically weed out the good traffic from the bad.
In fact, very recently, the bots are getting so good, they're fooling advertisers. They're basically eating up ad impressions because it used to be they would scrape using a headless ... Just like curl or very simple programmatic ways to access a site, and that wouldn't trigger any kind of ad impression, which is at least something. But now they're using headless browsers and other methods to basically completely fool you and we might be able to catch them, but ultimately they're eating up ad impressions, which is also really terrible back to the advertisers themselves. So this is really ever going arms race against basically bots that are just trying to extract as much information from you as possible while basically trying to pretend to be legitimate traffic.
Janice Manningham:
Yeah. Just like to feed off of that as well. I think we were kind of playing accountant masking for a while, don't you think, Josh? Where we were doing more of an active block approach where ad hoc we identify some malicious activity from specific user agents or bots and then we just add them to a block list. And then we continued doing that approach, and that block list started to get a little bit unwieldy.
Josh Zhang:
Definitely hard to scale that approach. We're basically just play whack-a-mole, right? And there are other tools you can use like fingerprinting and Cloudflare's bot score and things like that. But of course it's an adversarial relationship so the people writing the bots know what they have to defeat. So you're basically just, it's an ongoing cycle of trying to catch the next technology wave and things like that. And unfortunately, in a lot of cases, there's more of them than there are of us.
Janice Manningham:
Totally. And so then I think we looked into using Cloudflare and their existing tools and infrastructures to kind of support us in this new situation and give us the robust tools needed to manage that. So I wanted to talk to you, Will, about the product that you present with Cloudflare and how you've helped Stack Overflow with that managing bots and allowing us to control traffic.
Will Allen:
Amazing. Thanks again for having me and thanks for being just a phenomenal partner. I mean, we sort of start from a radically simple philosophy, which is that you, as an incredible business and partner, you should be in the driver's seat over what happens to the content and sort of how that content's being accessed on the sites that you control. And that's not saying bots are good or bad or crawlers are good or bad or AI is good or bad, it's just saying that you should get to decide, right? And there are real costs associated with serving this large amounts of traffic.
And you really do have to think about what's the sustainable business model for the future when you have the old model of all of my content would be scanned and indexed and I'd get a lot of referral traffic back over, and I could either monetize it directly or indirectly or at least get attribution. That model is broken down and continues to break down even further. So there need to be new models that are out there. And so our goal here is just to support folks like you to give you the tools to say, understand the identity of the bots and the agents that are coming, but then even more importantly, take different action based upon who that is.
Janice Manningham:
Yeah, that's exactly what we found really useful about the Cloudflare's tools. I really like the sort of bot categories and the registration process that allow us to, I guess, systematize how we look at bots and therefore the framework for how we manage those. To your point, each company has different ways to manage and think about how to work with those bots. And so just starting at a very foundational level of being able to help categorize them and then allowing the companies to do what they want with those was really helpful. I know Josh has also identified for us the different types of bot categories and we've been able to distinguish between the ones that we want to continue to support to access our site. And then the ones that we think we want to put on a different bucket so that we can assess what to do with them.
I don't know, Josh, if you want to say it more to that.
Josh Zhang:
Yeah. With so many different kinds of bots, some of them are always going to still be useful and you don't want to block them, right? Search engine crawlers historically, that's a lifeblood of any website on the internet. So if you stop them then you're asking for a lot of trouble unless you're a very specific kind of website. So being easily able to categorize them has been huge from Cloudflare. If our team had to manage all the different crawlers, registration, IP addresses and things like that, our team would have to be so much bigger. But luckily with the tools that we're given, we're able to basically say, "Okay, this is identify as a business, this is what we want, this is what we don't know about, this is what we don't want." And basically easily just pick those buckets from Cloudflare and just quickly produce rules that more or less enforce this.
Of course, there's still going to be new bots that we have to do research on and basically like block that are very much at the forefront, but that's part of the ongoing cycle, right? And eventually those bots will get identified and added specifically to the bucket, and that can hopefully help everybody on the Cloudflare network because those lists will keep getting updated.
Will Allen:
And that's one of our goals, I would say, is that to have that sort of bidirectional communication. And then just the fact that we see so like the volume of traffic that we see is so massive across so many different sectors and industries, it allows us to do that, right, so that you don't have to staff up entire teams or think about that day in. We want to really take that burden of understanding who are these agents that are coming, who are these bots, who are these crawlers? And then let you, again, be in the driver's seat to make that decision about, do you want to allow them, do you want to disallow them, do you want to rate limit them or do you want to charge them? But even just the categorization to understand like, "Wait, who's accessing this? What are they accessing and what are they doing with this content once it's been accessed?" We just spent a lot of time trying to make that as easy as possible for you.
Janice Manningham:
Yeah. It's almost funny, it's as if you've had internal views of how we were manually managing this before. We had our own Excel spreadsheet that had color coded categories and status and all that. And so when we started moving over to Cloudflare's tooling, it was almost like this intuitive, read our minds in terms of what we needed. So that was really great.
We started working with Cloudflare. We set up our management framework and had scalable ways to manage bot traffic that we talked about. And then we started looking into the Pay Per Crawl beta product that Cloudflare launched recently. So as part of our bot traffic monitoring, we use that against how we look at our data licensing opportunities as well. So Stock Overflow has been licensing our data and have it available for a model training with Frontier AI Labs, but we were seeing other intentions with bots maybe seemingly trying to access our data for similar reasons. And we thought that there was an opportunity there as well.
If we're not seeing interest from the human in the loop having these more formal licensed contract discussions, but yet we're seeing those bot traffic where there's increased requests to hit our sites, why not meet the interest and the demand where they are and see if there's ways to capture their interest through this Pay Per Crawl motion. So when Cloudflare and when you approached us about this product, it was really enticing for us because we were seeing so much interest from our view with our bot traffic and I thought that there was a great sort of opportunity to test our assumptions about whether there was any good, meaningful monetization opportunities with that.
So with the existing framework that you said that you allowed us to have with the bot categories, it was really seamless to be able to basically turn this on and essentially give these bots and new sort of instructions on what to do with this paywall that we were giving them. Josh, I would love for you to speak more about how we implemented that from a technical perspective.
Josh Zhang:
Yeah, sure. So when we were enrolled in Cloudflare's Pay Per Crawl Program, it was actually pretty simple. It was just a UI that basically wrapped some existing rules into their WAF, and they actually created a bunch of dashboarding. That's obviously the bigger piece that made my life a bit easier, right? So there were already pre-populated list of bots that basically were targeted and you can choose what to do, either allow the traffic through or basically charge them and you set a charge rate. But ultimately the dashboarding allowed you to kind of see once you flip on the switch what's happening with the traffic in a very holistic way. Interestingly, when we turned on Pay Per Crawl and we started serving, I think as a 402, some of the traffic from those bots that used to just get a block of 403, they stopped sending traffic our way. So it's almost like they got the message. So that was actually really simple to turn on, and it was really good because we could basically very quickly implement and test our theories without basically investing too much time into it.
Janice Manningham:
Yeah. That was one of the biggest value adds for us, being able to quickly turn on essentially a switch in our CloudFlare instance for this 402 message to be sent to bots and see if we get any activity. We were able to work very closely with you, Will, to talk through some of these edge cases and use cases around where we can expect productivity. And I think it sends a pretty good message in this sort of like body ghost here about what we're open for in terms of business and monetization. And I'm hoping that more and more these 402s will help drive conversion, and the humans behind these bots will provide payment information for us to start managing this.
And then a different aspect of this as well that's different from our bigger licensing deals is it helps support a different type of access usage, more of a pay per usage plus the word Pay Per Crawl. Typically, with our license deals, it's a much more comprehensive contract that includes the bulk majority of our data set. But with paper crawl, what we thought was also enticing potentially is the ability to just scrape what you need and the bots have the ability to control that, obviously with proper payment. And so I thought that ... I really liked that differentiation and that different pricing model to see if there's different ways we can hit the market.
Yeah. And would love to hear, Will, how you've seen this sort of workout with other organizations to date.
Will Allen:
I mean, what I love about the way you all are pushing this forward is that you're just an ideal customer for us. And really how you think about this rhymes with other people that we speak with in that you have recognized for a long time, again, that business model, the internet's changing and you're experimenting with multiple models, right? And so different ways to have this sort of be successful for you. And it all starts, I think, from that sort of message that you send to the crawler or the agent, and you got either let them come straight through, get the content, you can sort of block them out right, or you can send this HTTP 402 message, which just says payment required. And what I love about that is that it's not a no, it's a yes if, right? You are welcome to come get this if there's some sort of payment that happens in here.
And that payment can happen directly, programmatically, machine to machine. So it's sort of built that way. It could also happen that just people start to, on the receiving end of these 402 messages, look at their logs and they're like, "I just got to call them," right? "I got to call Janice and Josh to sort of strike a direct deal." And that's a win too, right? You've got incredible BD teams and executive teams who are thinking about this, and if they can get these sort of direct calls as well, and we see that with other customers of ours where maybe they do some level of machine transactions, but even more so, it's giving them the tools that they need to strike these deals across the board, right, to say that like, "Hey, we've got incredible content, we want to work together, we want to make this thing happen."
And so this payment required message is really, "Just give me a call." And that's how we think about this, is that we want to support all of these various mechanisms and all of these different ways for you to sort of, again, enforce your preferences for your content and build the business that you want to build, and it's going to take different shapes. And so that's we love this sort of experimentation and love this sort of collaboration because it allows us to build these great products and really to serve folks like yourself.
Janice Manningham:
Yeah, that's exactly right. That's exactly what we are also excited about in terms of opening this up. And it's not just expanding beyond data licensing deals, but also looking at other use cases. We've seen some interesting activity from companies who typically we feel like are not engaging in the AI arms race, but maybe finding our data useful in other ways. And so Josh and I have been curious and excited to see if there's ways to do business with other types of businesses where they still find our data to be valuable, but we're offering in a way that is, to your point, palatable and accessible on their terms versus having to go through these much larger unwieldy deals with our procurement and business teams.
And then I think just moving forward, we've been talking a little bit about how Pay Per Crawl works today. And moving forward, I think we talked a little bit about how orgs can use Pay Per Crawl. As long as I think bots are registered through Cloudflare, and we talked about the categorization earlier, it allows them to sort of be brought into this 402 HTTP code and then that gives them that message to say, "Hey, let's do business, let's talk," and then hopefully we can go from there.
Will Allen:
And we've got even more, I would say more coming on that front. And so there are different ways to think about sort of these programmatic payments. And so the way it's working today is that there is some sort of registration for that so we can make sure that you're only charging that exact specific crawler. But one of the reasons we're supporting new payment protocols like the X402 work that we're doing is to allow ... In some cases you might not care about who is operating the crawler or the agent on the other end as long as there's some sort of payment that's happening. And that allows us to work and support those use cases as well. So again, all of this sort of comes back to that same philosophy of you all should be in control. How do you make it as simple as possible to do business with you to get access to your content?
We've got one way that works today, give you all of the tools, all of the analytics that you have, and there's more coming on that front as well to make it even easier to make those payments.
Janice Manningham:
Yep. And I think more broadly, the message that we want to get across, too, is that we have a rich corpus, 15 years plus of high value content, question answer site that focused on helping developers get unstuck. And we want to make that data available, but for the right use cases and for the right access controls. And so this is just adding onto our existing sort of message around our new era, a new space with properly licensing our data.
Will Allen:
Thanks for being an incredible partner. And I love how pioneering you all are in this and to think about this. And I know so many folks that I have worked with and continue to work with have used Stack as sort of a foundation of how they learn and how they think. And I just love the innovation that you all are bringing to sort of these new business models of the internet. And so thanks for having so much fun and building these great products together.
Janice Manningham:
Yeah, totally. The partnership has been great and we really love working together with the same sort of shared goals and interests. And Cloudflare has been wonderful with that, so thank you so much, Will.
Eira May:
All right. That is going to do it for this episode. If you have suggestions for topics to cover or guests that you would love to hear from, you can email podcast@stackoverflow.com. My name is Eira May and I'm the B2B editor at Stack Overflow. Today I was joined by Janice Manningham and Josh Zhang, my colleagues at Stack Overflow, as well as Will Allen, who's the VP at Cloudflare. Do you guys want to say goodbye and let folks know where they can find you?
Will Allen:
Yeah, I'll go from my side. Will Allen, you can find me. Go to cloudflare.com, check out a lot of our great things. It's actually, we've got an incredible amount of services for free. So if you just got a domain and you want to test something out, kick the tires, I highly recommend it. All of my personal projects have been there for years and years. And then you can find me on LinkedIn and on Twitter with @WilliamAllen. So thanks again for having me.
Janice Manningham:
Yep. And you can find more about Stack Overflows, data licensing, and content available for commercial usage on stackoverflow.co, and I'm also available on LinkedIn.
Josh Zhang:
Yep. And I am also on LinkedIn, Josh Zhang, SRE. And I've also written a couple of blog posts on Stack Overflow blog, so I'm on there too.
Eira May:
Thanks everyone for listening, and we'll see you on the next episode of Leaders of Code.