As you can see from our network configuration, HAProxy is a much beloved part of our infrastructure. Willy Tarreau, the author, has been extremely responsive and helpful to us in the past.
So when we reached a surprising dead-end in our quest to find a reverse proxy that could block HTTP clients using too much bandwidth, or too many connections, we were happy to approach Willy with the idea of sponsoring this feature in HAProxy.
I'm pleased to announce that this new HAProxy feature we sponsored is now available to everyone as of August 26th!
Geoff Dalgas and Jeff Atwood described to me in great details what they needed to do : perform request throttling per IP address, possibly based on various criteria, in order to limit risks of service abuse. That was very interesting, because that feature was being thought about for about 4 years without enough time to completely develop it … … The last words naturally go to the really cool guys at Stack Overflow. It's very nice to see some sites and companies involve time and money and take risks to make Open Source products better. Of course they benefit from this work, but at no point during the whole development did they try to reduce the focus to their specific needs, quite the opposite. From the very first exchanges, their goal clearly was to make the product better, and that must be outlined. That's now achieved and I really appreciate their involvement. Thank you guys!
If you'd like more details, Kyle Brandt, our sysadmin extraordinaire, documented the details of how this new HTTP connection and bandwidth limiting feature works over at the Server Fault Blog. Kyle also worked extensively with Willy to make sure everything went smoothly, and it's a credit to both of them, because it absolutely did. This big new feature worked more or less as advertised right out of the gate.
We hope to be able to sponsor more open source projects in this manner. Our specific goal is to "make the internet a better place to get expert answers to your questions", but I believe this is still secondary to our primary goal: make the internet better. And having a freely available open source reverse proxy that lets you run a site of our size (top 500 on the internet by some accounts) without being accidentally undermined by abusive or poorly written HTTP clients, is a win for not just us -- but everyone!