Back in June, AWS announced BugBust: a worldwide competition for Java and Python developers to find and fix bugs. Teams could scan their code on any supported repository, even if it didn’t reside on AWS. Those repos would be scanned by Amazon CodeGuru, an AI-powered developer tool that identifies bugs. Trained on decades of knowledge and experience, CodeGuru Reviewer uses machine learning and automated reasoning, AWS and security best practices, and hard-learned lessons across millions of code reviews on thousands of open-source and Amazon repositories to automate code reviews. The contest awarded points to team members who successfully fix the offending code.
The 10 contestants with the most points nabbed the grand prize: an all-expenses-paid trip to AWS re:Invent 2021—which was IRL in Las Vegas from November 29 to December 3, 2021.
But at AWS re:Invent, AWS wanted to go bigger. They created a public challenge that ran during the conference where anyone—at the conference or at home—could participate in the world’s largest bug-fixing challenge. Instead of participants scanning their own repos, CodeGuru scanned multiple open-source projects so developers could start fixing bugs immediately. The final results not only cleaned up some OSS repos, but set a Guinness World Record in the process.
Across both contests, over 3,100 developers from around the globe participated, squashing more than 33,000 bugs.
Improving open-source projects
In the original challenge, participants brought repos from their own organizations to be scanned for bugs. Because this new AWS BugBust public challenge took place during a specific time period—the AWS re:Invent conference—AWS wanted participants to be able to start fixing bugs immediately. So they selected a variety of open-source repos across multiple categories and scanned those for existing bugs.
Ensuring the quality of open-source projects is critical because undetected bugs can create performance issues in any project that implements that open-source code. But it’s also unpaid work, meaning it’s hard to find volunteer contributors with the time and energy to fix code in open-source projects. The means bugs can go undetected for years, causing issues across a wide range of projects. A large-scale effort to find and fix bugs in open-source repos had clear benefits for anyone using those open-source repos.
AWS cloned and scanned multiple projects in a variety of categories, from Big Data and Blockchain to Data Science and Deep Learning.
Naturally, AWS scanned for AWS CloudFormation errors and violations of AWS best practices. However, other best practices were enforced for both Java and Python—the two languages that CodeGuru supports. CodeGuru looked for a wide range of fixable bugs: regular expression errors, code clones, unvalidated inputs, and known security issues. It was also able to spot resource leaks and concurrency issues, two of the hardest bugs to investigate.
These bugs are the kinds of errors that developers can unknowingly introduce in the course of their day-to-day work. Introducing bugs is easy, but tracking down their root causes can be hard. Some of the bugs even found issues that went against the official documentation. One team found a race condition with the Java `ConcurrentHashMap` type; the documentation said it was thread-safe, but if two threads picked up the process at the same time, the values of instantiated `ConcurrentHashMap` objects could be overwritten.
Competitor Juv Chan learned from two bugs through the AWS BugBust re:Invent Challenge, both security related. The first involved key derivation and password hashing using the Argon2 algorithm. Chan knew about password hashing, but not with the fairly new Argon2 algorithm. The second came from invoking a shell comment with `subprocess.Popen([cmd], shell=True)`, which could risk unwanted privilege escalation in the shell. Instead, he used the `shlex.split()` and `shlex.quote()` commands to avoid invoking a shell at all.
Without Amazon CodeGuru, many of these bugs may have continued to go undetected, creating performance problems in any project using these open-source repos. To find and fix this number of bugs without CodeGuru would have required volunteer developers to spend a significant amount of time combing through the code. Since open-source projects struggle to find contributors as it is, stealth-mode bugs could have lived on almost indefinitely.
It wasn’t just bugs that were fixed; sometimes it was the developers’ processes. “I learned how to be able to make use of GitHub to create pull requests in a faster and more efficient way,” said Chan.
Getting into the Guinness Book of World Records
As part of their AWS re:Invent conference, AWS wanted to attempt to set the Guinness World Record for the world’s largest bug-fixing challenge. Over 600 developers, both in person at the conference and virtually around the world, started working to fix bugs and set a world record. Every developer who participated got an exclusive certificate from AWS and Guinness to commemorate their participation in the world record attempt, even if they just fixed one bug.
Over the course of the event, those 600+ developers successfully set the world record for the largest code-fixing challenge. Their commitment to commits squashed over 30,000 bugs in the course of just a few days.
The world record wasn’t the only competition happening; individual BugBusters also competed for a cash prize of $1,500 USD. Each bug squashed earned the fixer a number of points, depending on how severe the bug was: 1 point, 3 points, or 5 points for the heavyweights. By the end of the competition, danielsoft was the winner with 686 bugs fixed for a total of 2,162 points. Interestingly, he didn’t fix the most bugs; he won by prioritizing the highest-value bugs.
If you’re interested in hosting an AWS BugBust event inside your organization or want to take CodeGuru for a spin, head over to their website and try it out for 30 days free of charge.