September 20th marks the third year anniversary of Natty, which is a chat based robot that detects non-answers posted as answers. Built by a team called SOBotics, which consists of members from the Stack Overflow community, Natty has helped detect more than 150k non-answers, all of which can be seen on the dashboard, Sentinel. In February 2017, after talks with the Stack Exchange staff, Natty started to flag answers automatically once it had detected a particular answer as a non-answer with considerable accuracy. As of now, there have been almost 17,000 automatic flags raised that have been marked as helpful (and five other flags that have been declined).
Natty is just one of the many bots that SOBotics operates to watch Stack Overflow around the clock. SOBotics started in May 2016 to utilize the Stack Exchange API in order to help reduce the size of the close vote queue on Stack Overflow. The close votes review queue is a moderation tool that allows users with 3,000 reputation to go through the questions that already have a close vote on them and decide to either close the post or leave it open. The problem with the close vote queue is that you cannot filter it based on the number of close votes a question already had. That information was readily available in the API. Keeping this in mind, a group of three users started working on a bot designed to query the API and retrieve the questions that were in the close vote queue in a format that supports filtering the content. That bot was called SOCVFinder. SOCVFinder helped reduce the number of ignored questions in the close vote queue and decreased the size of the queue significantly.
While building SOCVFinder, the SOBotics team was scanning all the comments to pick up information about questions that could be duplicates. However, they discovered many of the comments that were being picked up were rude, snarky, or otherwise not in line with the then “be nice” policy of Stack Overflow. They wanted to create a bot to detect rude comments using machine learning. SOBotics requested a dump of rude comments to seed their ML bot and started working on Heat Detector, a bot designed to detect rude comments. This bot brought moderator attention to rude users, and resulted in the creation of a new automatic flag whenever a particular user was posting too many rude comments.
But their work to help ease the moderation burden didn’t stop there. Along with SOCVFinder, SOBotics also came up with several other bots:
- FireAlarm, which helps detect low quality questions on creation using machine learning. This helped reduce the number of posts that were going to the closed vote queue by aiding users to close the questions quickly.
- Badger tracks problematic badges like Cleanup, which are indicative of certain issues with the post on which it was awarded.
- Burnaki monitors burnination efforts (removing problematic tags from questions) on Stack Overflow.
- Rodgort tracks the ever growing list of tag related discussions and has had a significant impact in the ongoing tag monitoring efforts on Stack Overflow.
Apart from bots to help users, SOBotics has built many bots to help moderators spot harmful user behaviors on Stack Overflow. This includes Guttenberg, which detects plagiarism on Stack Overflow. Numerous moderators have utilized Guttenberg to help catch serial plagiarists and ensure the reputation system remains fair and reflective of actual contributions. Similarly, Spot Detector detects bad reviews in the low quality posts review queue and has helped aid moderators to catch folks who consistently post bad reviews. After talks with some of the moderators, SOBotics built Belisarius to detect vandalism of post content.
The bots they made have not only helped the developers at SOBotics to moderate Stack Overflow, but also in real life. The knowledge gained by building these robots has helped to expose them to new technologies. With experience in these new technologies under their belts, some of the SOBotics developers were able to apply that to their careers.
Stack Overflow has been instrumental in helping the developers at SOBotics achieve their goals. From catering to their requests for data dumps, to providing relevant permissions to automatically flag use of the API, the Stack Overflow staff has been really helpful. In recognition of all their contributions to the community, Stack Overflow provided them with a complimentary version of Stack Overflow for Teams to help the developers of SOBotics share their knowledge with each other.
The future has more in store. There have been efforts to build a tracker for sites other than Stack Overflow called Boson, a bot to help detect sock puppetry and other voting fraud, as well as bots to detect the creation and re-creation of bad tags. If you want to help in the development of these bots, you can contribute to the efforts for different SOBotics projects on GitHub. If you are interested in reviewing bot reports or just helping to lighten the mood, you can join their chatroom. There is a bot to sing you a song, too! Drop in to the chatroom if you are interested to know more about SOBotics.