Iterating on Inclusion

Last year, we wrote about our important work to build a more welcoming and inclusive community. This summer, I wrote about how we are evolving our product to encourage better interactions and guidance. While there’s still a lot more to do, we’ve begun to take concrete steps toward a better coding community for everyone. This is the most important work we’re doing in our community, and our first wave of improvements are starting to show results. 

Here’s one concrete example: For some time, we’ve been using machine learning to flag comments that might be potentially troublesome in the community, especially if they could be seen as hostile to new users. (Our tool for evaluating and flagging those comments, called the Unfriendly Comment Classifier, is pretty great! We’ve got details on how it works here.) After we tuned the flagging tool to ensure it was reliably detecting troublesome comments, we used it to measure the impact of the new policy and product changes we’ve been making to improve community health.

Those changes, along with thousands and thousands of you working to make the Stack Overflow community more friendly, have cut the number of troublesome comments in half. Our moderators are the front lines of this, and they’ve helped to ensure this place is more welcoming for everyone. We appreciate all their hard work, and we’ve seen a reduction in the odds of a new comment being detected as abusive go from 2.6% of new comments down to 1.3% in the past 18 months. All our analysis indicates this is a real decrease in unfriendliness in our community, not just an artifact from how the model is trained or other effects.

Now, we’re not declaring victory here. There’s still a lot of work to do—but we want to acknowledge the incredible work that the community, our moderators, and our product teams have been able to accomplish together to make meaningful improvements.

We’ve also been learning a lot through this process, such as clarifying policy, setting explicit expectations, and empowering the community to raise the bar. All of these changes make Stack Overflow more welcoming. And as you’d expect, that means we’re going to do more of it.

What’s Next

Following on this work, we’ve begun our next community initiative to make improvements specifically designed to make Stack Overflow more welcoming. This takes the form of another similar clarification and setting of expectations, this time around a different section of the Code of Conduct. Since a lot of our readers are developers, here’s a diff:

Our original text says:

Be kind and friendly.

Avoid sarcasm and be careful with jokes — tone is hard to decipher online. If a situation makes it hard to be friendly, stop participating and move on.

Our new tenet is:

Be inclusive and respectful.

Avoid sarcasm and be careful with jokes — tone is hard to decipher online. Prefer gender-neutral language when uncertain. If a situation makes it hard to be friendly, stop participating and move on.

This section:

No bigotry.

We don’t tolerate any language likely to offend or alienate people based on race, gender, sexual orientation, or religion — and those are just a few examples. When in doubt, just don’t.

is now:

No bigotry.

We don’t tolerate any language likely to offend or alienate people based on race, gender, sexual orientation, or religion — and those are just a few examples. Use stated pronouns (when known). When in doubt, don’t use language that might offend or alienate.

In software, we release, learn, and iterate to make our products robust and effective. Though people are not code, this is one place where the analogy holds: learning and improving as we go will help us reduce errors and makes it more likely information will get accurately transmitted. On a more human level, it’s a kind and respectful thing to do.

By adding this update, we want to make it clear that the Code of Conduct requires people to use the correct gender pronouns when someone shares their pronouns or makes them public. It also means that respecting anyone less because of their gender identity or pronouns is off limits. This has always been true of our Code of Conduct and we are making it more explicit with this language. This isn’t a new rule or a change to our policy. We found there was confusion, and we’ve clarified the language to make things abundantly clear. 

We recognize using different pronouns can sometimes be challenging for people who don’t speak English as their first language. There are lots of great resources online to help guide folks with questions, including this. We will also be sharing FAQs with our moderators and community, and are here to answer your questions. 

The goal here is an obvious one: we want to build a better culture, where community members don’t have to justify who they are or explain why their identity is worth respecting. Stack Overflow and the Stack Exchange network are here for everyone to be who they are.

Running the Code

Now that we’ve got an updated Code of Conduct, it’s time to put the code into production. This is a good chance to practice being mindful of people’s pronouns, and, as there are people who may be new to this kind of consideration, it’s also a great opportunity to support other community members learning to get this right.

The key thing is, this goes for all of us. Whether we’re regular users, moderators, or even employees at Stack Overflow, we all work together to make sure the community is welcoming. When we see violations of our Code of Conduct, we’ll work with users to ensure that they follow the same clear expectations as everyone else in the community. Though it’s very rare, it’s worth saying explicitly that in cases where the Code of Conduct is willfully violated by someone after we’ve addressed the issue to them before, that user will be removed from the community—even if they’re a moderator or a user with a lot of rep. It’s especially important that new or returning community members who are building (or rebuilding) trust in the community see that everyone—users, moderators, employees, everyone—is held to this standard.

That way, we all work together to look out for people who may need protection in online communities the most, like our transgender, non-binary, and gender non-conforming members.

Going Forward

We’re excited about the things we’ve released in the first phase of helping our community get to a place that is more welcoming and inclusive. How can we tell if we’re succeeding at this? Well, there are a few ways. Metrics like “number of comments flagged” and “users that come back to ask a second question after their first experience” help us. In Q2, we released a Site Satisfaction Survey that polls a sample of anonymous and logged in users on Stack Overflow. In the survey, we specifically asked what we could be doing better and report against how many people tell us they don’t feel included or welcome. This is what we’re watching closest. 

Over the next few months, you’re going to see a lot more of this from us. As part of this next phase of work, we’re going to work to fix features that make people feel like they’ve been publicly shamed, and reduce some of the social friction that is built into our product. Soon, there will be no more big yellow boxes that say “these five people voted your question is bad,” because nobody likes that feeling! Instead, there will be clear guidance on how to improve questions when they need more information or better formatting.

Think of the difference between how great it feels when you get a really good code review or pair programming session as compared to that terrible feeling if you make a code commit or pull request and someone slaps it down. Stack Overflow should feel as good as the intentions of those who help each other in the community.

We’re also working on providing more help and support to our moderators and plan to continue to improve the tools they use every day. Our team is developing clearer guidance around how to detect when someone is being excluded and what to do about it. Each of them should feel valued and accepted. These stewards of the community are the backbone of what we do and we greatly value the time they put in evaluating flags, questions, and comments in service of making this community a better place. We’re thankful to them for continuing to improve Stack Overflow, and especially thankful to our community for pushing Stack Overflow to be every bit as welcoming as it is informative.

Tags: announcements, bulletin, community, company