It’s easy to wonder how leaders, even the good ones, sleep at night. The decisions that come across their desks affect so many lives, jobs, healthcare, human rights. All of these decisions impact large sums of people and their ability to sustain their personal Maslow’s hierarchy of needs. It’s hard to imagine how they can reconcile these decisions with themselves at the end of the day.
However, once you view people as numbers on paper, making decisions that affect large groups gets much easier. However, when you do that, it’s easy to lose perspective and underestimate the impact individuals and smaller groups can have on the whole. At Stack Overflow, things affecting large groups of our users require a lot of thought and reflection. We try to make the best decisions that will benefit all the diverse segments that make up our audience. Sometimes we do this well and the community prospers, making folks happy and better equipped to use our product effectively. Sometimes we don’t do this well and it’s up to us to determine that as quickly as possible, improve our position, and iterate.
When I first started on the Community team last year, there was a decision that needed to be made around how much time we spend on our Meta platforms. I heard feedback from a significant number of Stack Overflow employees that they avoided on Meta because of the reaction it garnered. Though this was not felt by all staff who interacted on Meta, and it affected different people to varying degrees, many felt discouraged or experienced full on anxiety at the thought of making a Meta post. Also, the data we were operating from cited a small number of engaged users on the platform (200-300). It seemed, at the time, that Meta was causing more pain than adding value.
I made the tough call to stop asking employees to make announcements there and to pull back from the platform a bit. There were a bunch of people internally that thought this was a bad decision, and that we were abandoning some of our most valued community members. It’s a discussion that has continued nearly daily over the past six months. In hindsight, it would have been preferable to reach consensus through more research earlier, given that many people in the company (and community) care deeply about these issues, it was a discussion that was front of mind.
Challenging assumptions
Several people across different teams felt that those numbers didn't match their experience with meta and community interactions. Yaakov Ellis, an engineer on the Community Product team, took the initiative to do a deeper dive into the data and discovered that we were operating under false assumptions after all. Digging into it, while there are about 200-300 Meta users that are responsible for most of the posts, the vast majority of our curators (the folks that edit and flag posts and perform reviews) are avid readers of our Meta sites.
The results of our “Through the Loop” survey also indicated that stepping away from our Meta sites had negatively affected the community. To get an understanding of how our respondents would like to see us improve, we coded open-ended responses (see: this blog post for more information on that process) to the following question: “What do you find most frustrating or unappealing about using Stack Overflow?” 3,202 survey respondents answered the open-ended question, we randomly selected 350 of those responses to code.
In these results, we found data that was different from our monthly Site Satisfaction Survey on Stack Overflow. On an average month, the Site Satisfaction Survey sees about 3% of responses that fall in the “other” category, which is why our encoding standards are usually dependable. In this case, however, there was clearly data that wasn’t parsed by our usual standards.
When we looked into the data, we found most of it (85% of all “other” responses) fell into one of three groups: negative feedback about Stack Overflow company/leadership, concerns around issues happening in the community, and lastly, general community concerns. There was a lot of concern around support and how we communicate with our community. We believe that this survey, as it was presented to the whole network instead of just Stack Overflow, showed heavier participation from Meta users. While our Site Satisfaction Survey represents a more wide selection of user types (by getting a random sampling of anonymous and logged in users on Stack Overflow), the power users that participated in the “Through the Loop” survey are an important input to our decision making process.
Moving Forward
All of these things together made it obvious it was time to revisit my decision around how we interact with our Meta sites.
When public figures (the good ones) make the wrong call, it’s admirable when they admit it publicly and share what they are doing to right the wrongs. In response to these learnings, we posted our rededication to supporting our users on Meta. We also released a new set of internal guidelines aimed at supporting staff who will be interacting with the Community on Meta. While the negative issues that we have encountered in the past have not completely gone away, we are not letting these deter us, and are optimistic that a combination of open communication and adherence to the Code of Conduct can ultimately keep this manageable. We’ll be checking in with you to share how this is going and what we learn from it. We’re not stepping away from our work to make the site more welcoming, or making improvements to Stack Overflow. What we’ve learned is that in order to make it to a more welcoming community it’s important we support and have the support of the people that interact with us on Meta.
I’m grateful that people followed their guts here and challenged our assumptions. If they hadn’t, we might still be in the same place, or making other decisions that go against the best interests of our users. I’m personally looking forward to getting to know more of the users that frequent our Meta sites as we spend more time communicating with them. Recently, I’ve spent time talking to them over chat and in person. It’s been a great experience learning from them and hearing more about their day to day. I’m looking forward to doing that much more.
Please leave all comments and feedback on this post on MSE.