TLDR; We’re going to be sharing our product development process with you, from feedback loops to timelines. We’ll be doing so through our new series - The Loop. You can give us your thoughts on what you’d like to see us do by filling out this survey: Through the Loop[closed for responses 1/25/20]. We’ll also be releasing Moderator Training and some new feedback mechanisms to help us form decisions as we grow.
Since the early days of Stack Overflow, our community has seen a lot of growth and change. As we reflect on 2019 and start thinking about 2020, our company also continues to rapidly grow. We have new leadership, amazing partners using our Stack Overflow for Teams product, and north of 100k new users signing up to the public Q&A each month (coders are everywhere!). Growth on the business side of our company enables us to do more for our community, like relaunching the Stack Overflow podcast, creating a newsletter for developers by developers, and launching new features like authentication with GitHub, which is exciting for everyone. With so much change and growth, we have been reflecting on how we can improve our communication paths to and from our community.
We’ve observed that systems that used to work when we had 50,000 users don’t work as well now when we have 50 million coders asking questions and teaching others on our platform.
Software developers often talk about technical debt, the little bugs and shortcuts that you don’t have time to fix, and which over time, can slow, hamper, or completely cripple the system in which they reside. Just like a system can accrue technical debt, as a 10-year-old company, we have accrued community debt. For a community, debt can take many forms: Long-time users can be surprised by things changing out of the blue for reasons they don’t understand, moderators don’t feel supported by tooling that hasn’t been updated in years, and too many users, despite their best efforts, see their questions closed, or are met with derisive comments challenging their knowledge and skills. We have been making little changes to try to address these things (e.g. we published our current Code of Conduct in August 2018), but we have come to the realization that we need to do more in order to sustain, serve and support a much larger community and keep it growing and thriving.
We want to address all these things and be transparent about how we’re paying down some of this community debt. We’d like to share with you some of the reasons why we make decisions and what inputs we listen to, as well as give you a place to weigh in. We are striving to:
- create an experience that works for all users
- show everyone how we think about serving the larger developer and technical community
- be clear about the "why" behind our decisions
A month ago we formed cross-functional teams of Stackers (employees of Stack Overflow) to create strategies to start addressing some of these concerns. It’s been inspiring to see people from our Community Management team work hand in hand with folks from Engineering, Sales, and Marketing to come up with solutions for our community’s most pressing concerns. We’ve worked together to build new communication frameworks that take into account how we’ve scaled and to replace old frameworks that don’t work now that we’re larger.
The themes these groups took on are:
- Better mechanisms for community feedback
- Building a moderator advisory group
- How we communicate and interact with Meta sites moving forward
For each theme, we recruited cross-functional teams, agreed on the problems we needed to solve, discussed solutions, and came up with timelines. You can read about each below.
Better Mechanisms for Community Feedback
The problem
Over the 10 years that Stack Overflow has been around, the way that we collect user feedback has changed significantly. If you’ve been with us for a long time, you may remember when our research process involved regular, direct exchanges between users and staff on Meta. Today, this process has been largely replaced by 1:1 user interviews, as well as other methods like surveys and contextual research. This shift happened quietly, and many Meta users felt understandably concerned that we weren’t listening to users or making data-driven product decisions.
The solution
We’ll share regular updates about what we learn through our research, as well as create a new working group of users that we’ll lean on for regular feedback. This working group will be made up of a diverse group of folks excited to see Stack Overflow grow.
Existing Feedback Mechanisms
We already use comprehensive approaches to gather signals about how our users are doing, from quantitative analysis to user interviews to our annual Developer Survey. Another feedback mechanism for us (new this year), which we haven’t shared publicly yet, is our Site Satisfaction Survey. Our site satisfaction survey fields responses from a representative sample of users on Stack Overflow, both logged in and anonymous. We ask people how satisfied they are with the site and what we can be doing better. This has resulted in great data that, along with other feedback mechanisms, have been informing our product development process.
Overall, both anonymous and registered users are highly satisfied with Stack Overflow and tell us that their favorite things about our community include finding solutions to their problems, vast access to information, and the knowledgeable people who participate.
We’ve learned a lot from you, and we work to distill your feedback into themes. One theme we examine is what you find most frustrating about using Stack Overflow. Themes that you said frustrated you over the past few months were:
- an unwelcoming community
- site design (cluttered pages, confusing navigation across the Stack Exchange network, etc.)
- artifact quality (outdated answers, poorly framed questions, etc.)
- barriers to participation
These responses help to inform our product decisions and allow us to ensure we’re serving all types of users. Our commitment to you, moving forward, is to continue to share reports on what we’re learning here publicly on the blog. We want to give insight into the community we have, and the data behind this kind of feedback is fascinating.
For those that haven’t gotten to participate in a site satisfaction survey, we’ve put together a survey we’re calling Through the Loop [closed for responses 1/25/20], so we can hear what’s on your mind about Stack Overflow. Tell us about the things you’d like to see us doing better; we’ll share these results publicly with you.
New Feedback Mechanisms
With our new mixed method research approach, one thing we lost was regular, in-depth conversation with a group of folks highly invested in Stack Overflow’s growth. We also wanted to keep seeking out feedback from a broad range of perspectives.
That’s why we’re creating a working group of users made up of people from all corners of the developer community — from folks new to programming, those who don’t participate in Stack Overflow but are passionate about programming, experienced Stack Overflow users, frequent contributors, and more. We’ll hand-select folks of diverse backgrounds who are excited to chat with us regularly about everything from new ideas to features, to how we communicate with the broader Stack Overflow community.
Timeline
By the end of this year, we will have completed discovery for the working group of users and will have identified the people we would like to recruit. We’re looking forward to hearing from representatives from different types of users and backgrounds starting in 2020 and regularly thereafter.
Creating a Moderator Advisory Group
The problem
When we started this site we didn’t need moderators, as we had Joel, Jeff, Jarrod, and Geoff. Now, we have over 550 moderators who volunteer to help by taking a leadership role to make Stack Overflow and Stack Exchange places where everyone can contribute. This has created a lot of community debt as the moderation challenges have grown over time. Some of the challenges we identified were:
- A growing, broadening moderator team
- Aging moderation guidance
- Difficulty scaling support for moderators from our Community Management team
The solution
To address the challenges we identified, we’re putting together a moderator advisory team drawn from our 550 existing moderators (folks who volunteer their time and donate their knowledge and leadership to the community): a small, self-replacing council of moderators who will be tasked with keeping our moderation guidance and methods up-to-date, along with ad-hoc working groups of moderators to work on specific initiatives as needed. This advisory team will be a two-way channel for contact with community managers (who are Stackers/company employees) and other teams within the company. This council’s focus will be on how we moderate and the challenges that arise as we do. Community managers will look to moderator advisors in matters that impact the moderator community, with an eye toward making sites as self-governing as possible. Our moderators are the experts here, and we want to work with them to foster a welcoming community (our community Code of Conduct provides the ground rules for that) and meet community challenges as a team.
But wait, there’s more! Moderating communities is a skill, and one that's too often poorly documented, poorly understood, and pursued without robust best practices. Teaching moderator skills through folklore and informal mentorship results in uneven practices at best, and perpetuates problems at its worst. Robust training for moderators is as essential as ensuring that anyone else on our team is getting the proper, ongoing education and training towards world-class skills.
We’re going to create a new system to offer training for our moderators. The training will be a great way for new moderators to learn the ropes, and will be available to all moderators seeking help as they struggle to make important decisions, learn the tools available to them, and plan how to move forward on sensitive issues involving race, gender, and sexuality. We’ll work closely with the moderator advisory team to create the curriculum and resources.
Timeline
In order to implement the moderator advisory council, we will be hard at work to finalize its scope and vision and in the coming weeks share it with our moderators. We want our moderators to play an early role in shaping it. By the end of Q1 2020, we will establish our first council and begin working together.
Strategy for How We Communicate with Meta
When conversations about Stack Overflow started to happen on Stack Overflow, our founders invented a site called Meta. Meta Stack Overflow was created as a way to talk about the website without distracting us from the important things: questions about programming. Since then, it’s become almost a catch-all for everything: bug reports, general complaints, feature requests, and ideas about the site. With thousands of accumulated bug reports and feature requests, it’s a lot of community debt.
The problem
We have identified these fundamental challenges with Meta as we’ve grown and accrued community debt:
1. It’s hard to capture structured feedback on Meta. There are now so many conversations that we aren’t often able to participate. As a result, users end up not feeling heard and a lot of confusion (including some misinformation) is generated.
2. On Meta, there are discussions, some that go on for a long time without a clear answer.
3. Meta tends to exclude people that aren’t super immersed in the Stack Overflow/Stack Exchange culture.
4. Meta requests don't integrate with any existing ticketing system, so our Community Managers need to prioritize the best they can and answer the threads deemed most important at the time.
The solution
When this group started discussing solutions, we agree that we need to double down on figuring out what the next iteration of Meta looks like. We analyzed data on how Meta is being used, who is using it, and all the functions that Meta serves. We then refined the functions into five areas:
- Self-governance discussions
- Support
- Bug Reports
- User Feedback
- Announcements
We looked at the three data points—how Meta is being used, who is using Meta, and the functions—and determined if our users are best served by keeping the function on Meta or if our users are best served by moving the functions to other tools and processes.
Timeline
Finding the right ways to serve each of these functions is not a small task. In early December, we are putting a strategy together for the tools and systems we’ll need. Bug reports, in particular, have been a significant pain point internally and externally! We plan to transition things like bug reports, user and customer support, user feedback, and company announcements off of Meta over the course of next year.
In mid-December, we’ll share high-level transition plans about how Stack Overflow will interact on Meta sites going forward.
In Q1 2020, we’re going to start the transition process. We are following a product development and release process for each function that we transition. It’s a robust process that involves discovery, feedback loops with stakeholders and community members, developing documentation and FAQs, and external communication plans. We are committed to getting these transitioned in 2020. We’re excited to transition some of these functions off Meta, giving them the attention and focus they deserve.
Closing the Loop
In machine learning, there is a concept called “Human in the Loop.” Some processes can’t just be done by machine alone; instead, a human adds value to the feedback loop. The solution to community debt, like the debt that we as Stack Overflow have accrued, is humans in the loop.
“This concept leverages both human and machine intelligence to create machine learning models. In this approach, humans are directly involved in training, tuning and testing data for a particular ML algorithm. The intention being, to use a trained crowd or general human population to correct inaccuracies in machine predictions thereby increasing accuracy, which results in higher quality of results.” - Hackernoon
In December, we’ll be kicking off a new series we’re calling “The Loop”. We’re asking you, the Stack Overflow community, to be our Humans in the Loop as we pay down our community debt. Through your help and feedback, we’ll have the best opportunity to build a better future together. Through our existing feedback mechanisms, along with our Through the Loop survey, we can plan to build the things that really matter to you and build healthy ecosystems. Posts in this series will share our research and our product planning based on what we learn from these different mechanisms.
Summary
- We’re committing to improve our relationship with all users by showing you what we’re doing consistently.
- We’re building a moderator council to work with when issues arise in the community. We plan to have a council in place by the end of Q1 2020.
- We’ve revisited the best use of our Meta Sites and will communicate the transition rollout to the community in Q1 2020.
- We’re compiling a small diverse group, including new users, power users, and moderators, to be a constant partner for feedback by Q1 2020.
- We’ll continue sharing user research with you here so you can see behind the curtain about WHY things are being built with our new series, The Loop, beginning in December.
- This is our initial step, your voice matters here, and we want to hear it. We’re asking you to grow with us and to be a part of the discussion. The right direction is the one we can feel good about together. Tell us what you think by responding to our survey, Through The Loop [closed for responses 1/25/20].
- We want to work with you to move past focusing on what’s wrong in order to strengthen the things that are right and build the future we want together.
Stack Overflow, since the beginning, has been a partnership with the software community. We want to make sure that the tools we have to interface with our users scale with us. We will need your continued partnership, as we learn new behaviors together and find the best way to provide feedback loops for the 50 million people that visit Stack Overflow. We know that during times of change, communication is important; you’ll be hearing from us the entire way.