Loading…

The Loop #2: Understanding Site Satisfaction, Summer 2019

We’re excited to share research highlights about the work we’ve been doing to understand how satisfied people are with Stack Overflow. We’ve been working hard to explore what users like best about Stack Overflow and what their top pain points are, with the goal of improving the overall experience of using the site. To this end, we’ve launched a site satisfaction survey, in which we continually survey users about their experiences using Stack Overflow.

Article hero image

Welcome to CHAPTER #2 of The Loop, a blog series from the Stack Overflow Community team. This post is a deep-dive into how satisfied users are with Stack Overflow.

We’re excited to share research highlights about the work we’ve been doing to understand how satisfied people are with Stack Overflow. We’ve been working hard to explore what users like best about Stack Overflow and what their top pain points are, with the goal of improving the overall experience of using the site. To this end, we’ve launched a site satisfaction survey, in which we continually survey users about their experiences using Stack Overflow.

We want to make sure that our users feel heard, and this site satisfaction survey is one of the many things we’re doing to gather feedback from our users so that we can incorporate this feedback into our product development and product roadmap decisions. In this post, we’ll be sharing more about this survey and the specific insights we gained from it.

Survey Background

With the site satisfaction survey, we wanted to gain insight into the pain points users encounter when using the site. Since July 2019, we have been using a monthly site satisfaction survey to ask a random sample of logged-in and anonymous users to report their satisfaction and provide us with open-ended feedback. This gives us a new wealth of insight into our users and also supplements other forms of feedback loops with users like user testing and 1:1 interviews.

In this survey, we asked users to answer the following question: “What do you find most frustrating or unappealing about using Stack Overflow? ” We allowed users to write in their own answers so that we could gather rich and nuanced data in their own words. We manually coded 2,942 of these open-ended responses and organized responses into different themes so that we could better understand users’ frustrations with using the site. We labeled the open-ended responses in the survey with up to three codes each. Note that the findings in this post are from users who completed the survey in July, August, or the first half of September, 2019.

Main Findings

The graph below depicts the breakdown of the top themes from the open-ended survey responses:

Unwelcoming community 10.6%, 
Design 9.8%, 
Artifact quality 9.7%,
Barrier to participation 8.3%, 
Discovery 8.0%, 
Overmoderation 7.1%, 
Voting 5.1%, 
Question quality 4.2%, 
Timely answers 3.5%, 
Other 3.2%, 
Comments 2.2%, 
Onboarding 2.1%, 
Social friction 1.8%, 
Subjective content 0.8%, 
Mobile app/site 0.6%, 
Welcoming backlash 0.5%, 
Job quality 0.4%, 
Review queues 0.3%

The top three themes were:

1. Unwelcoming community (10.6% of responses): A perception of an unwelcoming community was the top thing that people found most frustrating or unappealing about Stack Overflow. We categorized responses that mentioned condescending or rude replies, and general comments about toxicity and lack of friendliness issues into this theme.

“The toxic nature of the community… Scares people from even signing up let alone asking questions”

“Some people are often condescending or rude”

2. Design (9.8% of responses): The next most common thing that people cited as an issue with Stack Overflow was design. Responses that discussed the user interface being confusing or cluttered, requests for a dark theme, and issues with the information architecture between Stack Overflow and Stack Exchange were categorized into this theme.

“Cluttered UI, confusing UI (e.g., I don’t know what to do when I’m on the page)”

“It seems like it has a messy layout with a lot of unnecessary text before the answer to a question”

3. Artifact quality (9.7% of responses): Another common theme we saw in the data was people citing issues with the quality of the answers on the site. Responses which mentioned outdated answers, wrong answers being accepted, answers not being accepted at all, or lack of answers were categorized into this theme.

“Sometimes the answers assume prior knowledge which I do not possess”

“When users mark questions as solved without giving the answer”

How these research findings tie into our product roadmap

This data is one of the key inputs that we use to inform our product roadmap.

Your feedback about unwelcoming community and perceptions of overmoderation led us to prioritize two important user-to-user feedback loops: the question closing experience and review queues. We know that getting a question closed or sharing and receiving feedback about a question can be frustrating and confusing to question askers, question closers, and those working hard to curate site content. That’s why we’re changing the question closing experience so that questions that need improvements are hidden more quickly and question authors are given specific guidance about how their questions can be improved. We’re also revamping the review queue experience so that it’s easier and more efficient to use.

Next, we heard your concerns about design loud and clear. Our design approach is focused on improved usability and accessibility through reduced clutter, clearer patterns, and integrating best practices into our site design. We’re prioritizing optimizations to the question page as well as reduced complexity of the reputation and privileges system. We’re also continuing work with the Stacks team to continue rolling out user-friendly design patterns.

Finally, we are taking steps to improve question and answer quality. The feedback that has historically been accessible to us frequently pointed to concerns with newly-asked questions, whereas the site satisfaction survey shared frustrations with the existing library of 18 million questions. We know both are important to everyone – which is why we’ve already revamped the question asking experience – but the site satisfaction survey feedback has helped us prioritize work focused on the library of questions. We want Stack Overflow to remain a source of high quality question and answer pairs, so we’ll soon be starting to explore ways to continue to keep the library of questions fresh and make it a trustworthy part of all developers’ toolkits.

We look forward to sharing additional updates about this work with you as time goes on, and we’re grateful to all the people who’ve generously shared their experiences and feedback with us.

✋Opt-in to our Research email list (must have a Stack Overflow account). You’ll receive invitations to participate in surveys, user interviews, and more. You’ll receive up to a few emails per year and can opt-out at any time.

📖Read CHAPTER #1 of The Loop, where Donna takes a deep-dive into our research approach: what it used to be, what it is now, and how it continues to evolve.

Login with your stackoverflow.com account to take part in the discussion.