The Loop #2: Understanding Site Satisfaction, Summer 2019
Welcome to CHAPTER #2 of The Loop, a blog series from the Stack Overflow Community team. This post is a deep-dive into how satisfied users are with Stack Overflow.
We’re excited to share research highlights about the work we’ve been doing to understand how satisfied people are with Stack Overflow. We’ve been working hard to explore what users like best about Stack Overflow and what their top pain points are, with the goal of improving the overall experience of using the site. To this end, we’ve launched a site satisfaction survey, in which we continually survey users about their experiences using Stack Overflow.
We want to make sure that our users feel heard, and this site satisfaction survey is one of the many things we’re doing to gather feedback from our users so that we can incorporate this feedback into our product development and product roadmap decisions. In this post, we’ll be sharing more about this survey and the specific insights we gained from it.
Survey Background
With the site satisfaction survey, we wanted to gain insight into the pain points users encounter when using the site. Since July 2019, we have been using a monthly site satisfaction survey to ask a random sample of logged-in and anonymous users to report their satisfaction and provide us with open-ended feedback. This gives us a new wealth of insight into our users and also supplements other forms of feedback loops with users like user testing and 1:1 interviews.
In this survey, we asked users to answer the following question: “What do you find most frustrating or unappealing about using Stack Overflow? ” We allowed users to write in their own answers so that we could gather rich and nuanced data in their own words. We manually coded 2,942 of these open-ended responses and organized responses into different themes so that we could better understand users’ frustrations with using the site. We labeled the open-ended responses in the survey with up to three codes each. Note that the findings in this post are from users who completed the survey in July, August, or the first half of September, 2019.
Main Findings
The graph below depicts the breakdown of the top themes from the open-ended survey responses:
The top three themes were:
1. Unwelcoming community (10.6% of responses): A perception of an unwelcoming community was the top thing that people found most frustrating or unappealing about Stack Overflow. We categorized responses that mentioned condescending or rude replies, and general comments about toxicity and lack of friendliness issues into this theme.
“The toxic nature of the community… Scares people from even signing up let alone asking questions”
“Some people are often condescending or rude”
2. Design (9.8% of responses): The next most common thing that people cited as an issue with Stack Overflow was design. Responses that discussed the user interface being confusing or cluttered, requests for a dark theme, and issues with the information architecture between Stack Overflow and Stack Exchange were categorized into this theme.
“Cluttered UI, confusing UI (e.g., I don’t know what to do when I’m on the page)”
“It seems like it has a messy layout with a lot of unnecessary text before the answer to a question”
3. Artifact quality (9.7% of responses): Another common theme we saw in the data was people citing issues with the quality of the answers on the site. Responses which mentioned outdated answers, wrong answers being accepted, answers not being accepted at all, or lack of answers were categorized into this theme.
“Sometimes the answers assume prior knowledge which I do not possess”
“When users mark questions as solved without giving the answer”
How these research findings tie into our product roadmap
This data is one of the key inputs that we use to inform our product roadmap.
Your feedback about unwelcoming community and perceptions of overmoderation led us to prioritize two important user-to-user feedback loops: the question closing experience and review queues. We know that getting a question closed or sharing and receiving feedback about a question can be frustrating and confusing to question askers, question closers, and those working hard to curate site content. That’s why we’re changing the question closing experience so that questions that need improvements are hidden more quickly and question authors are given specific guidance about how their questions can be improved. We’re also revamping the review queue experience so that it’s easier and more efficient to use.
Next, we heard your concerns about design loud and clear. Our design approach is focused on improved usability and accessibility through reduced clutter, clearer patterns, and integrating best practices into our site design. We’re prioritizing optimizations to the question page as well as reduced complexity of the reputation and privileges system. We’re also continuing work with the Stacks team to continue rolling out user-friendly design patterns.
Finally, we are taking steps to improve question and answer quality. The feedback that has historically been accessible to us frequently pointed to concerns with newly-asked questions, whereas the site satisfaction survey shared frustrations with the existing library of 18 million questions. We know both are important to everyone – which is why we’ve already revamped the question asking experience – but the site satisfaction survey feedback has helped us prioritize work focused on the library of questions. We want Stack Overflow to remain a source of high quality question and answer pairs, so we’ll soon be starting to explore ways to continue to keep the library of questions fresh and make it a trustworthy part of all developers’ toolkits.
We look forward to sharing additional updates about this work with you as time goes on, and we’re grateful to all the people who’ve generously shared their experiences and feedback with us.
✋Opt-in to our Research email list (must have a Stack Overflow account). You’ll receive invitations to participate in surveys, user interviews, and more. You’ll receive up to a few emails per year and can opt-out at any time.
📖Read CHAPTER #1 of The Loop, where Donna takes a deep-dive into our research approach: what it used to be, what it is now, and how it continues to evolve.
39 Comments
While 2,942 of these open-ended responses could be enough for a reasonable sample, how many responses did you actually get to the survey?
Also, NLP and automated sentiment analysis would be useful against the complete data set even if you don’t code all of the open-ended responses.
One thing that would help with overmoderation and design would be including features such as customizable front-pages. Right now, the watched tags and ignored tags are simply highlighted or grayed out. This could be improved upon by allowing users to hide ignored tags from even appearing in their “feed” and moving highlighted tags to the top ( or something like that ).
By allowing users to customize their own feeds it could both improve overmoderation, design, discovery, and participation.
This already kinda exists, in multiple ways… depending on what you mean exactly by “front page”. The home page of the site, once you are logged in, is a custom tailored list built based on your previous interactions with the site. Tags/questions are weighted based on the previous questions you’ve visited/interacted with, thus giving you a tailored result set without you having to ignore tags or follow tags to do so. (It also shows a small number of questions that fall outside of that resultset mixed in so that you get to explore a little outside of your comfort zone)
Additionally (on the /questions page,) when you ignore a tag you have the option to just gray it out or hide it completely so users are already allowed “to hide ignored tags from even appearing in their feed.”
I personally use the filter included in Adblock Plus.
For example ###hot-network-questions blocks the “Hot Network Questions” (which are sometimes really distracting).
By the way, they added this as a profile preference.
Profile > Edit profile and settings > Site settings > Preferences > Sidebar > Check “Hide Hot Network Questions”
Could elaborate on the quality of the sampling data and the analysis itself, and the limitations?
Summer 2019… really? It was winter for half of the world.
From thist post : “Note that the findings in this post are from users who completed the survey in July, August, or the first half of September, 2019.”
From Wikipedia : “June, July, and August are the warmest months in the Northern Hemisphere while December, January, and February are the warmest months in the Southern Hemisphere.”
So I think it was Summer for tha majority of the users 🙂
The majority of SO users are male, but we don’t always want to use pronouns like “he/him”. Maybe a middle point, like “mid-2019” instead of catering to the majority?
y so picky?
Because Stack Overflow is seeking to foster a culture of inclusivity. Which is commendable.
But it believes that such an aspiration justifies its attempts to police language. Which is highly questionable.
What’s good for the goose is good for the gander. If Stack Overflow believes it to be acceptable (or even a positive measure) to police language, then to avoid hypocrisy, it must submit to having its own language policed.
Fixing posts with poor language is one thing. Bludgeoning people about their poor language is quite another. I’ve edited hundreds of horribly written posts, many from top-5-on-stack posters who would be voted in as mods if asked (like this one fella whose keyboard’s newline key is inop).
Hi Pam,
Using something variable like the four seasons of the temperate and sub-polar climate as a way of describing a time period is ignorant, confusing, and just plain bad.
There are much better ways to do this, eg. list the actual dates/months or say Q1, Q2, Q3, Q4.
Yes, this. If SE wants to do all this virtue signalling they should *at least* get the basics right themselves first. See also: loop #1 originally forgot that black people exist. Much of the company’s content is not accessible (no transcripts of audio presentations). Routinely makes things US-centric. Basic research. Again, wouldn’t be *such* a big deal if they weren’t spewing vitriol at everybody else on this topic.
Reflexively upending the long-standing policy that question upvotes get 5 rep and answer upvotes get 10 rep, because some a study of dubious quality had found that *women are more likely to ask questions than answer them* was my personal favourite. Of course the evidence of this reasoning was censored from the site and DMCA’d off the Wayback Machine, using the rationale “contains personally identifying information” (??), because that’s how open this company is now.
I’ve thought before that SE’s definition of “diverse” seems to be “looks like a crowd of randomly selected educated English-speaking New Yorkers”. They seem to have very little conception of linguistic or cultural diversity, or indeed religious diversity.
I can imagine that Review queues scores fairly low, because most people don’t care about them and don’t use them.
Some questions arise:
* “We manually coded 2,942 of these open-ended responses and organized responses into different themes so that we could better understand users’ frustrations with using the site.” As rjzii said: 2942 of how many?
* The percentages sum up to nearly 80%, what are some statements from the other 20%?
* Why is the graph highly misleading? Only 10% answered “unwelcoming” but the graph makes it look like nearly 80% gave that answer.
What in the world is the unnecessary text before the answer?! The Question? The pages literally have almost nothing else than the question and the answers in the center area. Even the comments are collapsed down!
true, haha
What I find frustrating is that some users complain about the community being “toxic” when in reality they put little effort into their questions and ask for good answers. Just take tag misuse for example; there are probably hundreds of questions that use tags incorrectly because the OP couldn’t be bothered to read the tag usage guidance.
Never mind the tags.
People ask questions that amount to little more than “It doesn’t work, what do I do?” without providing any information about what “it” is and what it should (but doesn’t) do and then complain because the question gets closed.
Yeah. I reduced my contrubution by about 90% because I’m sick of the questions. Almost all of them are questions without any effort and when you ask for a bit of effort (a snippet, maybe, as I’m main JavaScript), you don’t receive any answer.
Then you need to explain to them gently that SE has a quality standard (and that’s why it’s so good). The farthest I’ll push the “reverse psychology” is saying flat-out that it’s not everyone’s cuppa joe.
I don’t see how this applies to StackOverflow:
“When users mark questions as solved without giving the answer”
You can’t. You can accept an answer, but you can’t otherwise mark a question as solved.
Looks like the folks who answered your survey don’t know how the site works.
Only questions can be considered as “solved”, not answers. So a question is marked as solved when and only when an answer is marked.
That’s what the sentence “When users mark questions as solved” refers to without too much text 🙂
Then the quote makes no sense, which is what I said.
You can only mark a question “solved” by marking an answer, which means you have an answer to the question. so, the quoted text is BS. Every “solved” question has an answer, so a question can’t be solved without an answer.
Perhaps an answer was chosen that wasn’t even an answer to the question or wasn’t actually a good/useful answer.
You have an attempt to answer, often those answers don’t actually solve the problem. Sometimes user post something along the lines of “I solved it” and accept it as an answer, or someone gives a non constructive answer like “Yeah, you can do that”
You can mark a question as “duplicate”, which implies that it is solved. Close enough?
Haven’t you encountered question with no answers, that ends with
“**Update**
Nevermind. I’ve alreade solved it”
My problem is that when people select “unwelcoming community” I don’t think they mean it in the SJW way but in the way what you see in meme communities: their questions get closed easily. The way SO currently tries to handle this issue helps nothing it only alienates their already existing userbase.
I feel that SO has to decide what it wants to welcome, if you want to welcome new users regardless of the effort they put in their questions, you’re going to become “unwelcome” to the people who have put countless hours to provide excellent answers for the community.
They’ll simply perish by being unable to find questions worth answering, and it’s going to get to a point where the quantity contributors are going to dominate providing mediocre answers sometimes borderline incorrect, and then it will lose it’s purpose as Q&A repository.
Ahhhh… you’ve nailed it, Lt. JustSayin’.
Very compelling analysis.
Users said: “We don’t like our questions being shut down.”
Stack Overflow heard: “The SO Community is pretty transphobic.”
I think artifact quality would be even higher if you used the word answer. I had no idea what an artifact was, and in your summary you straight up say answer quality?
Questions are artifacts too. Improving answer quality often requires first improving question quality. In the case of questions put on hold, for example, this requires better communication about the next steps for both the asker and the community.
It is SO funny seeing the same, perfectionist S.O community criticize a survey showing proof of their perfectionism.
Only a small minority of professionals (even smaller if they’re experiencing frustrating problems) are willing to put in the effort to thoroughly explain their problems. Anyone who has worked in a retail or customer support job knows that frustrated, impatient people aren’t very good at asking questions and the worst possible thing you could do to someone feeling that way is criticize their question-asking skills. People go to StackOverflow looking for help, not trolls bickering about the question-askers’ grammar or coding abilities.
Main finding of the survey: Unwelcoming community [10.6%]
Response:
“Your feedback about unwelcoming community and perceptions of overmoderation led us to prioritize two important user-to-user feedback loops: [… … …] questions can be improved. We’re also revamping the review queue experience so that it’s easier and more efficient to use.”
Conclusion: Back to square one.
“When users mark questions as solved without giving the answer”
That is literally impossible on StackExchange. Why would someone answer that and why do you highlight it?
Also, please be careful when looking at these complaints about being “unwelcoming” as just a single box: Sure, there can be issues with people leaving comments that insult askers for not knowing everything, but personally I have pretty much never seen such a comment. The much more common complaint about being “unwelcoming” is when someone dumps a million lines of code into the question box, says “PLZ FIX” and gets their question closed or downvoted, which is exactly how the site is supposed to work, but people still label it as “unwelcoming”, because they don’t understand the site and don’t put in even a minute of effort.