Research update: Improving the question-asking experience
Hi, my name is Lisa. đ I am a Product Designer on the Community Product team and Iâm excited to provide this monthâs UX research update! If youâve asked a question recently, you may have noticed some changes to the question-asking form. This is the result of months of research reviewing the question-asking experience onâŚ
Hi, my name is Lisa. đ I am a Product Designer on the Community Product team and Iâm excited to provide this monthâs UX research update!
If youâve asked a question recently, you may have noticed some changes to the question-asking form. This is the result of months of research reviewing the question-asking experience on Stack Overflow. Regardless of how much reputation users have on Stack Overflow, they have told us that asking a question on the site can be an intimidating experience. And following the success of the Ask Question Wizardâour guided ask question form for users with less than 111 reputation â we decided to take a step back and look for ways we could make a more unified, positive experience for everyone.
Objective
The goal of this research project was to understand people’s experiences and perceptions of asking a question on Stack Overflow. We took a look at the question-asking experience more holistically and examined the current state of Stack Overflow to understand how people were solving their programming issues on and outside of the site.
We focused on these research questions:
What are people’s experiences asking a question on Stack Overflow?
What information do users need before asking a question on Stack Overflow?
How does Stack Overflow fit into peopleâs problem solving workflows?
Methods
This research project used multiple methods of collecting both qualitative and quantitative data. We conducted data analysis on the behavior and characteristics of our current user base, dove deep into the performance of the Ask Question Wizard, gathered insights from 1:1 interviews with low-reputation lurkers to high-reputation veterans, and studied other sites in UX teardowns.
Results
Based on our research, we were able to generate a better question-asking flow that unifies the traditional and guided (Ask Question Wizard) modes and works equally well for users of all reputation and experience levels. Here are some of the insights that informed our designs:
Writing a good programming question takes time and effort
The Ask Question Wizard was originally intended to help first time question-askers learn how to write good programming questions. Despite their level of experience, users agreed that writing a good programming question was a challenging, learned skill that takes instruction and practice.
The Ask Question Wizard has proven to be impactful in assisting new users in writing better quality questions, so we decided to take what weâve learned from this update and make it available to all of our users.
The new design offers upfront guidance and a more approachable design. Users can focus on drafting their question without distraction and are provided instruction all in one space with less need to navigate away from the form.
Users anticipate negative interactions when asking questions
Users expressed that the post-publishing experience caused anxiety and frustration. Concerned with how the Stack Overflow community would receive their question, users wanted to make sure that they were not posting a duplicate question and that their question fit into community standards and expectations. As part of user interviews, when participants were asked to review a sample of existing Stack Overflow questions they noted how certain questions were likely to receive snide remarks from the community.
We want all of our users to feel confident in asking a question. To assuage these user concerns, we have plans to include a single âreviewâ interface that consolidates error messaging and provides suggestions to help improve question quality.
Itâs not clear what happens after submitting a question
Most participants mentioned the importance of an eye-catching title, but were otherwise stumped as to how or why certain questions seemed to solicit answers. We heard a lot of questions around what to expect after a question is submitted. Many users were uncertain about what happens after a question is submitted. Additionally, not receiving an answer also caused anxiety. Users wonder, âIs there something wrong with my question?â
To give users a greater sense of awareness of what happens once they publish a question, we introduced an informational modal that appears once a user submits their first question.
Whatâs next?
Since releasing the design on Stack Overflow to a sample of users, we have been collecting feedback and monitoring data in A/B tests. We plan to take your thoughts and suggestions alongside our analysis of the quantitative data to make continued improvements on this design. Weâre excited to work towards a single question-asking experience which helps all users ask questions with confidence and weâll continue to provide updates on the blog as our work progresses.
If youâd like to hear about opportunities to participate in future user research, join our research list (youâll need a Stack Overflow account).
Thank you everyone involved in this project so far: Jane Willborn, Brian Nickel, Jon Ericson, Jon Chan, Donna Choi, Dr. Julia Silge, Aaron Shekey, Meg Risdal, Sara Chipps, and many others.
Tags: bulletin, community, questions, stackoverflow
The Stack Overflow Podcast is a weekly conversation about working in software development, learning to code, and the art and culture of computer programming.
Charles âCobihâ Obih and Radek Markiewicz of the Stack Overflow platform team join Ben and Ryan to talk about changes to the inbox and what itâs like to build Stack Overflowâs public platform.Â
Dr. Jeannette (Jamie) Garcia, Senior Research Manager of Quantum Applications and Software at IBM Quantum, tells Ryan about IBMâs 433-qubit quantum computer and the real-life applications of quantum computing today.Â
Did you do any research into how painful it is to read all the badly written questions? How about improving the experience for the people willing to *answer*?
Did you do any research into how painful it is to write a good answer to a good question, then see itâs erased along with the question due to people who have no clue whatâs asked clicked âtoo broadâ downvote?
I stopped writing answers there for this exact reason.
Have you considered that a lot of software engineers aren’t the best communicators? I’ve personally asked a grand total of three questions on this network since April 2012 and it all left me bitter even though I know they are valid questions. To add to the insult, people downvote without any feedback. Isn’t that even worst than asking a “bad” question? The reason why I don’t ask questions is because I normally find all my answers on Google. I come here as a last resort and I get bullied by ignorants.
Are you aware of how hard new users try to write a good question only to be bashed with downvotes with NO explanation… If a person wants painless questions he would be well advised to give feedback to new users on WHY there question is lacking…
Overall I think this is a good idea, however I have severe reservations about this line:
“You’re helping us build this community where everyone can learn from each other.”
The mission of Stack Exchange has *never* been the creation of a community. As stated on *every single tour page in the network,* it’s:
“With your help, we’re working together to build a library of detailed answers to every question about ____________.”
What has made Stack Exchange valuable is NOT its “community relationships.” What has made Stack Exchange valuable is its library of detailed answers, which *everyone* can use for reference, whether or not they are a part of the “community.”
This has always been the guiding principle behind design decisions for Stack Exchange. For example, the reason there is no “follow” feature (to follow answers from some top user who is very expert in XYZ technology) is because SE is NOT a social media site. If you think about it, a “community where everyone can learn from each other” should *absolutely* have a “follow” feature; it contributes directly to that goal. But a “library of detailed answers to every question about ____________” should emphatically not have a follow feature.
Please, please, please re-focus on this goal. Fix the wording in the “What to expect” message shown above.
Suggested replacement wording:
“You’re helping us build a library of detailed answers to every question about _____. Answers can’t exist without good questions!”
Looking at this reply on Oct 3 and I can’t find the quote you mention.
Nevertheless, I agree. I agree with the original mission as stated. Let us build a compendium of intelligent questions and answers to ______ topics, in varying degrees on complexity and specificity.
“Before you post, search the site” is bad recommendation. I’d say “google for answers”, google does the job much better.
“We hear”, “based on research” – where you hear, what research? Can we get statistic? Since you are not posting results on some meta – it seems you don’t need feedback from comunity and already know what to do. Good good.
The problem with Google is that you get hits on Quora, Reddit, Geeksforgeeks etc etc – various low quality sites that are very prone to provide technically incorrect, low quality answers. So as it turns out, Google does a much worse job than the SO search engine.
Though it would perhaps be nice to provide a Google search restricted to the SO domain only.
Google can do that and many other fancier things with search operators. You add site:stackoverflow.com anywhere in your query to only show results from that domain. Google “google search operators” and I’m confident you’ll find a full list.
That can easily be fixed by adding ‘site:stackoverflow.com’ to the search terms. In fact I often do this to find a duplicate to VTC. Perhaps that tip would be more worthwhile in the question guide.
âConcerned with how the Stack Overflow community would receive their question, users wanted to make sure that they were not posting a duplicate question and that their question fit into community standards and expectations. â
My own (limited) experience on StackOverflow is that the majority of users do little or no prior research, care not a jot whether they are posting a duplicate question, and have little or no understanding (let alone âregardâ) for community expectations. Poorly worded questions abound.
It is missing information about the time scale Stack Overflow operates at (attention usually stops in less than 15 minutes). Shouldn’t there be a sentence about the expectation of staying around to promptly respond to comments (response time in minutes, not hours or days)? In fact, the part about email indirectly suggests hours or days (and reinforces the forum expectation, hours or days).
Doesn’t the time scale really depend on the tag? In my experience, Javascript and C# questions get a lot of immediate interest, while questions in VBA and VB.NET get responses in about a day or so; more esoteric tags can take even longer
Please could you emphasize that the user needs to stick around for a while immediately after asking their question.
This could prevent a lot of the incidents where a question is posted, further information is requested a few seconds later, and the OP returns the next day to find their question has been downvoted to oblivion and closed just for lack of a simple clarification or two.
To be honest with you, for newbies there is a big learning curve. I made some posts that were too “vague” and I was penalized for it. Hardly a user-friendly experience for me. I am told now that I cannot post any more questions until my reputation goes up which hardly gives me warm feelings for the community. I must admit I didn’t read the guidelines, my mistake which I take full responsibility for. I also understand that ubuntu is a community which is strong and protects itself and I respect that. Still I feel like I am in the proverbial doghouse and trying to work my way out of feeling penalized for my vague postings.
It is very surprising that this new interface were thrust upon *everyone* regardless of whether they already have asked many (in my case over 1000) questions. It is not at all clear it were helpful even for *new* users: but it *is* clear that is wasteful of real estate and distracting to seasoned ones.
“Users expressed that the post-publishing experience caused anxiety and frustration. Concerned with how the Stack Overflow community would receive their question, users wanted to make sure that they were not posting a duplicate question and that their question fit into community standards and expectations. ”
True. I did that in the beginning. After I learned how to write a good question, this feeling went away. I guess a few downvotes are good for the beginner.
I think you miss a big point with your words “Concerned with how the Stack Overflow community would receive their question, users wanted to make sure that they were not posting a duplicate question”. Finding duplicate questions is not easy for someone new to a technology, but some experienced people take great pleasure in knocking you down for asking a duplicate question.
Other times the question is duplicate but different enough to be a value to the person asking the question. Again, being knocked down for posting a duplicate question is darn annoying.
Why not provide a mechanism for experience people doing the knockdowns to group these questions so that people new to a technology see the groupings.
btw, the best way to find duplicate or similar questions is “google stackoverflow xxxx”.
It is really questionable whether this UI helps new users. It is *not* a question that it is a distraction and annoyance for experienced ones. If we already have double digit questions then why show it. I have over 1000 – yet the same garish UI is shown to me. Also see: https://meta.stackoverflow.com/questions/389830/opting-for-the-traditional-new-question-ui
I get that new users can feel intimidated, because sticking your nose out on a site with many users who may/may not know more than you, is always going to be that.
While we want to make it welcoming them to encourage them to ask sensible questions, despite any forms and pages of help, half of them still expect us to be able to tell what they meant, and imagine the errors they are seeing, work out which of the 5 languages they listed they actually used, and to form an intelligent response to not what they asked but what they meant.
As a new user, I completely understand and appreciate these attempts. I’ve had multiple times where I was stymied in my attempt to ask what was a well-thought-out question.
However.
A well-thought-out question is NOT always a good question. Here, in my opinion, is why:
I recently asked a now-deleted question about the C language that essentially said
“When I run the following line
printf(“%d\n”, ULONG_MAX);
the console outputs ‘-1’. However, I know that the value is supposed to be
18,446,744,073,709,551,615.”
It turns out there was nothing wrong with the library holding ULONG_MAX, but with my format specifier “%d”. Consequently, the question was a duplicate of the question about the format specifiers for long integers. My question was closed as a dupe, and this was the correct decision. My question, despite the effort I put into it, was not a good question.
The first problem (in my view) is that I don’t know enough to ask the correct questions. No amount of SO question-asking tutorials will teach me how to ask the right question – that’s something I postulate comes through experience. And you know what? On the whole, this particular interaction with other SOers was decent, and I learned from it.
The second problem (in my view) is that more experienced users don’t always know that I don’t enough to ask the correct question. I CAN’T google it, darn it, because I don’t know what to google. I tried. And yes, I failed where you succeeded. But, IF POSSIBLE, (cause we all know sometimes it just ain’t so) please tell me how I failed so that I can succeed. Another question I asked had 4 unexplained downvotes – the answer could have been googled if I had known that a C++ website was an OK source for a C question, but I didn’t. The downvotes didn’t really hurt – it was that they were unexplained (I assume they came because my question was very low-level). I presume it’s a bad question, but I come away learning little.
In short, I doubt that the proposed solution will work, but frankly, I don’t know what will.
I appreciate the immense amount of work put into this new feature, though, and I wish that it weren’t so summarily dismissed.
You didn’t know the full nature of your problem, because that’s what problems tend to be in the first place. So you blamed the wrong thing (we all do that) and didn’t know your problem was a duplicate. Don’t blame yourself for it
It’s helpful when mods add “Possible Duplicate” links. It’s goddam **hostile** when legitimate questions get downvoted or closed because of the existential horror that a duplicate might ever exist. I stopped contributing to Wikipedia when even _spelling_ corrections got reverted.
The few questions I’ve asked on SO, I’ve had great experiences with. I’ve also seen plenty of other users get hostility from self-appointed technocratic guardians.
SO could be improved 1000 fold overnight with one simple change: give new users feedback on why there questions are being downvoted. This would improve the quality of the questions and consequently lead to better questions for the answerers and a more cordial atmosphere at SO. Currently the atmosphere is one of hammer bashing unfriendliness and anxiety for new users. I got down voted on my first three questions with no explanation in spite of my best effort to ask correctly AND ask for feedback on the question (Guess its easier to just hit a button). Lucky for me your reputation can’t go below 1. It took about two months and me being desperate for an answer to finally take another chance with SO. I finally started to increase my rep.
A simple solution would be to not deduct rep points if a person gives an explanation for the downvote and deduct 5 if they don’t
33 Comments
Did you do any research into how painful it is to read all the badly written questions? How about improving the experience for the people willing to *answer*?
Did you do any research into how painful it is to write a good answer to a good question, then see itâs erased along with the question due to people who have no clue whatâs asked clicked âtoo broadâ downvote?
I stopped writing answers there for this exact reason.
Is this not exactly what the point of this is?
If this raises the quality of answers, then the experience for people willing to answer also increases.
Have you considered that a lot of software engineers aren’t the best communicators? I’ve personally asked a grand total of three questions on this network since April 2012 and it all left me bitter even though I know they are valid questions. To add to the insult, people downvote without any feedback. Isn’t that even worst than asking a “bad” question? The reason why I don’t ask questions is because I normally find all my answers on Google. I come here as a last resort and I get bullied by ignorants.
Are you aware of how hard new users try to write a good question only to be bashed with downvotes with NO explanation… If a person wants painless questions he would be well advised to give feedback to new users on WHY there question is lacking…
Overall I think this is a good idea, however I have severe reservations about this line:
“You’re helping us build this community where everyone can learn from each other.”
The mission of Stack Exchange has *never* been the creation of a community. As stated on *every single tour page in the network,* it’s:
“With your help, we’re working together to build a library of detailed answers to every question about ____________.”
What has made Stack Exchange valuable is NOT its “community relationships.” What has made Stack Exchange valuable is its library of detailed answers, which *everyone* can use for reference, whether or not they are a part of the “community.”
This has always been the guiding principle behind design decisions for Stack Exchange. For example, the reason there is no “follow” feature (to follow answers from some top user who is very expert in XYZ technology) is because SE is NOT a social media site. If you think about it, a “community where everyone can learn from each other” should *absolutely* have a “follow” feature; it contributes directly to that goal. But a “library of detailed answers to every question about ____________” should emphatically not have a follow feature.
Please, please, please re-focus on this goal. Fix the wording in the “What to expect” message shown above.
Suggested replacement wording:
“You’re helping us build a library of detailed answers to every question about _____. Answers can’t exist without good questions!”
I agree one hundred percent with this comment.
Maybe treating people like text factories is part of the problem.
I agree, they should be taking responsibility for asking good questions not how to pander to their feelings cos a bad one was asked.
Looking at this reply on Oct 3 and I can’t find the quote you mention.
Nevertheless, I agree. I agree with the original mission as stated. Let us build a compendium of intelligent questions and answers to ______ topics, in varying degrees on complexity and specificity.
“Before you post, search the site” is bad recommendation. I’d say “google for answers”, google does the job much better.
“We hear”, “based on research” – where you hear, what research? Can we get statistic? Since you are not posting results on some meta – it seems you don’t need feedback from comunity and already know what to do. Good good.
P.S.: I am not Jon Skeet.
The problem with Google is that you get hits on Quora, Reddit, Geeksforgeeks etc etc – various low quality sites that are very prone to provide technically incorrect, low quality answers. So as it turns out, Google does a much worse job than the SO search engine.
Though it would perhaps be nice to provide a Google search restricted to the SO domain only.
I mean… SO is also prone to technically incorrect, low quality answers. I would even go so far as to say the majority of the answers are low quality.
You can, just add site:stackoverflow.com to the search query, it will only fetch results from url that match that query.
Google can do that and many other fancier things with search operators. You add site:stackoverflow.com anywhere in your query to only show results from that domain. Google “google search operators” and I’m confident you’ll find a full list.
That can easily be fixed by adding ‘site:stackoverflow.com’ to the search terms. In fact I often do this to find a duplicate to VTC. Perhaps that tip would be more worthwhile in the question guide.
âConcerned with how the Stack Overflow community would receive their question, users wanted to make sure that they were not posting a duplicate question and that their question fit into community standards and expectations. â
My own (limited) experience on StackOverflow is that the majority of users do little or no prior research, care not a jot whether they are posting a duplicate question, and have little or no understanding (let alone âregardâ) for community expectations. Poorly worded questions abound.
Regarding the informational modal [sic]:
It is missing information about the time scale Stack Overflow operates at (attention usually stops in less than 15 minutes). Shouldn’t there be a sentence about the expectation of staying around to promptly respond to comments (response time in minutes, not hours or days)? In fact, the part about email indirectly suggests hours or days (and reinforces the forum expectation, hours or days).
Doesn’t the time scale really depend on the tag? In my experience, Javascript and C# questions get a lot of immediate interest, while questions in VBA and VB.NET get responses in about a day or so; more esoteric tags can take even longer
Please could you emphasize that the user needs to stick around for a while immediately after asking their question.
This could prevent a lot of the incidents where a question is posted, further information is requested a few seconds later, and the OP returns the next day to find their question has been downvoted to oblivion and closed just for lack of a simple clarification or two.
To be honest with you, for newbies there is a big learning curve. I made some posts that were too “vague” and I was penalized for it. Hardly a user-friendly experience for me. I am told now that I cannot post any more questions until my reputation goes up which hardly gives me warm feelings for the community. I must admit I didn’t read the guidelines, my mistake which I take full responsibility for. I also understand that ubuntu is a community which is strong and protects itself and I respect that. Still I feel like I am in the proverbial doghouse and trying to work my way out of feeling penalized for my vague postings.
Noticed this the other day. Looks good. Thanks!
This new interface is terrible for seasoned and frequent users. PLEASE add an “opt out” for this interface so we can use the traditional one.
It is very surprising that this new interface were thrust upon *everyone* regardless of whether they already have asked many (in my case over 1000) questions. It is not at all clear it were helpful even for *new* users: but it *is* clear that is wasteful of real estate and distracting to seasoned ones.
“Users expressed that the post-publishing experience caused anxiety and frustration. Concerned with how the Stack Overflow community would receive their question, users wanted to make sure that they were not posting a duplicate question and that their question fit into community standards and expectations. ”
True. I did that in the beginning. After I learned how to write a good question, this feeling went away. I guess a few downvotes are good for the beginner.
I think you miss a big point with your words “Concerned with how the Stack Overflow community would receive their question, users wanted to make sure that they were not posting a duplicate question”. Finding duplicate questions is not easy for someone new to a technology, but some experienced people take great pleasure in knocking you down for asking a duplicate question.
Other times the question is duplicate but different enough to be a value to the person asking the question. Again, being knocked down for posting a duplicate question is darn annoying.
Why not provide a mechanism for experience people doing the knockdowns to group these questions so that people new to a technology see the groupings.
btw, the best way to find duplicate or similar questions is “google stackoverflow xxxx”.
It is really questionable whether this UI helps new users. It is *not* a question that it is a distraction and annoyance for experienced ones. If we already have double digit questions then why show it. I have over 1000 – yet the same garish UI is shown to me. Also see: https://meta.stackoverflow.com/questions/389830/opting-for-the-traditional-new-question-ui
I get that new users can feel intimidated, because sticking your nose out on a site with many users who may/may not know more than you, is always going to be that.
While we want to make it welcoming them to encourage them to ask sensible questions, despite any forms and pages of help, half of them still expect us to be able to tell what they meant, and imagine the errors they are seeing, work out which of the 5 languages they listed they actually used, and to form an intelligent response to not what they asked but what they meant.
Nothing will change on SO until the current culture of downvote-by-default is eliminated. No amount of UI changes will fix this.
As a new user, I completely understand and appreciate these attempts. I’ve had multiple times where I was stymied in my attempt to ask what was a well-thought-out question.
However.
A well-thought-out question is NOT always a good question. Here, in my opinion, is why:
I recently asked a now-deleted question about the C language that essentially said
“When I run the following line
printf(“%d\n”, ULONG_MAX);
the console outputs ‘-1’. However, I know that the value is supposed to be
18,446,744,073,709,551,615.”
It turns out there was nothing wrong with the library holding ULONG_MAX, but with my format specifier “%d”. Consequently, the question was a duplicate of the question about the format specifiers for long integers. My question was closed as a dupe, and this was the correct decision. My question, despite the effort I put into it, was not a good question.
The first problem (in my view) is that I don’t know enough to ask the correct questions. No amount of SO question-asking tutorials will teach me how to ask the right question – that’s something I postulate comes through experience. And you know what? On the whole, this particular interaction with other SOers was decent, and I learned from it.
The second problem (in my view) is that more experienced users don’t always know that I don’t enough to ask the correct question. I CAN’T google it, darn it, because I don’t know what to google. I tried. And yes, I failed where you succeeded. But, IF POSSIBLE, (cause we all know sometimes it just ain’t so) please tell me how I failed so that I can succeed. Another question I asked had 4 unexplained downvotes – the answer could have been googled if I had known that a C++ website was an OK source for a C question, but I didn’t. The downvotes didn’t really hurt – it was that they were unexplained (I assume they came because my question was very low-level). I presume it’s a bad question, but I come away learning little.
In short, I doubt that the proposed solution will work, but frankly, I don’t know what will.
I appreciate the immense amount of work put into this new feature, though, and I wish that it weren’t so summarily dismissed.
You didn’t know the full nature of your problem, because that’s what problems tend to be in the first place. So you blamed the wrong thing (we all do that) and didn’t know your problem was a duplicate. Don’t blame yourself for it
It’s helpful when mods add “Possible Duplicate” links. It’s goddam **hostile** when legitimate questions get downvoted or closed because of the existential horror that a duplicate might ever exist. I stopped contributing to Wikipedia when even _spelling_ corrections got reverted.
The few questions I’ve asked on SO, I’ve had great experiences with. I’ve also seen plenty of other users get hostility from self-appointed technocratic guardians.
SO could be improved 1000 fold overnight with one simple change: give new users feedback on why there questions are being downvoted. This would improve the quality of the questions and consequently lead to better questions for the answerers and a more cordial atmosphere at SO. Currently the atmosphere is one of hammer bashing unfriendliness and anxiety for new users. I got down voted on my first three questions with no explanation in spite of my best effort to ask correctly AND ask for feedback on the question (Guess its easier to just hit a button). Lucky for me your reputation can’t go below 1. It took about two months and me being desperate for an answer to finally take another chance with SO. I finally started to increase my rep.
A simple solution would be to not deduct rep points if a person gives an explanation for the downvote and deduct 5 if they don’t
I don’t even bother asking questions on stackoverflow anymore.