What’s in the Works: Improving Feedback for All Users

I’m Meg Risdal, the new Product Manager for Public Q&A. My team works on the public platform and the community tools that help power it (in contrast to our private Q&A products, Stack Overflow for Teams). Recently, we’ve shipped a new tag synonyms dashboard, a redesign of the mod flags dashboard, and Custom Filters. And now, over the coming months, we will be focused on a small number of new initiatives based on the strategy Sara Chipps, our Director of Public Q&A, shared in her blog post “What a very bad day at work taught me about building Stack Overflow’s community”.

Before I dive into specifics about what we’re working on, I’ll tell you a bit more about myself. I came to Stack Overflow specifically to work on public Q&A. I’m passionate about open access to high quality knowledge that helps developers successfully learn and do their jobs. I come to Stack Overflow from Kaggle, a Google company and online data science community, so I appreciate how special it is to be able to learn and work so closely with users in developing products.

As someone who’s worked on a public community product, I’ve always valued transparency with users. For this reason, I’m thrilled to tell you more about what we’re currently focused on and why.

What we’re working on

Last year, we kicked off an initiative to make Stack Overflow more welcoming. We launched a new Code of Conduct, an improved question asking experience, and improvements to the moderator dashboard. We spoke to many of our users about their personal experiences using Stack Overflow, and through these conversations, we heard that the limitations of the Q&A system continue to create the very environment we set out to discourage. People who need help with coding problems feel attacked when their questions are closed or downvoted, while those curating site content feel blamed for doing what the system has asked them to do.

It’s simply not enough to ask people to be nice or change their behavior when the software that underlies everyone’s interactions doesn’t facilitate this. That’s why we’re looking at ways to revitalize the Q&A system, starting with the way questions are handled. We’re working on changes to reduce friction among all users: making embarrassing, public feedback less harsh and more helpful for everyone, reducing the many burdens of our outdated commenting system on questions, and applying what we’ve learned from our new Ask Question Wizard to help all users ask successful questions.

Improving feedback

As Sara noted in her recent blog post, it’s hard not to take feedback personally when it’s piled on, no matter how constructive it is. Unfortunately, this overwhelming pile-on is exactly what users who ask an imperfect question are confronted with. And asking questions that other people can readily answer on the first try is hard enough as is. Not only is it a learned skill to research and ask a well formulated programming question as a novice (yes, I approve of putting “Googling” on your resume), there’s lots of intimidating written and unwritten rules to grapple with. This is why we’re prioritizing improving ways our system facilitates feedback.

The first thing we’re tackling is post notices. If you’ve come across a duplicate question or closed question, you’ve probably seen a post notice. These are the pale yellow informational banners that sometimes appear on questions. For people who ask questions today, if your question is closed, feedback that is directed toward you privately is shared publicly with anyone who views your question. Plus, the names of people who voted to close the question are highlighted publicly, too, setting them up for attack when they’re just trying to curate content according to the system.

Here’s what our holistic redesign of all post notices will prioritize:

  • Delivering improved, private feedback to post authors
  • Not putting users who curate content on the spot
  • Giving actionable, understandable information for the vast majority of public viewers
The graphic below compares an example of a current “on hold” post notice and what it could look like if we provide tailored feedback for authors versus the public view:
A comparison of a current post notice with an example new design.
While working on building post notices, we’re also doing discovery on ways to make our question close workflows and review queues better facilitate feedback and content curation for seasoned moderators, technology experts, and new question-askers alike. We’ve already heard lots of ideas from our moderators and curators who use the review queues to know that there are immediate improvements we can consider making. This will supplement the qualitative and quantitative research we do to identify near-term improvements. Finally, we will be undertaking this work with an eye towards long-term changes to ensure sustainability.

“They told me not to read the comments…”

On Stack Overflow, comments were originally intended to be used to facilitate clarifying questions in order to improve the quality of the question (both as a long-term artifact and so it can be answered accurately). However, it probably comes as a surprise to no one that the comments section on Stack Overflow doesn’t always serve this purpose in practice. Comments can be distracting, outright harmful, pure spam, and everything in between.

We encourage people to flag content that doesn’t belong, and our moderators do incredible work to review everything. But there’s a lot of opportunity to reduce the burden for moderators and users who flag inappropriate content in ways that make using Stack Overflow more pleasant for everyone.

For example, some of the things we’re exploring right now:

  • Better distinguishing the “Answer” and “Comment” actions (due to a high volume of helpful “Not an answer” flags indicating problems with the interface).
  • Ways to reduce the number of unhelpful comments that are likely to get removed in the first place.
There’s a lot we could do to make commenting and interacting with questions more useful and intuitive for everyone. If this is something that interests you, share your feedback with us! Join our research list (you’ll need a Stack Overflow account) for opportunities to participate in UX research.

Improved question-asking guidance for everyone

Several months ago, we launched the Ask Question Wizard on Stack Overflow. The new experience defaults users with low reputation to use a guided mode to formulate their questions. So far, we’re pleased with how it’s helping users ask better questions and as a result have a better experience on the site. In fact, we’re so pleased that our next iteration is focused on incorporating aspects of the guided mode into the question-asking form for everyone.

We’re still in early stages, but some of the changes we’re working on so far include:

  • Upfront guidance for first-time question-askers
  • Setting expectations for what happens after asking a question
  • Improved “how-to-ask” guidance while drafting a question
  • Making it easier to improve question quality by consolidating the many dozens of validation messages into a single “review” interface

We’re excited to begin working on these changes because it will make it far easier for us to extend the improvements across the rest of the Stack Exchange network sites as well as for our Stack Overflow for Teams private Q&A products. It will also give us an opportunity to create a better user experience for everyone. More and more, we hope to look for ways to simplify our platform to seamlessly and intuitively meet the needs of all users who come to Stack Overflow to find and share knowledge.

Dr. Julia Silge, our data scientist at Stack Overflow, will be sharing more insights from how the Ask Question Wizard is doing very soon. Stay tuned!

What’s next

We know that millions and millions of users come to Stack Overflow each month and rely on and contribute to the site and community in so many unique ways. Stack Overflow wouldn’t be what it is without those contributions. This is why I’m excited for our team to focus on ways we can make it easier for everyone to participate.

We will share more updates on the blog as our work progresses. Subscribe to the blog to keep up with the latest on Stack Overflow.

Related Articles


  1. Journeyman Geek says:

    The on hold message was supposed to be ‘friendlier’ than closed – indicating that well, there was *hope*. Might be worth emphasizing that there’s a chance to reopen once fixed. As is, I fear that new folks might pick the ‘ask a new question’ option over fixing it up.

    1. I agree, “closed” is often seen as a final decision — all their effort thus far has gone to waste.

      The phrase “not currently accepting answers” tries to communicate that this state can be temporary, but it is too easily glossed over and missed.

  2. Well, this looks hopeful.

    the only suggestion I have so far would be to extend “Not putting users who curate content on the spot” to the “author view”. It’s really not relevant who closed to post… Fix it as per the provided guidance, and you should be fine.

    1. The “provided guidance” is, by design, extremely general and of course nothing that isn’t in the guidance shown when writing a new question in the first place. I think it’s a good thing to have the option of contacting somebody involved in the moderation process for private help understanding what improvements need to be made, especially given that the embarrassment of being the subject of a public notice that essentially signals that you haven’t understood how SE works is the problem they’re trying to fix.

  3. Marcel Krüger says:

    The new interface says “Private feedback for you”, but the content afterwards is almost exactly the same as the content in the public part. That feels a bit dishonest. The only truly private part is the list of close voters and I’m not sure that this should be completely hidden. Maybe the Reopen queue could show this list too? Especially when thinking about reopening a question it is nice to have the full context of why it was closed in the first place.

  4. I agree with Marcel Krüger. The list of close voters should still be public, but its visual significance should be reduced.
    Another important thing which I would suggest is getting rid of the *two* post notices on duplicate-closed questions, one above and one below the question text. Many new posters are confused by this.

    Finally, please don’t suggest to them to post a new question right away. Sure, it’s easier for the author because the new question won’t be closed and they’ll more likely get an answer, but it makes curating questions difficult. Often some information is shared in the comments of the closed question that isn’t available on the new one. Instead, please focus on editing, and make a substantial edit count as a reopen vote.

    1. I think the list of close voters should be public to users above a certain rep privilege only (maybe > 1000 rep). It’s only useful to users with an understanding of SO’s curation system. And close voters should be notified if the question body is edited (I don’t think edits need to count as reopen votes then).

  5. Nathan Oliver says:

    Only showing the close voters to the OP seems backwards. If you don’t want the OP felling targeted you shouldn’t show it to them. The general users don’t really care and there is no need to hide who closed the question to the general public.

  6. How do we square “increased transparency” with hiding the usernames of closed voters from the community? The author is almost never a person who can accurately call out when close voters have inaccurately closed a question. Likewise this will directly affect our ability to maintain transparency in groups like SOCVR and potentially SOBotics.

    Please at the *very* least allow this to be a rep privilege that allows users to see who close voted a question at 3k, 5k,10k, something.

    1. Yes, current plan is that there will be a variant for users with the close/reopen voting privilege which shows user names.

      1. That’s reassuring. Thank you!

      2. Mark Amery says:

        I hope that comments will likewise be visible to all users above some fairly modest rep threshold, for similar reasons. I can’t say I’m sold on the change at all; it seems to me that its main effect will be to shield assholery from public view, and therefore to encourage it.

        (Think, for instance, of some of the screenshots of vile DMs that get shared by Twitter users who are being harassed. The average Stack Overflow user is, of course, a good deal less horrible than the average Twitter user. But I suspect that a general trend will hold true that people feel empowered to say nastier things when their words aren’t visible to the public.)

  7. Oh good. As a site curator and *especially* as a mod, I want to casually see (without having to dig into history) who’s casting those votes, in case there’s a pattern I ought to be aware of. People with the close/reopen privilege should see it; others needn’t. And we already change the link based on having or not having that privilege, so changing the post notice doesn’t require any additional checking.

  8. I always felt that being listed as a user who voted to close a question for “close reason A” when I actually voted for “close reason B” (just because the majority voted for close reason A) was more problematic than having my user name displayed in the first place. I don’t mind people seeing my name there; it probably makes me consider things more carefully before voting to close.

  9. Kolappan N says:

    This is a great improvement. Keep up.

  10. Please reconsider this for small sites. On small sites, where the users know each other, having your username displayed there is a huge incentive to do reviews and flags as good as possible.

    If the username is no longer shown, what prevents users from doing robot reviews? On SO audits do that, but small sites don’t have review audits.

  11. James Coyle says:

    Why show the asker the people that closed the question and not the other way around? Surely most abuse towards closers comes from the asker themselves? This information is pretty much useless to the asker but is important for transparency with the rest of the users.Having my name publicly visible on these messages adds accountability that my action was the correct one and gives me feedback that I’m making a difference.

    You still seem to be using ‘closed’ which makes it sound like the question is done. I fear this will lead to a lot of new questions when the user should have simply edited their existing question. ‘On hold’ gives the asker hope that their question is still viable and just needs some work. I feel like closed should be reserved for off topic questions which are definitely not salvageable. Perhaps the close vote reasons could be reworked to better define this separation.

    The thing you guys need to focus on is the actionable information you provide to the asker which details why their question is on hold and exactly what they need to do to fix it. If the question is closed as lacking the required code the user needs to understand how to create a minimal reproducible example as well as how to mark up code (or snippets). If the question is closed as duplicate, they should have tools that let them review duplicates and optionally edit and vote to reopen with a message explaining why it isn’t a duplicate. If it’s too broad, explain that questions need to ask one specific thing and it’s ok to post multiple questions. If it’s unclear, provide grammar and structure guidance.

    I feel a rework of the close votes would greatly help with this issue. Focus on what guidance can be given to the user and have a close vote for each reason: duplicate, unclear, too broad, lacking repro. If closed with one of these reasons, mark as off topic and provide guidance to the user based on the close reason. The remaining close reasons should then be grouped into the off topic category and should actually close the question but allow the user to request to reopen.

    It might also help to look into how the system works when the user edits their question. The user could be given feedback sooner when their question is marked for closing and could be encouraged to edit their question to address the close vote(s). This could then be sent back to the original close voter(s) for review who can then choose to retract or modify their vote(s) as necessary. I feel this would be less harsh on users as the language could be less definitive and suggest changes the user could make before their question is closed.

    1. James Coyle says:

      Apologies. The line:
      If closed with one of these reasons, mark as off topic

      Is supposed to say:
      If closed with one of these reasons, mark as on hold

  12. Mark Amery says:

    After posting my previous comment, I realise I don’t understand something crucial about this post: is it part of your intent to make comments on closed questions private and only viewable by the question author?

    That was how I first understood this, since the only thing I really think of as “feedback” is comments, and comments are mentioned here… but on a reread I realise it’s not explicitly said. Have I misunderstood, and is the intent actually to only make the close reason and voter list private?

  13. I wonder if question feedback similar to Word/Google Docs review modes (comments and suggested edits to the side as actionable items), and locking+disabling comments altogether would help clarify that distinction.

    Feedback could be collapsed by default, and expanded by those who want to interact.

    It also means a comment like “what do you mean by…” can be directly attached to a (highlighted?) part of the question text.

    Obviously this is still abusable, but it may be an interface that strongly encourages question feedback over discussion.

  14. This isn’t friendlier or clearer, particularly for a new user: “Closed. This question needs details or clairity. It is not currently accepting answers.”.

    Translation: “Smack. You did it wrong. Here’s something else you won’t understand.”.

    While I don’t like the change a better way to word that would be:

    “Our users feel your question is unclear or lacking details, please edit it to better describe your problem in accordance with the [Help](https://meta.stackexchange.com/help/how-to-ask).”. We may vote to reopen it afterwards or you can choose to delete it.

    See: No smack. An explanation, and options.

    You say: “… the names of people who voted to close the question are highlighted publicly, too, setting them up for attack when they’re just trying to curate content according to the system.”.

    So you give the information to the person whom had their question closed, inviting comments to them from the affected user, and risking revealing their information (plus it’s available by other means, but probably not to a new user).

    Basically the unknown person affected is bound to be nice, and the rest of us; well we’re just …

    Best to leave things rather than making them worse.

    Also better to write this on Meta, I’d be willing to accept votes on this answer.

  15. This may just be the jaded old timer in me, but I feel like there are better, more low hanging projects that have been suggested in metas across the entire network that would have been better things to start off on: https://meta.stackexchange.com/questions/tagged/feature-request

    Instead, we’re getting “We need to be nicer! Let’s try and fix that, again, for the third time at least!!!” This is a nebulous, on-going problem and isn’t going to be fixed straight out of the box. Having it as the first project for this public-facing team just… makes me feel like our voices mean nothing.

  16. Also, IMHO the real fix to all of the “Be Nice” problems that Stack Overflow has is to break it up into smaller sites. I rarely hear about people complaining about rudeness on the smaller sites, it’s ALWAYS Stack Overflow.

  17. T.J. Crowder says:

    As you’re updating the post notices, please fix this problem: https://meta.stackexchange.com/questions/269073/dont-say-i-marked-something-as-a-duplicate-when-i-didnt It’s REALLY important not to say user A did something user A didn’t do. This is true for all close reasons, not just duplicates. Don’t say I closed it as unclear when I voted to close as a duplicate; don’t say I voted to close as off-topic when I voted to close as unclear.

    I don’t see why it has to be one reason at all. If everyone voted the same way, great, just go with a single message. If not:

    * * * *
    This question is not currently accepting answers:
    * It needs more clarity and details – UserA, UserB, UserC
    * It needs to be made more specific – UserD
    You can [edit the question] to correct those issues, so the question can be answered.
    * * * *

    I Agree with Bergi that you shouldn’t suggest posting a new question there.

    Another thing related to question closure is the need to **notify close voters when the question is edited**. Some people won’t want that, so let them turn it off in their preferences, but by default, if you’ve voted to close a question, you should reevaluate that vote when it’s edited. Related: https://meta.stackexchange.com/questions/289620/allow-opt-in-subscription-to-a-post
    If a user wants to normally be opted in but mute a particular question, he/she should be able to do that.

    Probably need something similar for downvoters. There’s too much fly-by downvoting. If you downvote something, and someone tries to improve it, you should be notified of that. (Again: Make it possible to opt out, in general and per-post.) This one applies to answers as well as questions.

    1. Yes, I think it’s FAR more important to accurately report on the feedback from close voters than it is to hide their names. Actually, I think it is *undesirable* to hide their names.

      1. I agree that hiding names is undesirable. Those who provide feedback (including upvotes and downvotes) should be accountable for their actions, identifiable, and open to challenge.

  18. “People who need help with coding problems feel attacked when their questions are closed or downvoted, while those curating site content feel blamed for doing what the system has asked them to do.”

    I find this an excellent summary of the problem, and hearing it phrased this way felt great.

    The image of the improved notification confused me. Was that supposed to be an example of improved, tailored feedback? I don’t see how it’s saying anything different, or how it’s supposed to be better than the original. There was also no mention of why “On Hold” changed to “Closed”. Are you planning to do away with the “On Hold” status?

  19. Casey Crookston says:

    One thing I would LOVE to see is not being able to downvote a question without an explanation. Often new users who are still learning how to ask questions will get downvoted, and most of the time, there is NEVER an explanation as to why… leaving the new user feeling lost and confused.

    1. Agree absolutely. I hate downvotes without explanation, and I see lots of people asking “why was that downvoted?”. Last time I said this on SO Meta, I got downvoted without explanation.

    2. This comment section needs votes!

      I totally agree. And with downvotes on Answers also. As well as it being confusing for new users receiving a question or answer that is downvoted, I often find it confusing as a reader and seeker-of-more information. I sometimes read a Q or A and see it has a negative score but no comments. And in those times I sometimes think it is a good Q or A (not worth a downvote at least). I’m left wondering WTF! I will add a comment asking for the down-voter to say why. And I’ll upvote it normally also. Downvoters don’t normally return. They may just be trolls for all we know. The analytics around votes and moderator actions is a good idea.

      I think downvotes should require comments (perhaps a free field input as well as some canned responses). And allow them to be moderated also.

  20. historystamp says:

    This place isn’t friendly to new users. With one closed question. I had an answer. I directed the user over to discussions.apple.com. The user posted the same information on there and I answered the quested with only one reply.

    The moderators are to quick to put posts on hold and free posts. I think this activity should be banned. I do not like questions and answers being down voted. Should be able to only up-vote questions and answers. While the results would be about the same, no down-votes would eliminate all the negative feelings.

    It a real art to ask informative questions. A lot of beginners don’t have the skills to ask informative question. They get beaten up around here.

    Most questions need clarification. Let’s say a program is interacting with the OS. It would be good to see a trace off all system calls. You can say get a system trace, but where do you explain how to do the system trace. I put such detailed information in an answer only to see my answer get dumped on.

    The format of these forums works well for certain questions, but not for all questions. This format doesn’t work well with the “ask different” forum. For instance, someone says my system is running slow. You can clearly imagine what this poster get bombarded with.

    Why is is illegal to put my name at the end of a post. Its a mistake to take out the personal touch in the forums.

  21. As regards the distinction between comments and answers, I think one of the difficulties here is where the problem-solving process needs dialogue. “I do X, and Y happens”. “Are you running on Linux?”. “No, Windows”. In that case, does it work if you do W?” “No, it still fails”. “In that case, consider doing V”. “Thanks, that worked”. It’s never been clear to me (as a very experienced SO user) how this kind of chit-chat is supposed to be conducted. Some people recommend continual editing of the question and answer, but that often feels unnatural, and it hides the route by which the solution was reached.

  22. One thing I find frustrating is when a question has been closed, based on the views of people who didn’t understand the question, when I think that I DO understand the question and can answer it. For example, the reason the question can’t be understood might be because the writer is not a native English language speaker, but if you read enough English-as-written-by-Germans (to take an arbitrary example) you start to learn how to read it. Or it might use incorrect technical terminology, e.g. SQL terminology in an XML question, which only becomes comprehensible if you understand both technologies.

    1. This is really a great idea! I think a lot of friction between curators and askers can be reduced by improving the “on hold” dialogue. In its current state it sends confusing signals to both curators and askers (c.f. T.J.Crowder’s and Bergi’s comments).

      I’d also like to thank you all for involving the community at this early stage, as well as your recent Meta participations (and the increased number of [status-completed] FRs…), that really reduces friction between the Meta Community and the company …

      As the current plan seems to be that there is a third version for curators, here is my personal wishlist 🙂 (the other versions were already covered extensively by others, I thought it might be good to share some thoughts at this early stage):

      (1) Whenever a question gets into close/reopen conflicts, I have to dig through /timeline/ to find out what went wrong. It would really help to see when a question was closed / reopened / edited by whom (experienced user / sleeping reviewer) directly from the main site (maybe unfoldable to not hide too much content ?).

      (2) I’m really glad to reopen the questions I initially closed. It’s great to see if people actually improved their question. Yet, only a small amount of askers pings me in the comments, and the review queue is fundamentally broken. I would like to (opt-in to) receive notifications when the asker made a (substantial) edit.

      (3) There is currently no way to change the close vote reason. Sometimes the asker edits an unclear question to be a clear duplicate, in that case changing the close reason would be useful for all.

  23. Michael Freidgeim says:

    “I’ve always valued transparency with users.”
    There is Meta StackExchange that is very good to discuss new features like announced here. The format of MSE is much better for feedback, including answers and votes(not available in blog comments). I do not believe that professional who confident in their proposals,”have panic attacks and nightmares when they need to post something to Meta”(from https://meta.stackoverflow.com/questions/387546/we-re-removing-hot-meta-posts-from-stack-overflows-sidebar-for-now-moderator/387633#387633).
    If you haven’t published these proposals on MSE, please publish any future proposals to get better feedback

  24. Евгений says:

    I read the correspondence and saw the second side that not everyone wants to notice, I think that there is fear from two sides in communication and this leads to a decrease in the level and quality in the study of new applications and an underestimated assessment of the second side, since a number of actions and solutions are not known to the second side and this leads to a number of incorrect actions that lead to the fact that there is simply a closure and deviations from the norm of studying and deciding to exit the current situation correctly thought out which can only be given by a person with experience and knowledge of business (and knowledge of English)))))
    The conclusion is drawn from personal experience and most of what happened could have been turned into a positive result, which would have led to a great beneficiary of benefit both for you and for the second side.

  25. I rarely use stack forums as when I feel a need, I have already spent days, often weeks, looking for an answer and when I come here, stressed, I pull back because the last thing I need at that moment is to experience more negative of anything. As a result, I am a searcher here and when I do not find a response, I move back out to the internet. The hardest aspect is that the mission of the site is for learning and yet the oxymoron is that the best questions are asked by those who are experienced. I can fully appreciate that some folks do not do homework and ask questions that causes people to do their work for them. I lived within the Drupal forums for years, and as I became more experienced there, I found myself becoming less patient with newcomers who I believed had not worked as I had worked to learn. I failed to acknowledge that there are people who learn best in to-fro discussions and others who prop their heads on their elbows and dig in, reading until the light bulb comes on. I still battle that impatience but my point is that I understand a need to head off, well, laziness.

    I am posting now because I am overjoyed to learn that there is recognition of a problem in this world and to read of the steps being taken to make it less intimidating to new posters.

    I suggest that instead of saying that the question is not currently accepting answers, state the truth instead:

    “This question cannot be answered well because it lacks clarity, needs xxx, requires repeating answers already given elsewhere, etc.

    “When it is edited so a satisfying answer can be provided, we’ll re-open it to respondents.”

    Obviously that needs polishing but the point is to emphasize that the wording of the question makes it impossible to be helpful or wastes other human’s time — rather than emphasizing “closed” which is like hearing a door slam.

    Just a suggestion…

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.