Rolling out the Welcome Wagon: June Update

Back in late Spring, we posted “Stack Overflow Isn’t Very Welcoming. It’s Time for That to Change.” We know it’s all too common for companies to announce they really want to do a thing, and then… not much happens. So, it is time to share the work we’ve been doing behind the scenes and how our team of data scientists, developers, long term community managers, and UX specialists will continue iterating on our plans for the future.

Done

Shipped the “Ask a Question Wizard” prototype

Ask a question wizard

This new workflow should help new askers craft questions more successfully with less stress. Each step gathers specific input that’s generally needed to help answerers. As we test it, we can expand this design to support additional question types. We’ve already done a couple rounds of user testing and are now gathering feedback from the community. You can check it out here and leave feedback on this post.

Built an “Unwelcoming or snarky” comments classification app

Our users have always been really great at flagging and removing outright abusive comments, but we’ve long struggled to deal with (or even define clearly) lower-key unpleasantness: condescension, snark, dismissiveness. We just finished our first step to change that: 57 employees provided 13,742 ratings of “fine,” “unwelcoming or snarky,” or “abusive,” on random comments on SO, and we’ll be sharing the initial results and our next steps in a separate post.

Created a prioritized queue of 67 other next up initiatives

Some of these are already in active discussion with the community, like removing auto-downvotes that come with certain flags. The key here is that we will always have a next initiative to move forward for discussions, input, and implementation, so we won’t lose momentum.

In progress

Comment flagging improvements

We are working on various improvements to comment flagging to increase visibility and allow more people to participate in this critical activity.

Community/civility partnerships

We’ve connected to individuals (researchers, academics, community managers and data scientists) from many institutions and companies that are concerned about civility on the internet. We’ve kicked off collaborations with a couple companies to build machine learning approaches that can address civility in internet scale communities. More data can train smarter analysis, so we are optimistic about these collaborations.  Machine learning isn’t a magic solution nor will it be our only avenue for partnerships, but it will be a part of our toolkit. We hope to have more to share soon.

Upgrading our community standards

We are undertaking a full review of our community standards, including an overhaul of our “Be Nice” policy to evolve it into a full fledged code of conduct, which better articulates exactly what we expect and how those expectations are enforced. Our first draft is almost done and we’ll be sharing it for feedback early next week.

Our internal framework to attack the problem

We’ve created a framework to help us ensure we’re being disciplined in how we define the problems and prioritize solutions.

Understanding exclusion

We need to fully understand the scope and depth of the problem. Using data insights and UX research we are attempting to answer questions like these:

  • What are the negative experiences people have had while using Stack Overflow?
  • What are the site functions that contribute to negative feelings, expectations, and/or experiences?
  • What are the behaviors that contribute to negative feelings, expectations, or experiences?

Learn from others’ inclusion efforts

By engaging and working with others who are actively tackling inclusion efforts (particularly other companies dealing with internet scale communities) we want to:

  • Learn, learn, learn. We aren’t the experts.
  • Investigate solutions from other online communities who had to address similar issues.

Solve for one, extend to many

We believe that by focusing on excluded people we can make Stack Overflow (and the entire Stack Exchange network of Q&A sites) better for everyone. We are committed to making changes in:

  • community standards
  • enforcement mechanisms
  • the software that runs the site.

Engage the broad community

We commit to an open dialog about our goals, understanding of the problem, ideas for research and solutions and progress on both Meta and our blog.

Next up

We will be providing regular updates on the work we are doing around inclusion. Soon, we’ll be sharing more details in separate posts on the Ask a Question Wizard, what we learned from our comment classifier, and our proposed new code of conduct. We’re on a long road here, but we’re excited about our progress so far. We hope you’ll stick with us for the rest of the ride.

Author

Joe Friend
Product Manager
Joe is the product manager on the Developer Affinity and Growth team at Stack Overflow. His work is focused on our core Q&A engine and the public Q&A communities. Based in Seattle, Joe's worked in the tech industry for several decades including a long stretch at Microsoft. He loves to travel and his family spent 6 years living in Indonesia.

Related Articles

Comments

  1. That is a great strategy, please keep rolling the Welcome Wagon!

  2. Thanks for this write up. I look forward to seeing the revision of the Be Nice policy; hopefully it will clarify a lot of the uncertainty around acceptable behavior or Stack Overflow’s direction moving forward.

  3. Great work! I’d also like to see more politeness baked in to the platform. Thanking people for answering your questions (both preemptively and after they answer) with something more personal than an anonymous up-vote / acceptance.

    1. Rowan Collins says:

      I’ve never really got why people go out of their way to *remove* comments like “thanks in advance” from questions. I get that excessive pleasantries decrease the signal-to-noise ratio, but removing them outright comes across as “how dare you be polite and personal! we only allow faceless people here!” which isn’t a very encouraging start to someone’s experience.

      1. The basic idea behind this is the ethos of being an encyclopedia or library of knowledge rather than a support forum.

        No one laments the absence of a personal touch in a printed encyclopedia or on Wikipedia.

        1. Joe Friend says:

          Pekka, the reality is an encyclopedia is different than Q&A. I think we have gone too far to remove elements that are normal expressions of our humanity. They are a critical part of human collaboration. Wikipedia hides all this on the “talk” pages. The main pages are bereft of them by design. Our design isn’t quite so clean. The collaborative part is all out there for everyone to see, thus seeing the normal social remnants of that collaboration is appropriate.

          1. Joe, this is why SO’s recent efforts are failing miserably. You are trying to make the site better without first understanding its goals. If you don’t understand the goals, you won’t make progress toward them, only toward some performance metrics of your own creation that take time and energy away from the real purpose of the site.

          2. Well said

          3. lankymart says:

            The reality is this isn’t a Q&A (that is what most people don’t understand). StackOverflow is meant to be a Knowledge Base, not Quora or Yahoo answers.

          4. Joe Friend says:

            It is both. It is a knowledge base using the Q&A format.

          5. And in a knowledge base, (to steal Pekka’s wording) no one laments the absence of a personal touch. Even if that knowledge base follows a Q&A format.

          6. Rowan Collins says:

            The comparison to Wikipedia is interesting. The separated “talk pages” on that site started as an innovation to make the main content look more “polished” than a traditional wiki (which absolutely, by design, mixes discussion with other content). Notably, however, they didn’t attempt to ban all discussion, and leave only the content pages; they created specific features to nurture collaboration. User talk pages are a further evolution of this: user-to-user messages, but publicly visible, to encourage an open community.

            So maybe the solution to the conflicting aims in this thread is to come up with similar mechanisms on SO: if we’re discouraging pleasantries in the main content areas, what tools can we provide for people to give and receive that feedback? I’m not sure what that would look like, but if people are set on keeping it out of questions, answers, and comments, the alternative is a site devoid of all humanity.

          7. Félix Gagnon-Grenier says:

            It’s great that you’re doing all you are doing, but meanwhile, you are really ignoring a sizeable portion of the site’s users, that really prefer posts with as much signal in it as possible.

            Your definition of human collaboration could not be the only acceptable one.

          8. Dave Tillman says:

            To me, the questions and answers are the “encyclopedia” part of the information, and the comments are the social interaction.

            I should be able to scan the questions and answers when I’m looking for an answer. This should not have a bunch of history or social interaction in it.

            The comments naturally capture the discussion about the question or answer. This naturally captures human interaction (even if it does not include pleasantries. Having “thank you” from a participant in the discussion in a comment doesn’t seem to me to be a lot of noise. If it comment section fills with “hundreds” of thank you from third-party readers (as I’ve seen on some other forums, it would become a problem.

            My experience browsing StackOverflow posts is that there is too much useful “answer” information that lands in comments and never migrates into an answer. Therefore, I’m drawn into reading all the comments to see what additional information I need.

          9. “No one laments the absence of a personal touch. Even if that knowledge base follows a Q&A format.”

            I sure do. I think a lot of users lament the absence of feeling like real human beings are interacting with each other. Otherwise these initiatives wouldn’t be in the pipeline. Please don’t speak as if you are the authority on all internet users who ever lived. This is a part of why sites feel exclusive; if my viewpoint isn’t represented, the site is not welcoming to me.

        2. Liam Hughes says:

          Does SO (the company) even want this anymore? My feeling is that the “build a resource” has been sacrificed on the alter of “get more people to use the site to generate advertising revenue”. SO seems somewhat quiet on this front, no one from SO has mentioned the “make the internet better” ethos in some time. I would like to see SO be explicit on their aims here. Do they still want to “make a resource”, etc.? Or do they just want to “helpz the peoplez”. I’m not saying either is right but at the moment the aims are unclear.

          1. Joe Friend says:

            Not at all. We definitely believe that Stack Overflow is a resource. It’s simply not the same kind of resource as Wikipedia. We wear our collaboration on our sleeve and as such need to be more open to exposing natural positive attributes of our human collaboration as well. I would also say that we are questioning whether “make a resource” and “helpz the peoplez” are in conflict, and if so how/when are they in conflict.

        3. Cássio Renan says:

          I agree with you, but this comparison with Wikipedia or an enciclopedia is not well thought. Wikipedia doesn’t display the name of the author right below the content (as SO does), which is why beginners try to be polite in SO, but not in Wikipedia. On Wikipedia talk pages (which are signed), saying “Thank You” is ok.

          1. Henrik Karlsson says:

            well said

  4. GiantCowFilms says:

    I am really looking forward to see the results of using machine learning to detect uncivil comments. I feel that there is a unique opportunity here to enhance public discourse by providing well intentioned users with automated feedback on their behavior, and automatically preventing trolls an users with bad intent from posting.

  5. Petter Friberg says:

    *”57 employees provided 13,742 ratings of “fine,” “unwelcoming or snarky,” or “abusive,” on random comments on SO”* Can we have those? https://meta.stackoverflow.com/questions/326494/can-i-have-a-dump-of-rude-abusive-comments

    1. Joe Friend says:

      Petter, next week we will post details on the initial findings and next steps of this effort. I’ll check with the team on whether we are planning to release the data set, but if we do it will likely be later after we’ve done more. This was just a first pass.

      1. Will we get to see the actual content of (some of) the comments? It won’t be super useful for us if we see just “n% were snarky”; we need to also see what some of those comments were that were labeled as snarky.

  6. “we are questioning whether “make a resource” and “helpz the peoplez” are in conflict”

    I would say that they are in conflict, and that SO did a lot of work early on to get serious users of the site to get fully on board with the “make a resource” approach — and that this is the reason for a lot of the later problems. I suspect that a solution will require stepping back further and thinking hard about that conflict; I can imagine a future in which all questions are deemed worthy of answering — but only certain questions end up in the “resource pool”.

    1. Joe Friend says:

      I disagree with your statement that they are inc conflict, but that’s probably just semantics.

      I love the direction of this thinking. “I can imagine a future in which all questions are deemed worthy of answering — but only certain questions end up in the ‘resource pool'”.

      1. bradley.dotnet says:

        The “second-class-question” has been discussed on meta quite a bit, with the usual conclusion that the proposed area would be a wasteland.

        Not sure SO wants to travel down that road, many (most?) contributers want their answers to help the public at large, not one person’s temporary issue.

      2. John Dvorak says:

        On Yahoo Answers, all questions are deemed worthy of answering. There is a reason nobody likes Yahoo Answers. This is the reason.

        1. Well, that and the fact there’s no quality control on answers from the community. You can’t edit someone else’s posts, they don’t ever get reviewed in queues, etc.

  7. Why didn’t Tim follow up to his own Post?

    1. Joe Friend says:

      What post?

  8. Nicol Bolas says:

    So, the SO staff are developing an AI to detect snarky comments. Since most snarky comments are caused by bad questions, wouldn’t it make more sense to develop an AI that can detect bad questions before they’re asked? Not only that, but filtering out poorly thought-out questions would reduce the amount of downvoting and other “unwelcoming” behavior. While simultaneously improving the lives of the people who curate the site.

    Yes, the ask question page improvements are nice. But stopping bad questions before they’re asked would solve basically 99% of the “welcoming” problems.

    1. Joe Friend says:

      Bad questions don’t justify bad behavior. And, did you read the part above about the Ask a Question wizard prototype? We are directly working on question quality as a part of this effort.

      1. The question wizard might help here – although, if we see fewer snarky comments in the future, it will be hard to attribute this to a causation. (In doubt, the restrictions for comments will simply be increased further … things quickly derail when trying to solve a problem where the cause is not really well understood – or simply not accepted for some odd “political” reasons…).

        And of course, the question wizard will only help the people who actually *try* to ask a *good* question. In many cases, the bad question ITSELF already is the “Bad Behavior®”…

  9. StackOverflowUser says:

    I doubt the question template is going to improve the question quality. I agree bad questions don’t justify bad behaviour, however, personally I prefer a snarky comment over a down vote anytime, because at least it tells me I am doing something wrong

    Regarding “thank you comments”, wouldn’t it be better to add functionality to send a private thank you message, rather than polluting StackOverflow? Questions and answers often remain useful for years, however, the endless “thank you” and “me too” would seriously reduce the values of other comments on questions.

    I am also concerned that you are solely focusing on making StackOverflow a more friendly place for people to ask questions. You should probably do more to get your users to sign in to SO in the first place. 95% of my colleagues use StackOverflow on a daily basis, but never login to the website.

    Your attempts are likely going to mean SO will be flooded with even more low quality questions and drive people that have the skills to answers those questions away. It will soon become a barren desert, rather than the useful resource it is today

  10. Nicol Bolas says:

    I never claimed that bad questions *justify* bad behavior. I said that bad questions *cause* bad behavior. The more bad questions are allowed to appear, the greater the chance of bad behavior in response to them. This is simply human nature. You can either fight against human nature or work within it.

    What is more important to you: punishing people or making the site better?

    And yes, I did see the “Ask a Question wizard prototype”; I mentioned that in my second sentence. It’s useful for dealing with a certain *kind* of bad question. But it’s not nearly as good as training an AI to weed out bad questions before they’re asked.

  11. Luciano F Castelfranchi says:

    “Yes, the ask question page improvements are nice. But stopping bad questions before they’re asked would solve basically 99% of the “welcoming” problems. ” Seriously? If nobody asked questions of course nobody would feel unwelcome. Maybe just put an end to all questions? This is why people feel unwelcome, they ask a question and it gets judged as a bad question, it is hit with negative responses. This is an attitude shared by more than one person. There are more problems with bad attitudes than bad questions. It seems like ever comment, every question and every answer comes with a risk. This is why I don’t like to get too involved. The aggravation does not justify the effort. Hopefully this will change.

    1. Well said!
      People with unwelcoming behaviour need to stop blaming others, it’s their own behaviour that’s the problem. There’s no reason not to be civil no matter the question.

  12. Mike Williamson says:

    How about an “opt in” mentor possibility for SO members already in “good” standing (sufficiently high points, and maybe sign a contract that explains what is expected of a mentor)?

    This seems the most community-driven approach, and that is what makes S.O. great. I could see something like, say:

    – Once you get 5000 points (or whatever level), you will be approached to become a mentor if you would like.

    – As a mentor, each time you help **your mentee** (the idea is that you would only have a handful of mentees, maybe 10 or whatever seems appropriate), you earn some points for doing that.

    – Since many (most??) of us would not have the time or maybe patience to do this, it would be entirely opt-in, but with more hands-on touch from S.O. staff to make sure the mentor is in fact reaching out.

    Personally, my first few years on SO I found frustrating because I only received 2 types of comments: (1) snark (often RTFM, when I didn’t know where the manual was), and (2) neutral. If I just once in a while had someone to talk to me nicely, it would have meant the world. (To be specific, I have a stats background and started on Cross-Validated, the stats equivalent, and not on SO. In **my** experience, that site has way more “you aren’t smart enough to understand what I’m explaining to you” snark on it than stackoverflow does, even to this day.

  13. Despite being in IT since the 90s, I find there are a number of things I find very offputting about SO which directly disincentivise me from participating more.

    The first one is that 90% of the time when I have a specific problem, and the answer exists on SO, when I look at it I find that some trumped up micro-Dictator has closed the question, using one of the umpty-tillion excuses that exist for doing so, and the only reason that the answer exists at all is because someone snuck it in under the radar in the small window of time before the question got locked.

    This is stupid and broken. (To be diplomatic and delicate)

    The second reason is related to the first. Let’s take the example of the version of Tomcat which comes bundled with Netbeans. There are some … peculiarities … associated with that particular ‘flavour’ of Tomcat, and therefore it is really damaging when questions related to that are killed off as ‘duplicates’ by people with a passing familiarity with Tomcat in general (but no knowledge whatsoever of this particular situation).

    The third reason is that there is obvious vote manipulation going on. E.g. the user with ~250k reputation who trolls through the site telling people they are idiots and they don’t know what they are talking about. Their slightest move is getting instantly upvoted – one suspects by bots – and helpful answers which give the correct solution (e.g. pointing out the line of code the error occurs on, and the solution) are left languishing – even if the original questioner says a bunch of nice things about how you solved their problem and marks your answer as the correct one (which yes, it does provide a rep boost which outweighs the voting, but it still is a bit demoralising to see their anti-social and anti-participation behaviour rewarded with ‘popularity’ points, even if it’s clear they’re just engaging in an onanistic ritual of patting themselves on the back with their own bots).

    Of course, all voting/karma systems struggle with this, from slashdot through digg and reddit. People see points, people want points, people will cheat to get points and will attack other people to try to prevent them getting points.

    ——–

    Anyway, yes there are umpty-tillion people asking for help with their homework. But as someone trying to find good quality troubleshooting information on detailed technical problems … that doesn’t bother me. The blatant homework questions don’t come up in the search, because they’re not real-world problems.

    Ergo, overly aggressive closing of questions has no beneficial use cases, only negative ones.

    1. Teemu Leisti says:

      I agree that this is a problem with Stack Exchange. I’ve seen many, many useful questions that have been closed as “not useful”. This could be summarized as the “soup nazi” problem, as is usefully analyzed in this blog post: https://www.embeddedrelated.com/showarticle/741.php

      The gist:

      ” Closing someone’s question sends a message that their work isn’t good enough, that they aren’t wanted here. From my point of view, even if only 1% of the close-votes are wrong, that’s too many. Even if a tiny percentage of people are harassed by Soup Nazis, it has a pervasive effect that just encourages hostility and banishment, without any accountability. It sends the message that it’s okay to continue the practice. …

      ” The Quality Nazis, like the Soup Nazi, want people to come into the soup shop of Stack Overflow, and abide by the rules, or No Soup For You! and your question gets closed. If it’s a blatantly bad plz send me teh codez question, that’s one thing. But there are a lot of questions in the gray area, and there’s something a little alarming to me about erring on the side of quality rather than giving people the benefit of the doubt. “

  14. I like where this is going and really like the idea of the comment app to thwart snarky comments. I also listened to the podcast episode with Jon Skeet talking about inclusion twice. It continues to sink in to me just how awkward it can be for a new developer (or just a new SO user) to ask a question and get help. I’m personally guilty of dropping in a “have you tried Googling it?” type comments at times as I’ve forgotten that I’ve been doing this (coding) for 25 years and what comes naturally to me now, didn’t always.

    FWIW – I’d be quite fine as a user if the system warned me when I’d entered a comment that might be taken offensively. I’d much rather revise my comment and help the community.

  15. Jamie Cantrell says:

    Hey Joe, Was there an update on the ML solution you mentioned? Would love to know how it’s working for you & any special insights you’ve gained.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.