The quality of the Q&A on Stack Overflow continues to outshine any other on the Internet - thanks to the awesome community. Like any community, unspoken rules eventually become expectations. In the previous post in this series, Joel talks about how the community developed its own set of rules and norms that new recruits simply don’t know about. When a new comer walks into the group and puts her hand up for a high-five and gets chastised by the group because they don’t give high-fives, she walks away embarrassed with head hanging low. That's unfortunate. This isn't a new concern of course - almost four years ago, one day after Stack Overflow left private beta, Chris Upchurch wrote one of the most famous pleas for kindness in response to attacks he observed on new users in Could we please be a bit nicer to the noobs? A year and a half later, we saw the opposite opportunity for self-reflection when Satoru.Logic, then a member for just over 4 months, asked Why are Stack Overflow people nice? - which was followed up a year later by veteran member dmckee with Are Stack Overflow people still nice? There've been dozens of discussions along these lines over the years, reflecting an increasing perception of our Jekyll & Hyde nature. But always lacking was anything more than anecdotal evidence. And as Stack Overflow grew, it became easier and easier to cherry-pick examples that showed the community as either friendly or fierce. So we decided to gather some objective data:
Comment Friendliness: The Science Hammer
To investigate, we sampled 7,000 comments written on questions on SO and collected 20 independent ratings of attitude for each and every sampled comment (ratings obtained via experienced raters on Mechanical Turk). Comments were randomly selected over the past 3 years. Then we calculated “friendliness scores” for comments based on all 20 ratings. The first thing we found is that comments on Stack Overflow are, in fact, getting friendlier. As we see in the chart, friendliness ratings are generally positive and continue to trend that way. Since May 2011 at least 75% of all comments sampled are rated positively. Statistical modeling of the data supports these observations: comments now are significantly friendlier than they were three years ago. What about the unfriendly portion? We’ll get to that later. The next thing we looked into is friendliness differences between tags. According to our sample, comments tagged in ‘C’ tend to be rated as less friendly compared to others. And subtly, ‘Android’ is friendliest. However, the data only reflects minor differences so we should interpret this trend with a grain of Kosher salt…nevertheless, this does address another long-standing question: are programmers using certain languages or technologies more welcoming of newbie questions? We found that comments on first posts are significantly less friendly compared to the rest, regardless of time period. Though the total percentage of nice comments is increasing (awesome!), the few unfriendly cases can unfortunately drag down a new member’s experience. Experience has taught us that newcomers tend to really remember their first interactions within a community; in this case the small percentage of rude comments carry disproportionately more weight in the memory of the newcomer and affect their impressions of the community.
Conclusions
So now that we have some hard data, the question arises: is this a problem, and is it worth addressing? If the majority of comments are friendly and getting friendlier, why risk rocking the boat? The short answer is simply that 3/4 "nice" is still a long way from "Total civility [...] one hundred percent of the time." It doesn't take an overwhelming amount of rudeness to create that impression in casual readers, and becoming complacent about our "niceness" is the quickest way to become blind to its absence. We'll delve into this further in our next installment, but for now I'll leave you with a question from dmckee:
Is there something else we can do to encourage our big city to keep the small-town feel we grew up with?
If you're curious on how exactly we collected and analyzed this data, feel free to download the full summary. Look forward to the next post in the series discussing mechanisms and community solutions! And don't forget; at Stack HQ, we love you all.
Update: Comment examples on MSO