Adios to Unfriendly Badges! Ahoy, Lifejacket and Lifeboat

Sometimes it’s helpful to look back at past decisions and reconsider them in a new light. As part of our ongoing efforts to make Stack Overflow more welcoming and inclusive, we’ve been looking at how our incentive systems, including reputation and badges intended to encourage positive contributions, may be building discouraging barriers instead.

In this new light, two badges stood out to us as different from the rest: Tumbleweed and Reversal. In this post, we share the reflection that led to our decision to retire the Tumbleweed and Reversal badges and introduce a pair of more encouraging badges: Lifejacket and Lifeboat.

Eradicating Tumbleweeds

The Tumbleweed badge is awarded to questions with zero score, no answers, no comments, and low views for a week. Recently we questioned if it was serving its purpose as a light-hearted consolation prize.

Back in 2009, it was originally intended to be “sort of fun” and also kinda rare. Unfortunately, we have anecdotal evidence (including my own experience) that Tumbleweeds are not fun and, on Stack Overflow, they are no longer all that rare either:

I included the Scholar badge for reference since it’s awarded the first time a user asks a question and accepts an answer. In other words, the system is working for about 200,000 users a year and failing for about 150,000. That’s sobering. What’s worse, the badge reminds people a week later that their question came up empty.

Still, it could be that the badge still serves as a consolation prize. Maybe it communicates “Dust yourself off and climb back on that horse.” I suggested to Dr. Julia Silge, our resident data scientist, that we might be able to show the badge is working by looking at all questions that received one (and only one) interaction over the last year:

InteractionNumber of Examples
Tumbleweed badge 152,422
Got a comment 30,017
Got an answer 17,733
Got a downvote 4931
Got an upvote 407
As you can see, getting an upvote and only an upvote is far less common than getting a Tumbleweed. Upvoted (and downvoted) questions are likely to either get a comment or an answer as well. Obviously getting a downvote discourages an asker and other interactions encourages them. If the Tumbleweed badge is successful as a consolation prize, we’d see people who get it asking more questions on the whole. The data does not support that hypothesis; instead, people who get a Tumbleweed badge are much less likely to ask a question ever again.
So the Tumbleweed badge seems to be a worse consolation prize than if the system just gave out a downvote. Now you might wonder if Tumbleweed askers just have a knack for asking uninteresting questions. The data doesn’t bear out that hypothesis either.
Askers who get one upvote tend to ask better subsequent questions and those who get a downvote tend to do worse. But people who “earn” a Tumbleweed do about as well as people who get an answer or a comment. In summary, the Tumbleweed needlessly discourages askers who might very well be asking good and interesting questions.

After looking at the data, we decided to retire the Tumbleweed badge on Stack Overflow and the Stack Exchange network. That means everyone keeps the badges they have, but we won’t be awarding any new ones.


Lifejacket and Lifeboat to the rescue

Once we started thinking about badges and what purpose they serve, we considered behavior we’d like to recognize. As it happens, we test badge ideas in our annual Winter Bash event in which people pursue temporary hats to decorate their avatar. Like badges, hats can only be earned by doing something positive on the site and are triggered automatically.  One hat trigger stuck out as really interesting: Red Baron. We even reused the idea a few years later and called it Maverick. The requirement was: “answer +5 a -3 question that goes to +3”. In other words, find a misunderstood question that has promise and answer it. But that’s not always enough. Sometimes the people earning the hat also edited the question in order to help it get upvotes. (Read about how one of our users, Air, went above and beyond to get the hat.)

Instead of one badge, we decided to introduce two related badges: Lifejacket and Lifeboat. These badges reward reversing the score of a negative question by answering it in a way that sheds new, positive light on the question, raising its score. It’s an answer so good it makes the question look good by association! You can think of it as like someone finding a diamond in the rough and putting a little bit of work into showing the world its hidden value. Conceptually, it’s similar to Reversal, but with a more encouraging twist in that the question itself goes from negative to positive rather than staying negative.
NameLifejacket
TriggerA user posted an answer to a question with a score of <= -2 at the time
The answer has a score of >=5
The question goes on to get a score of >=2
Don’t count the user’s votes on the question
TierSilver
Short descriptionPosted answer of +5 to question with a score of -2 that then increases to +2.
CategoryAnswer Badges
Can it be awarded multiple times?Yes
Is it retroactive?Yes


The gold version of the new badge pair is identical in concept to the silver tier; it’s just a lot harder to obtain.

NameLifeboat
TriggerA user posted an answer to a question with a score of <= -3 at the time
The answer has a score of >=20
The question goes on to get a score of >=3
Don’t count the user’s votes on the question
TierGold
Short descriptionPosted answer with a score of +20 to a question with a score of -3 that then scored +3.
CategoryAnswer Badges
Can it be awarded multiple times?Yes
Is it retroactive?Yes


We’re retroactively awarding these badges so if you’ve rescued any questions in the past, that achievement will be recognized within about 24 hours. Again, this is for all sites on the network.

Reversing Reversal

Replacing Tumbleweed then brought us to reflect on the Reversal badge. The criteria for the Lifejacket and Lifeboat badges are tweaks of Ben Voigt’s suggestion for replacing Reversal from 2014. (Better late than never, right?) Reversal has nearly the opposite problem from Tumbleweed: it’s only been earned 304 times and takes real skill to pull off. Getting a score of 20 is not easy (there’s a separate badge for getting 25) , but it’s more difficult when the question has been downvoted to -5.

Reversal rewards exceptional answers, but often the question gets deleted (along with the answer) after the badge is awarded:

Reversal badges on Stack OverflowQuestion result
118Deleted
92Still negative score
66Zero or positive score


For a badge named “Reversal”, it’s odd that the fortunes of just 22% of those questions were turned around. And 39% of those great answers are not even available anymore. One quirk of the Reversal badge is it’s more common on Meta Stack Overflow. A typical scenario is someone suggesting an unpopular feature in a question (which gets downvoted) followed by an explanation of why the feature is a bad idea (which gets upvoted). Unfortunately, the outcome for those meta questions is much worse:

Reversal badges on Meta SOQuestion result
963Deleted
119Still negative score
31Zero or positive score


It’s an interesting badge to earn, but it’s not at all positive for the asker and I’m not sure it always encourages productive behavior. Certainly the new badges will result in all-around better content than most of the answers that trigger Reversal. As a result, we’re retiring Reversal everywhere. You’ll keep your legacy badges, of course, and if the question has been upvoted enough, earn Lifejacket and Lifeboat too!

Badge retirement home

With two badges going out of service, the list of badges that are no longer awarded has grown:

One of the purposes of badges is to let users show off their achievements. So we don’t want to hide retired badges. But another reason for badges is to suggest things for people to try out. Since none of these badges serve that second purpose, we’re going to place them in a “Retired Badges” section on the Badges Help Center page

You’ll also see a note at the top of each retired badge’s page.

Thank you!

I’m excited to announce these changes because they represent a collaboration between many people over several years:

  • Julia Silge’s analysis demonstrated that the Tumbleweed badge wasn’t serving its purpose.
  • Meg Risdal is our Product Manager who led us to make these changes.
  • Jane Willborn wrote the code.
  • Nick Craver made sure the Lifejacket and Lifeboat triggers don’t break the site.
  • Ben Voigt suggested the trigger we used first for Winter Bash hats and now for two new badges.
  • A bunch of people tested those triggers on our sites and prodded us to keep considering something like these new badges. We also got a bunch of feedback about Reversal and especially Tumbleweed that helped us understand how they worked in practice. We’ve got an amazing community here!
Lead image credit“Whitstable Lifeboat. The Oxford Town and Gown.” by andrew simon uk – licensed under CC BY-NC-SA 2.0 

Related Articles

Comments

  1. I support the change in the badges that you’ve outlined here. However, the trigger for the Tumbleweed badge still seems useful. Why not continue to identify questions that have no votes for a week, low views, etc., and have the system automatically offer the author some tips on improving the question? Questions probably become tumbleweeds most often because they’re difficult to understand, don’t seem relevant, and/or aren’t tagged in a way that’s likely to make them visible to a broad enough audience, and new authors are particularly unlikely to realize those problems or to know how to fix them. A question that gets enough attention will at least get put on hold, usually with some indication of how it could be improved.

    1. It would also be useful to continue to be able to FIND questions like this – a list of unanswered questions that haven’t gotten much attention. I just now went to the tumbleweed badge, scrolled through a handful of the recent ones, and found a question I could answer. And then I answered it. Seems like a useful feature that perhaps should even be *promoted*.

    2. On that thought, why not make it a Review Queue item? Questions that meet the Tumbleweed criteria could get put into a queue for someone to come by and either help out or at least give a bit of advice on how to improve the question

  2. This is an improvement, but you really should hide the tumbleweed badge on all the profiles where you inflicted it. i don’t think anybody will miss it.

    Next how about getting rid of Peer Pressure?

    1. Here! Here!

    2. Hey, some users (myself included) are happy with their “tumbleweed” badges. Maybe adding the option for the user to delete theirs if they don’t like it would be a solution which makes everyone happy?

  3. The conclusion that `Tumbleweed` badges are discouraging might be correct, but isn’t quite disentangled from the idea that non-engagement-on-the-question itself (and not the badge recognizing that) is what’s discouraging. SO could have A/B-tested that, by splitting users into Tumbleweed-eligible and no-Tumbleweed groups!

  4. Just for the record: I absolutely loathe and hate those special badges.

    Maybe it is just me, but I simply don’t believe that a badge like “lifeboat” can cause much (any?) positive effect. Thing is: you earn those things by pure coincidence.

    Depending on the day of the week, the phase of the moon, more importantly, the tags on the question … mediocre questions/answers might receive multiple upvotes, and some great content just sits there with 0 to 1 upvote, if lucky an accept.

    So, chances for me to ever get to such a lifeboat badge: zero zip nada niente.
    But hey, somebody else earned it. Because … luck.

    Honestly, I find that extremely demotivating. I spent countless hours reviewing thousands of posts for steward badges, and somebody else … is just lucky but shines as golden or even more.

  5. All this is very interesting. Thank you for sharing not only the system but the deeper fact-based analysis used to justify the change.

    Now, I wish we could do something to move StackOverflow pages up on Google. I tend to run a search then tend to favor StackOverflow pages over all other sources.

    1. I just add stackoverflow to the google search (or stackexchange if I think what I’m looking for is on the network). It works pretty well.

  6. I love my tumbleweed. I never learned how to install a SQL Developer plugin, but I did get a nifty badge!

    1. @Mark Harrison, have you tried downloading the extension, and then clicking the “Install from local file(s)” radio button in the lower left of the “Check for Updates” window?

  7. With the analysis that “people who get a Tumbleweed badge are much less likely to ask a question ever again”, did you consider whether the number of positively received questions prior to the Tumbleweed is a relevant factor? I got Tumbleweed on SO in 2013, but I’d already been active on the site for a couple of years and had some questions with accepted answers.

  8. @Caleb

    A very worthwhile idea, but I confess I’m sceptic that it will work very well. The tumbleweed phenomenon is sadly *very* prevalent, at least on some SE sites. Of the half-dozen questions I’ve asked on SuperUser, for example, only one has received an answer at all, and only two have received a comment. I’ve gotten one or two upvotes on five of them, so no actual Tumbleweed badges, but apart from the one answer, I haven’t gotten anything that actually helped solve the problem.

    The questions were all carefully written (and edited) so as to be clearly structured, easy to understand, well-tagged, with a clearly delimited problem… but they’ve received virtually no attention at all, even after several clarifying edits (which would also have bumped them to the top of the Top Questions page). I’ve tried setting a bounty twice, even, but both times it just expired, leaving me now with a lower rep than when I joined SuperUser, and a certain aversion to wasting my time asking on there because nothing useful tends to come from it.

  9. Strangely enough, I just got a ‘tumbleweed’ badge, and was happy about it! Made me think of moving the question to MO. However, I might be an exception… I was initially hoping somehow to get the tubleweed badge, but not actively trying.

    Perhaps, to further encourage people, you could add an option for a user to delete this badge if they already have it? (I’m keeping mine, though!)

  10. Patrick Lewis says:

    This is potentially surprising, but I am rarely feeling lighthearted when I am on StackOverflow.

    Here’s my story. I came on just today after many years of lurking and posted a question, part of which I’ve been researching for a few months. Rather than dump the question out in detail, I asked the simplest form of what I was trying to figure out. I’m getting tumbleweeds. It’s an obscure question and that seems reasonable. So, I figured I should answer some questions to get traffic or up my karma or whatever. This was viewed by people on the site as an open invitation to be jerks to me. So I deleted the answer they found offensive and in investigating how to simply delete my account (I wouldn’t want to be a part of a place where people are jerks and it’s enforced by the system, plus, I truly have better things to do with my day) I found the Peer Pressure badge waiting for me.

    This was not inviting in the least, but I truly don’t care IF I could get assistance with my question.

    Instead of being “inviting” try to be “relevant”. Maybe Tumbleweed questions should be in a priority queue with incentive to get answers. There is nothing you can do about people being jerks, but you CAN try to not have questions rot and encourage answers.

    1. I agree that questions fitting the Tumbleweed criteria seem like good candidates for some kind of priority queue, or at least a special filter.

  11. Joseph Marks says:

    I had a tumbleweed badges.

    Did not know I had it till this post caused me to look.

    Not particularly offended.

    Like the idea of badges in general, they bring back my son’s ‘Royal Rangers’ meetings as I never got one in Cub Scouts … LOL

  12. That “Retired badges” section could use some links to pages with the reasons for retiring them, just like you did in this blog post.

  13. That “Retired badges” section could use some links to pages with the reasons for retiring them, just like you did in this blog post.

    Btw, also “Precognitive” and “Beta” should be placed in that section, since they cannot be earned any more.

  14. This is probably a good thing overall, but anecdotally, I have to say that the Tumbleweed badge did serve the intended purpose for me. It was kind of nice to know that the system was aware that no one was interested in my question.

  15. Vadim Ovchinnikov says:

    I think just removing the “Tumbleweed” badge won’t solve completely the main problem: a question that needed answer has been ignored. We need some positive replacement for “Tumbleweed” to encourage other user to answer these unheeded questions. Added suggestion about this on main Meta: https://meta.stackexchange.com/q/329780/346987.

  16. I got a Tumbleweed, and was very happy with it. I recall really enjoying this consolidation after my initial frustration of not getting my question answered, (especially since my Q:A ratio is less than 0.01, so I thought I deserved an answer).

    Also, I don’t agree with some of the conclusions here. The fact that Tumbleweed users don’t ask any more questions, might just be because they ask “boring” as you call it, or just very specific questions, and those users wouldn’t come back either without the badge.

    Also, maybe those questions score the same in terms of votes, but that could as well be because they are about specific topics that few people know or care about. If a question looks good, but not interesting to me, or outside of my area of expertise, I’m not gonna vote on it.

    So, my suggestion: Keep the logic for the Tumbleweed badge without visually rewarding it to new users, or even do it in an A/B test form, and check whether this badge is really chasing people away. If not, can I please keep it?

  17. I am afraid that the problem isn’t in badges. As you said, back in 2009, the Tumbleweed badge was relatively rare – and so was the Scholar. The count of Scholar badges increased rapidly over time, which means that Stackoverflow gained many new users. At same time there were more questions without any answers asked. Yes, you are guessing it right: The new users simply posted more questions than answers. And this problem can’t be solved by any badges.

  18. Elise van Looij says:

    Can’t say I’ve ever paid attention to my badges. Just checked: no Tumbleweeds but two instances of Necromancer! That’s a lot more disturbing than a few weeds, I’d say. Was going to skip evening mass, but I think I’d better hurry.

  19. I have been in Raspberry Pi Stack Exchange for a year or so. So far I have posted about 180 answers but only 1 question. A year ago, I was happy and encouraged when getting up votes, and sad for down votes. But after getting too many up votes (I am top 0.5% this quarter), I began to have the opposite reaction, ie, happy when getting down votes. I know this might be abnormal, so I thought deep to find my reasons. One reason that applies to me is that, in real life, I luckily got many “up votes” which I think I don’t deserve, and I saw too many unlucky people unfairly get too many “down votes”. In a sense, I have sort of “up vote guilt”, similar to “survival guilt”. After reading this article, I am glad I understand more about the badges system. A year ago I read that Stack Overflow is based on the “electronic games” psychology that so many gamers got addicted. After watching how many “reputation hungry” users here try to get more “reputation points”, I got sadly reminded of my unlucky friends addicted to electronic games (or even to drugs and gambling). One other reason I that I am happy to get down votes is that in real life I don’t get that many constructive (or destructive) criticisms. So down votes help me to improve. I know I am already good at many things, but I hope to become excellent, ideally along with everybody. Anyway, one result of reading this article or analysis is that I would become more generous in giving up votes. Many thanks to Jon Ericson. Cheers.

  20. You know what these badges were supposed to mean, but for the users they might mean something completely different. I remembered my first tumbleweed badge. Was thinking at that moment: “Hmm that question might have been a bit too niche. Maybe it would help to make sure there is more general context, or maybe some questions are not really fit for SO.” I just discovered a little bit of how SO works.
    Tumbleweed is kind of a funny thing. Nothing negative came to my mind regarding my experience with stackoverflow because of that badge.

  21. I feel that the tumbleweed badge was “spot on”, with a “10 gallon hat” as it’s counter. Please reinstate it with my suggested counter — the 10 gallon hat. 🙂

    This way, one can earn the 10 gallon hat for improving/answering the tumbleweed question.

  22. Haha. Whenever I got Tumbleweed, I felt kindof proud, like I’d asked a question difficult enough to stump the whole internet.

  23. staypuftman says:

    The badge switcharoo is fine but you’re missing the elephant in the room – Stack Overflow’s discovery mechanisms are inadequate for a site of this size. The tagging system helps but ‘Tumbleweed’ like questions from new users should be brought to the attention of experienced S.O. users who might go out of their way to help the newbies and bring them into the fold.

    How am I as a regular contributor supposed to know about these situations?

  24. Discouraging of Tumbleweed is a right approach

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.