[Ed. note: While we take some time to rest up over the holidays and prepare for next year, we are re-publishing our top ten posts for the year. Please enjoy our favorite work this year and we’ll see you in 2022.]
They say there’s a kernel of truth behind every joke. In the case of our recent April Fools gag, it might be more like an entire cob, perhaps a bushel of truth. We wanted to embrace a classic Stack Overflow meme and tweak one of our core principles. Our company was inspired by the founders frustration with websites that kept answers to coding questions behind paywalls. What would the world look like if we suddenly decided to monetize the act of copying code from Stack Overflow?
Ok, jokes over, hope everyone had a good laugh and no one got too freaked out. But wait, there’s more. Once we set up a system to react every time someone typed Command+C, we realized there was also an opportunity to learn about how people use our site. We were able to catalog every copy command made on Stack Overflow over the course of two weeks, and here’s what we found.
You are not alone
One out of every four users who visits a Stack Overflow question copies something within five minutes of hitting the page. That adds up to 40,623,987 copies across 7,305,042 posts and comments between March 26th and April 9th. People copy from answers about ten times as often as they do from questions and about 35 times as often as they do from comments. People copy from code blocks more than ten times as often as they do from the surrounding text, and surprisingly, we see more copies being made on questions without accepted answers than we do on questions which are accepted.
So, if you’ve ever felt bad about copying code from our site instead of writing it from scratch, forgive yourself! Why recreate the wheel when someone else has done the hard work? We call this knowledge reuse - you’re reusing what others have already learned, created, and proven. Knowledge reuse isn't a bad thing - it helps you learn, get working code faster, and reduces your frustration. Our whole site runs on knowledge reuse - it’s the altruistic mentorship that makes Stack Overflow such a powerful community.
You can stand on the shoulders of giants and use their prior lessons learned to build new things of value. You should still follow some basic best practices to prevent bugs or safety issues from sneaking into your code when copying, so make sure you educate yourself before grabbing and pasting. And of course, be aware that some code requires a certain license to use. Beyond that, we encourage everyone to share in the benefits of what the community has created.
That’s the high level TL;DR, but for folks who want a deep dive into all the things we learned while studying the copy data, please read on for some marvelous insights and charts from David Gibson, a data analyst on our product marketing team. If you want to hear about how we built the software modal and physical keyboard behind our April Fools joke, check out the podcast below.
As someone who has been unapologetically copying from Stack Overflow for years, I was not surprised to see the millions of copy events rolling in. What did surprise me was the number of questions we could finally answer. How many people really are copying from Stack Overflow? Are people just copying code? Are people more likely to copy the accepted answer?
To add some direction to the analysis, the team and I came up with a list of questions that we wanted to answer. What started as a joke has snowballed into a worthwhile exploration, producing new insights and sparking many internal conversations about how we can continue to innovate our public platform and bring more value to Stack Overflow for Teams.
The data
Using our homegrown web tracking tool, we created custom events to capture when a user copied from the site. With these events we were able to capture many different attributes; tags, question answer or comment, code block or plain text, copier reputation and post score, region, and if the post was accepted or not. We pretty much captured everything except the actual text being copied.
We collected data for two full weeks, from March 26th 2021 to April 9th 2021. The following analysis is based on the behavior during that time.
Questions
Ben already mentioned some of the high-level stats that quickly proved what people had long joked about: everyone is copying from Stack Overflow. We also quickly realized that the overall copy behavior closely followed what we already knew about our site traffic. Most copies occurred during the work week and during working hours. Our largest geographies make up the majority of copies; Asia 33%, Europe 30%, and North America 26%. Finally, 86% of all copies came from anonymous users, aka users with 0 rep.
Things started to become more interesting when we asked more detailed questions about who was copying and what they were copying.
Are higher rep users copying more?
To start, we wanted to see if our higher reputation users are copying more.
We can see that the majority of copies are coming from users with 0 reputation. These are our anonymous users because you immediately get 1 rep by creating an account. It is possible that some of these copies are from users with an account but are not logged in. Unfortunately, there is not a way for us to test this theory.
Since the majority of the users on our platform have a lower rep, let’s remove the groupings to see if we can normalize our data. By looking at Count of Copies Per User instead of Total Copies, we can see the average number of copies a user makes by their reputation.
When looking at this visualization, it appears that as Reputation increases, the Count of Copies Per User decreases. So the higher a user’s reputation, the less often they are copying. This relationship is present but is not very strong, so I am not confident in saying either higher or lower reputation users copy more. Developers who are learning often have a lower reputation and are looking for things that can accelerate their learning and get them started quickly. As developers build their expertise, they also build their reputation, and they focus on more precise challenges, things that may not be possible to copy from Stack Overflow.
Are accepted posts copied more?
When we think of an accepted answer, we may think it is the best one, and infer it is copied much more than non-accepted answers. Looking at the data, however, we find 52.4% of copies come from answers that are not accepted. But on average, accepted answers get seven copies per unique post while non-accepted answers get five copies per unique post. So more copies come from non-accepted answers, but there is higher knowledge reuse from accepted answers. At Stack Overflow, we define knowledge reuse as reusing what others have already learned, created, and proved.
It is worth noting that a question may not even have an accepted answer. Take this answer: it has almost 4,984 up-votes and was copied 7,943 unique times during our study, but is not accepted. Actually, none of the answers have been accepted. It could be because the question poster has not been seen since 2010, but also many of the other answers are valid.
Are higher scored posts copied more?
So if accepted answers are not copied more, then answers with a higher score answers must be copied more, right? Let’s find out!
We see for Answers it seems to be pretty evenly split across our defined score groupings from 1 to 1000. As for questions, the majority of copies are from posts with 1-5 points. I suspect that is because users are copying the question to reproduce it and eventually post an answer.
Similar to when looking at user reputation, the majority of posts on the site have a lower score. To normalize this, let’s look at the copies per post.
We can plainly see that as a post increases in Post Score so does the Copies Per Post. This makes sense because as a post increases in score it is more likely that the knowledge is being reused by our community.
Do people copy downvoted answers?
But what about those blue dots with a negative score? Why would anyone copy down-voted answers? Well, we never want to judge a book by its cover.
Take a look at this answer. It was our most copied down-voted answer with a score of -2 and a total of 288 copies. Looking closer, it appears to be a more concise version of the accepted answer above it that has a score of 29 and had a total of 493 copies. Although our negative score post did not have more copies, it is the perfect example of a “too long didn’t read” post.
What are the most copied tags?
Now for the question I was most excited to answer: what tags are being copied the most? Unfortunately, due to the scale of the data and available resources, I was unable to parse out nested tags. For example, the html tag will not include posts within the |html|css| tag grouping.
Top ten tags copied
Not to my surprise, the tags receiving the most copies are some of the most popular and active tags on Stack Overflow. The one thing that jumped out to me is python appears in four of the top tag groupings. Three of them are data analytics specific tag groups; |python|pandas|, |python|pandas|dataframe| and |python|matplotlib|. As a data nerd myself I love to see more people learning these tools.
Top ten tags with most copies per post
In addition to looking at the tags with the most copies, I wanted to see what tags have the highest copies per post. Filtering for tags with at least ten unique posts, we can plainly see as tags become more specific, they receive more Copies Per Post.
What are the most copied posts?
Now to answer the question I am sure many of you are interested in. What post received the most copies?
Answer with code block
With a post score of 3,497 and 11,829 copies, I am happy to announce that How to iterate over rows in a DataFrame in Pandas received the most copies. Answered in 2013, this question continues to help thousands of people each week.
Answer plain text
As for the most copied answer with plain text, we have TypeError: this.getOptions is not a function [closed] with a post score of 218 and 1,570 total copies. Although we were unable to confirm this I suspect that the `sass-loader@10.1.1` is being copied.
Question code block
And the most copied question with a post score of 2,147 and 3,665 copies, we have How to create an HTML button that acts like a link?
Question plain text
Finally, the most copied question with plain text with a post score of 322 and 261 copies, we have Updates were rejected because the tip of your current branch is behind its remote counterpart. This one is a little tricky because there are a handful of git commands not in code blocks that could easily be the copied part of the question. But as we are not capturing the actually copied text, we cannot confirm this.
Comment
It’s important that answers are not everything on Stack Overflow. Sometimes all you need is one useful comment. Here are the most copied comments!
https://stackoverflow.com/questions/332289/#comment47852929 - Comment score: 938, Copies: 4,924
https://stackoverflow.com/questions/41182205/#comment71717506 - Comment score: 5, Copies: 492
The first comment is our most copied comment across the site, and the second comment is our “unsung hero” as it only has a post score of five but was our sixth most copied comment.
UPDATE: There has been a lot of interest in purchasing a real life version of our prank. The good news is we anticipated this might happen and we’ve been working on something along these lines. Stay tuned for more!
TRANSCRIPT
Kyle Pollard As you grow a lot of it's just understanding from Stack Overflow and then kind of re implementing those concepts on your own in your own code. So not necessarily mindlessly copying, and then, you know, YOLO, let's see what happens, but kind of understanding the post and getting it in there. So, you know, in a sense, like, emotionally, you're still copying from Stack Overflow, but maybe not literally hitting Ctrl C.
[INTRO MUSIC]
Ben Popper Hello, everybody. Welcome to the Stack Overflow Podcast, a place to talk about all things software, programming, technology and life. Hi, I'm Ben Popper, Director of Content here at Stack Overflow. And today I'm joined by two excellent individuals, Cassidy Williams and Kyle Pollard, who helped me and a big team of people to put together our annual Stack Overflow April Fool's gag. Welcome to the show y'all.
BP Yeah, thanks Ben!
Cassidy Williams Thanks for having me.
BP So, I have not personally participated in a full April Fool's joke at Stack Overflow. It's a long running venerable and honorable tradition. Last year, we were getting ready to do ultra dark mode. [Cassidy laughs] And then we went into a pandemic, which is its own form of ultra dark mode. So we said no jokes this year. But people were excited about dark mode. Anyway, luckily, we had a new feature. But this year, we set out to recreate a much beloved Stack Overflow meme, the only keyboard you'll ever need, Stack Overflow copy and paste. And, Kyle, it was your job to build the little modal that popped up when people went to copy. Can you tell us a little bit about sort of like the genesis of this project and how you built it?
KP Yes, I am the one to blame for your blood pressure going up, the heart attacks, the fear that you're losing your job, and you'd have to go into another career path. Yeah. How did the genesis of this, we can go all the way to the start. We did an April Fool's workshop where we hashed out a bunch of different ideas of what we could do for an April Fool's prank. And we came up with the the copy paste paywall as the the winner for the year. So the kind of goal with this is that it would just be a one developer kind of job, something that I can do, I just started a few months ago. So this is kind of my first public project. Yeah, it was a relatively small scope in terms of the technical implementation, just got to track when a copy happened on the website, showing up that notice at the bottom right, that tracks how much your copy budget is left. And showing up that big, that big modal in the center of your screen advertising the best keyboard in the world, the key by Stack Overflow. [Ben laughs]
BPSo yeah, is there anything like that you think is interesting about the sort of the technical implementation here? Like, for people who don't know, what does it take to trigger, you know, the ability to sort of like, know what people when people are accessing that command on our site?
KP Thankfully, just like when you click on a button, it fires an onclick event, there's actually a copy event that fires in your browser every time you copy. So I just got to add an event listener on a copy event. And I got to read all the nitty gritty details of you know, where you've copied on the page. And if it was in a code block or not, and, and get to do some cool analytics with that.
BP And actually fires like 1000s of like, when you're holding CTRL C that you've actually copied, like 100 times or somethimg.
KP Oh, yeah, that was a bug that we had to squash pretty early on. If you, if you were, say copying something, and you you held Ctrl C, and you were thinking on it for a while, it turns out in the background, it would fire hundreds and hundreds of events and ruin our data. So I had to be a little more generous with how often you could copy and paste. If you if you copy the same element twice, we don't reduce your budget, you can you can get extra copies for free. [Kyle laughs]
BP Right, Cassidy as a seasoned developer, how many times a day would you guess people copy something from Stack Overflow?
CW Like in general, or individuals?
BP How many times do you think an individual does it? And then let's try to guess, because now we know, the total number of copies that happened per day?
CW Oh, man, I would guess an individual developer per day, depending on the bugs of the day, I'd say at least two or three times a day.
BP Tell me how many times a day people are copying on Stack Overflow. And roughly how many times that is like compared to the number of times people actually just visit a question.
KP Well, I'm not a data analyst. So I've leaned on smarter people on my team to try to figure this out for me. But you know, when I took a first glance at it, it looked like there was, you know, 10 million views of questions in a day, one in eight of those copied, and then someone looked a little deeper and it turns out that one in four people copy within five minutes of visiting a Stack Overflow question, which is ridiculous. It's mind blowing. Everybody's copying! Nobody's talking about it! [Ben laughs]
BP Yeah, I that it was pretty staggering. Yeah. To realize like how deep the truth runs on that particular joke, but yeah, so there's something you know, there's like 10, 12 million visitors a day and like one to 2 million copies a day, which is really wild. Yeah, I guess even one out of four. Which which brings us I guess to the bigger question of like, to what degree is this useful and saving people time? And just a net good? And to what degree is this lazy and perhaps a little bit risky if you don't do it safely?
CW I think like the whole core of Stack Overflow is that it's, it's helpful. Because you can try to do something yourself. But if you do something yourself, you're kind of reinventing the wheel. If the answer is already out there, if you're already looking it up, and you can get your answer and just move forward. It's a productivity add for sure.
BP I guess. Yeah, especially if you're if you're doing this all the time. I mean, for someone like me who's like, occasionally tried to dabble. Sometimes it feels like I can't just go in and like, get what I need and copy and paste it. And I'm not like really internalizing it or like learning it for the next time. But I think yeah, if especially if this is like your day in, day out job, why reinvent the wheel when like somebody has built the tool already for you? I don't know, Kyle, before you came to Stack Overflow, when you were working as the greatest webmaster in the Great White North, did you, how often would you say you copied from Stack Overflow?
KP You know, I think early in my career, I think copying was a necessity. But I think as as you grow a lot of it's just understanding from Stack Overflow, and then kind of re-implementing those concepts on your own in your own code. So not necessarily mindlessly copying. And then, you know, YOLO, let's see what happens, but kind of understanding the post and, and really get in there. So you know, in a sense, like emotionally, you're still copying from Stack Overflow, but maybe not literally hitting Ctrl C.
BP And then of course, after we posted the prank, and it started happening, there was there was some great threads on Twitter and on Reddit where people were trying to hack their way around the paywall
KP I didn't get that because our prank must have incited a lot of fear. Because I don't actually stop anyone from copying from Stack Overflow. Like you can still copy, we're on the honor system past, you know, your three free copies. But people were like, you know, I've made an ad blocker to remove it. And like there's a don't show button if you really don't want to see it. Like they were making scripts to change the budget. They were, someone suggested taking screenshots of the page and then running it through like an OCR system. So you could like bypass the copy and copy of it. [Kyle laughs] So people people need the copy. They're looking for ways around it.
BP Yeah, I'd like to give an award to the person who created the most complex and time consuming workaround.
KP It's like a Rube Goldberg machine at this point. And so then there there was sort of like a great second half to this. The software was the first part, but hardware was the second part. Cassidy when we first had you on the podcast, we discussed this, but you have a great love of mechanical keyboards and custom keyboards and have quite a collection yourself. So we reached out to you to see if we could actually get this thing made and take some photos. And you connected us with a cool company called Drop. Can you tell us a little about what Drop is? And how you got to know them? I think you worked with them in the past, right?
CW Yeah. So drop is a place where you can basically do they they're called 'group buys'. And so let's just say you want to have a keyboard made or you want some some keyboard part made, a lot of times you need multiple people to buy in on it for it to even be manufactured otherwise, like the manufacturers like this isn't worth it, because it's kind of a smaller community. And so that that's kind of the origin of the website. And I've launched a couple keycap sets on drop and so one is the Scrabble keyboard. And so I got a Hasbro to license me my designs for a Scrabble keyboard that was sold on Drop a few years ago. And currently there's another keycap set on there called Astralla keys where myself and my co-designer she goes by Sailor Mercury, we made Astralla keys and that sold on on Drop as well. So I had like this existing relationship with them. And so when you approached me it was is really a matter of reaching out to them and saying, 'hey, do you want to do this for a prank?' And then from there it was it was a very, very mad rush to get this done by April 1. I'm very happy we made it on time.
BP Yeah, it was cool. He reminded me I worked briefly at DJI which is a drone manufacturer based in Shenzhen, China and we were talking to the folks at Drop, who is a great partner to us he was like 'Okay, I'll you know it's going to be morning in Shenzhen in about three hours. So I'll call the people at the factory see if they can make some space on the line and then we'll you know, pump out a prototype and then they'll ship it here and then I'll ship—' It was just like the speed at which the electronic supply chain can operate is truly staggering.
CW Yeah, it was it was amazing. And so it was truly like I made the the design for the actual sets. I got Pantone colors from the Stack Overflow team and got those over to Kanal and then he 3d modeled the case so that way he could get that CNC and anodized aluminum and yeah, it was it was a mad rush. He was able to get it all manufactured and shipped out and then I got a couple samples and we decided between those to get the official colors than it was pictures and videos and more after that.
BP Yes, that's right. You're our true influencer helping us do our unboxing video for our online influencer marketing. I was just remembering. So I used to go to this annual robotics competition that DJ would hold where they'd have these little sort of like Robo cars battle in a big arena. And the competition was like a, like a round robin tournament. So you played like multiple rounds. And the local teams always had a big advantage. Because if some a part broke, or if they like, realized something was wrong with the design, they could literally get a prototype, print up and like, go get something made overnight at the factory and like, have it the next day, you know, and like, fix up their machine or do whatever they needed to do. Whereas like, all the other teams were just sort of like trying to hot glue things together after each round. But yeah, I saw a few people commenting on Twitter, that we should actually, or they'd actually pay for this. Like, what would it take to go from making a mock up, Cassidy, which is what we have now to like, an actual functioning keyboard?
CW Yeah, and so now now that we have the mock up, at least that part is done, the the hardest part is going to be the electronics underneath. And so that involves getting some firmware, probably QMK, it's an open source firmware that allows you to reprogram it. So let's just say people wanted more than just Ctrl Copy or Control C, Control V, they could turn it to 123, they could turn it to any other numbers pretty easily. But anyway, adding firmware to a PCB, and then just doing kind of electronics testing. And so that might, that might take probably a short as maybe three weeks, and that process is definitely longer, but then maybe as long as a month and a half. And then yeah, it's getting it in there, making sure everything works and shipping it and so typically, there's what's called a, an MOQ, a minimum order quantity. And so when when you want to get something made, the more you buy, the cheaper it is, so there's probably going to be a minimum order quantity where if we sell 200 then it's going to cost $50. But if we sell 500, then it'll be $30. If we sell 1000, then it could be 25 and that kind of stuff. And so the more that's bought, the cheaper it gets. And that's typically how it works.
BP You heard them listeners, it's time to unite to bring the power of Stack Overflow's enormous audience to bear. Well, we'll make the keyboard we'll sell it mo will donate the proceeds to charity. I'm sure this is, I'm sure we can make it happen. Email me.
CW Yes!
BP Email me podcast@stackoverflow.com. You can get in early. You can pre-order directly through the podcast, I will only take us very small commission. [Cassidy & Kyle laugh] Yeah, some people were suggesting that if it's a three key Stack Overflow keyboard, then we could we could add a few other things on there. Mark as duplicate, downvote. No, I'm just kidding. Don't be sour! [Kyle laughs] Alright, great. Well, thank you so much to both of you. Yeah, for making this all come together. It was a wonderful April Fool's joke. And I think there's a lot more to come. Yeah, I think well, maybe we'll try to follow up with a post about the data and give people a sense of how often this is actually happening. So we're not none of us are alone. We're all copying in spirit. [Kyle & Cassidy laugh]
[OUTRO MUSIC]
BP Alright, let's see here. It is that time of the episode, I will shout out a lifeboat badge winner, 'how to see the source of built in JavaScript functions' awarded to TJ Crowder. Thanks, TJ for helping out, answering a question that had a negative score getting it up to a score of 20 or more. Win yourself a lifeboat.
KP I feel like that's a scary day when you're trying to read internals of JavaScript functions, like what you what are you doing? [Kyle laughs]
BP 'Would I need to understand the language that an engine such as B is written in and read through that, or is there a simpler solution?' Hmm, you need to find the source code of the relevant projects. Yeah, you got to go deep on this one. Alright. I'm Ben Popper, Director of Content here at Stack Overflow. You can always find me on Twitter @BenPopper. You can always email us podcast@stackoverflow.com. Cassidy, Kyle, you want to just say who you are and where people can find you if you want to be found?
BP Sure. I'm, I'm Kyle Pollard. I'm a developer on the public platform team. You can find me @kylejrp on Twitter, or you can go to my website KyleJrp.com. I'd love to hear your feedback on the prank and how I scared you. Feel free to follow me and chat with me on Twitter.
CW I'm Cassidy Williams. I'm a principal developer experience engineer at Netlify and you can find me @cassidoo on Twitter, on GitHub, on all the things. Cassidoo is C A S S I D O O.