How the 2019 Stack Overflow Developer Survey Came to Be (And Your Last Chance to Take it!)
For nine years, we at Stack Overflow have fielded a survey, asking people who code about their opinions on a variety of topics, from whether they prefer a dark or light theme in their IDE to how their challenges change with experience. This is a huge project for us each year, with contributions in 2019…
For nine years, we at Stack Overflow have fielded a survey, asking people who code about their opinions on a variety of topics, from whether they prefer a dark or light theme in their IDE to how their challenges change with experience. This is a huge project for us each year, with contributions in 2019 from product managers, our UX researcher, designers, community managers, web developers, marketers, and me, Stack Overflow’s data scientist. We learn so much each year from our Developer Survey, both from the results and the process of fielding it; right now, we have a massive Google Doc of things to do better! One lesson that stands out is how important careful preparation is. Looking back at past years, we haven’t always been as successful at this as we’d like, but we’re proud of our progress on some challenging issues including survey design, data analysis reproducibility, and using language that connects with a broader population of developers.
There were three main folks involved in the planning of the actual survey questions and how you as a respondent engage with them. The first is Anita Taylor, a product manager here at Stack Overflow who is leading the 2019 Developer Survey. “We try to incorporate as much feedback as we can from two audiences, the external developer community and internal Stack Overflow stakeholders,” Anita says. “Internally, we get input on proposed questions from dozens of Stack Overflow employees and then ask our engineers to take the survey and give us feedback. On the developer side, we did three things: reviewed direct feedback from last year’s survey, took suggestions for new questions, and got input from 1,600 developers who took a test version of the survey.”
Anita also is responsible for managing the technical details of how respondents interact with our survey. We experienced a few hiccups in this area because of the third-party survey platform we use, such as geographical challenges and ad-blocker issues, and we are exploring options for future surveys.
The second main person involved in planning the Developer Survey this year is Beth Devine, Stack Overflow’s UX researcher. Beth’s deep user research experience has contributed to some significant improvements in question and survey design this year. “I was lucky to come on board with a great survey from 2018 to work from, but one of our biggest priorities this year was trimming the survey length,” says Beth. “Last year people overwhelmingly said it was too long.” It turns out this was harder than those of us planning the survey expected! “It’s a delicate balance between respecting that feedback and trying to make room for all the questions we’d like to ask,” Beth shares. “One solution was to identify questions and topics that had very consistent responses over the past few years. We realized that we weren’t learning a lot of new information, and that asking them every other year might be a better use of space. It still allows us to track trends in the data over time.” Last years’ median time to finish clocked in at almost 30 minutes. We successfully shortened it this year, without sacrificing too much, and as of today, this year’s respondents are finishing with a median time of 23 minutes.
The third main person involved in planning the 2019 survey is me, Stack Overflow’s data scientist. I worked with Beth in detail on question writing, so that our expectations for data analysis were aligned. I also used data from last year’s survey and trends in tags on our site (such as those visualized in our Trends tool) to choose which technologies to include on the survey. We wanted to include the most popular and fastest growing technologies without overwhelming the respondent with an extremely onerous list to scroll through. I used the write-in options from last year to check out technologies we may have missed, and we carefully considered which smaller or shrinking technologies we would be comfortable leaving off this year. If we cut one of your favorite technologies, a) I’m really sorry and b) please do write it in the text box!
So the 2019 Developer Survey is live now but for only a few more days — take it before it closes!
The final question on the survey this year is, “Do you have any feedback or thoughts you’d like to share?” What are people saying so far?
We’re doing our best to capture the opinions of everyone who codes, everywhere, so if you have a friend, family member, or colleague who codes but isn’t a registered Stack Overflow user, or is just getting started with coding, please send the survey their way!
The Stack Overflow Podcast is a weekly conversation about working in software development, learning to code, and the art and culture of computer programming.
The reason I don’t do this survey and haven’t for many years is because of things like “whether they prefer a dark or light theme in their IDE”.
Most professional coders I know don’t understand why they should only be given 2 choices of color for their main working tool (hence the number of them who work on web-orientated software who have been moving away from Adobe’s latest Dreamweaver abominations).
A coder should be able to have any color they want for any part of their work environment – given that it’s just software. Removing choice, as Adobe Microsoft et al have been doing is actually an anathema to the way many coders I know expect to be able to work. These days, most ‘coding’ tools seem to be more aimed at ‘non-coders’ or people who claim to be a coder but couldn’t tell you the difference between a while loop and a for/each loop. In other words, they are NOT coders, they are designers.
I’ve been a ‘programmer’ since the late 70s. I am still considered to be at the top of my game, I was involved with TBL (and others) to help create what you now call the ‘World Wide Web’. My advice is sought after by most of the major software companies in the world – and though I don’t ‘code’ as often as I used to do, I still keep abreast of contemporary standards and methodologies.
Despite more and more people actually learning to code, it seems to me through my limited, though expert level, experience, that the quality of code is dropping at an alarming rate. In fact it is a little concerning to me, that the best code on the planet seems to be being created by people who are intent on stealing from other people. You can understand from my viewpoint why a survey about coding that contains such leading and limited questions as the one mentioned above, does not warrant my time in completing it. Good luck.
Anonymous experienced expert coder: I can hazard a guess as to why the survey question (“whether they prefer a dark or light theme in their IDE”) seemed “leading and limited” with only two choices offered. It is partly due to the limitations inherent in surveys, i.e. free text fields are not tractable when aggregating answers.
The second cause is more supportive of your point: with proper survey design, and a sufficiently flexible third-party survey platform, it shouldn’t be a problem to accommodate the sort of question that you balked at. For example, there could have been a third answer, “any color theme IDE that I prefer”. I think some of these issues are due to the not-so-analytic SO staff tasked with writing and administering the survey. Such concerns were addressed by others in addition to yourself, see the comments in response to this meta SO question about the survey as of April 2019.
It has nothing to do with being a good programmer, but this is necessary to analyze other things. Like the growing of black people involved on IT, and other stuff. We need that to analyze our community.
Now I’ve been party to graduate destination surveys and very, very few of them earn that to start with, it’s typically around 25K i.e. the stackoverflow figure has the same 50% markup I’m seeing for my position. Not just me, other tools that gather salary information like linkedin and glassdoor report the lower salaries as the norm too..
Basically, how do you know your respondents aren’t just feeding you false info? Is this info really taking into account location (London skews the uk a lot)?Because if that happens all the nice analyses and visualisations in the world are fairly pointless 🙂
10 Comments
<3
good
The visualization of the feedback section is really cool, my favorite part is the free-floating < and 3 connection.
Awww, there’s a little <3 heart in the graph! 😀
The reason I don’t do this survey and haven’t for many years is because of things like “whether they prefer a dark or light theme in their IDE”.
Most professional coders I know don’t understand why they should only be given 2 choices of color for their main working tool (hence the number of them who work on web-orientated software who have been moving away from Adobe’s latest Dreamweaver abominations).
A coder should be able to have any color they want for any part of their work environment – given that it’s just software. Removing choice, as Adobe Microsoft et al have been doing is actually an anathema to the way many coders I know expect to be able to work. These days, most ‘coding’ tools seem to be more aimed at ‘non-coders’ or people who claim to be a coder but couldn’t tell you the difference between a while loop and a for/each loop. In other words, they are NOT coders, they are designers.
I’ve been a ‘programmer’ since the late 70s. I am still considered to be at the top of my game, I was involved with TBL (and others) to help create what you now call the ‘World Wide Web’. My advice is sought after by most of the major software companies in the world – and though I don’t ‘code’ as often as I used to do, I still keep abreast of contemporary standards and methodologies.
Despite more and more people actually learning to code, it seems to me through my limited, though expert level, experience, that the quality of code is dropping at an alarming rate. In fact it is a little concerning to me, that the best code on the planet seems to be being created by people who are intent on stealing from other people. You can understand from my viewpoint why a survey about coding that contains such leading and limited questions as the one mentioned above, does not warrant my time in completing it. Good luck.
Anonymous experienced expert coder: I can hazard a guess as to why the survey question (“whether they prefer a dark or light theme in their IDE”) seemed “leading and limited” with only two choices offered. It is partly due to the limitations inherent in surveys, i.e. free text fields are not tractable when aggregating answers.
The second cause is more supportive of your point: with proper survey design, and a sufficiently flexible third-party survey platform, it shouldn’t be a problem to accommodate the sort of question that you balked at. For example, there could have been a third answer, “any color theme IDE that I prefer”. I think some of these issues are due to the not-so-analytic SO staff tasked with writing and administering the survey. Such concerns were addressed by others in addition to yourself, see the comments in response to this meta SO question about the survey as of April 2019.
Hey…I’m really impressed with the visualisation of the feedback…can anyone please share how they could have created it
Seems like they have used the feed backs to generate word embedding. You can read it more here.
https://towardsdatascience.com/mapping-word-embeddings-with-word2vec-99a799dc9695
It has nothing to do with being a good programmer, but this is necessary to analyze other things. Like the growing of black people involved on IT, and other stuff. We need that to analyze our community.
I’ve tried the salary tool for my background and location and it reckons I should be on 50% more than I should be 🙂
In fact I tried it pretending to be a recent graduate for my location (edinburgh, uk) with a single skill in javascript and it reckons the median salary is £37,000!
https://stackoverflow.com/jobs/salary/results?l=Edinburgh%2c+United+Kingdom&ed=1&ex=0&ff=1&dr%5B0%5D=FullStackDeveloper&tl%5B0%5D=javascript
Now I’ve been party to graduate destination surveys and very, very few of them earn that to start with, it’s typically around 25K i.e. the stackoverflow figure has the same 50% markup I’m seeing for my position. Not just me, other tools that gather salary information like linkedin and glassdoor report the lower salaries as the norm too..
Basically, how do you know your respondents aren’t just feeding you false info? Is this info really taking into account location (London skews the uk a lot)?Because if that happens all the nice analyses and visualisations in the world are fairly pointless 🙂