For nine years, we at Stack Overflow have fielded a survey, asking people who code about their opinions on a variety of topics, from whether they prefer a dark or light theme in their IDE to how their challenges change with experience. This is a huge project for us each year, with contributions in 2019 from product managers, our UX researcher, designers, community managers, web developers, marketers, and me, Stack Overflow’s data scientist. We learn so much each year from our Developer Survey, both from the results and the process of fielding it; right now, we have a massive Google Doc of things to do better! One lesson that stands out is how important careful preparation is. Looking back at past years, we haven’t always been as successful at this as we’d like, but we’re proud of our progress on some challenging issues including survey design, data analysis reproducibility, and using language that connects with a broader population of developers.
There were three main folks involved in the planning of the actual survey questions and how you as a respondent engage with them. The first is Anita Taylor, a product manager here at Stack Overflow who is leading the 2019 Developer Survey. “We try to incorporate as much feedback as we can from two audiences, the external developer community and internal Stack Overflow stakeholders,” Anita says. “Internally, we get input on proposed questions from dozens of Stack Overflow employees and then ask our engineers to take the survey and give us feedback. On the developer side, we did three things: reviewed direct feedback from last year's survey, took suggestions for new questions, and got input from 1,600 developers who took a test version of the survey.”
Anita also is responsible for managing the technical details of how respondents interact with our survey. We experienced a few hiccups in this area because of the third-party survey platform we use, such as geographical challenges and ad-blocker issues, and we are exploring options for future surveys.
The second main person involved in planning the Developer Survey this year is Beth Devine, Stack Overflow’s UX researcher. Beth’s deep user research experience has contributed to some significant improvements in question and survey design this year. “I was lucky to come on board with a great survey from 2018 to work from, but one of our biggest priorities this year was trimming the survey length,” says Beth. “Last year people overwhelmingly said it was too long.” It turns out this was harder than those of us planning the survey expected! “It’s a delicate balance between respecting that feedback and trying to make room for all the questions we’d like to ask,” Beth shares. “One solution was to identify questions and topics that had very consistent responses over the past few years. We realized that we weren’t learning a lot of new information, and that asking them every other year might be a better use of space. It still allows us to track trends in the data over time.” Last years’ median time to finish clocked in at almost 30 minutes. We successfully shortened it this year, without sacrificing too much, and as of today, this year’s respondents are finishing with a median time of 23 minutes.
The third main person involved in planning the 2019 survey is me, Stack Overflow’s data scientist. I worked with Beth in detail on question writing, so that our expectations for data analysis were aligned. I also used data from last year’s survey and trends in tags on our site (such as those visualized in our Trends tool) to choose which technologies to include on the survey. We wanted to include the most popular and fastest growing technologies without overwhelming the respondent with an extremely onerous list to scroll through. I used the write-in options from last year to check out technologies we may have missed, and we carefully considered which smaller or shrinking technologies we would be comfortable leaving off this year. If we cut one of your favorite technologies, a) I’m really sorry and b) please do write it in the text box!
So the 2019 Developer Survey is live now but for only a few more days -- take it before it closes!
The final question on the survey this year is, “Do you have any feedback or thoughts you’d like to share?” What are people saying so far?
We're doing our best to capture the opinions of everyone who codes, everywhere, so if you have a friend, family member, or colleague who codes but isn't a registered Stack Overflow user, or is just getting started with coding, please send the survey their way!