Last month we posted a framework about the types of feedback we incorporate when building new features at Stack Overflow.
Qualitative Quantitative Intent/desire exploratory
- Exploratory user interviews
- New feature requests
- Comparative site analysis
- Historical archive research
- Traffic and usage data analytics
- Exploratory surveys
Attitude/perception product experience
- Product experience user interviews
- User discussions (meta, mod team, chat, mod council)
- CM feedback
- Surveys (site satisfaction, UX surveys, mod surveys)
- Site satisfaction survey
Action/behavior product usage
- Usability studies
- Bug reports
- Support requests
- Feature enhancements
- Usage analytics (feature usage, engagement, and task completion)
- A/B testing
- Observational testing
- Data Team analyses
After that post we received a bunch of requests to share how we use this feedback framework on specific features. We got excited about this, and given that we just released Dark Mode (and “Ultra Dark Mode”), we thought this was a great opportunity to show how we arrived at our solution.
For each unique feature, we use quite a few of the feedback mechanisms shared in our framework. It’s rare that we use all of them. We try to use as many as possible when building a big feature, as we want to be extremely thoughtful about what we are building for the network. Our changes affect the experience of millions of users.
Dark Mode
Dark Mode was a long time coming on Stack Overflow. Often I’ve heard people ask why building Dark Mode was such an undertaking. After all, it’s just CSS! Something that’s easy to forget is that Stack Overflow was built over ten years ago. There is a lot of code that was written before there were cool CSS preprocessors like Less and Sass. Our engineers have built thoughtfully over the years, but they aren’t fortune tellers. Predicting what an application’s technical debt will look like in five to ten years is an impossible task. Building Dark Mode meant going back and paying down a lot of that technical debt. Deciding if this was the right feature to dedicate precious developer resources towards was not an easy decision. We leaned hard on these feedback mechanisms to help us make it.
New feature requests
In April 2019, a user opened a GitHub issue for a dark theme on our repository for Stacks, our design system for Stack Overflow. It wasn’t the first time we’d heard the request. Having a design system like Stacks made this much more of an attainable feature. While there was still a lot of debt to be paid down, a large portion of Stack Overflow is implemented using Stacks. The issue got a lot of traction quickly and became the highest voted issue to date (well, the most thumbs up emoji) all of this brought this option front of mind for lots of people.
Comparative site analysis
During the discovery phase, we found a lot of great ways that other applications had implemented Dark Mode and some that left us wanting more. Without calling any site out specifically, we learned that we appreciated implementations that had less contrast. Some Dark Modes are difficult to read because of the range of colors, and we wanted ours to be muted and clear. We loved Twitter’s concept of “dim” or “lights out” as settings. Aaron Shekey, our design team lead, had some experience with adding a dark mode in his app Hum and applied a lot of those learnings here at Stack Overflow.
Historical archive research
Dark Mode is the #1 requested feature in the history of our Meta Stack Overflow site. It’s a thread that has gotten revisited quite a few times over the years. The topic was first brought up nearly six years ago and is one of our top voted questions on Meta Stack Overflow.
User discussions (e.g. Meta, Mod Team, chat, Mod Council)
Dark Mode has been something that most of our public-facing staff have been aware of—CMs, developers, designers, and our architecture team have all been asked "When's SO getting Dark Mode?" repeatedly—and probably wanted it themselves.
Dark Mode wasn’t just requested in one thread on Meta. There are quite a few very popular user requests under the [dark-mode] tag (both pre and post release). We used this information to determine what our users were really looking for in a Dark Mode beta. Additionally, users built their own scripts to manually add Dark Mode to Stack Overflow. Those demonstrated both a need and some practical applications of what Dark Mode could look like on Stack Overflow.
CM feedback
When it finally became possible to implement Dark Mode , we decided to make it part of our 2020 April Fools' Joke, so it had to stay secret. However, with COVID-19 looming, we switched gears last minute and rolled it out as a regular feature. We made it possible for staff—including CMs—to test it live on Stack Overflow through a site setting and set up a Slack channel to capture feedback, bugs, and improvements in early February.
Once released, the direct work with the community began—one of our CMs created a feedback post on MSO where SO users could point out problem areas and suggest improvements. Some of the MSO users even compiled the post's 150+ answers into the accepted answer, a digest of all the feature requests and bugs. Even now, Aaron is working hard to fix errors and make improvements based on what the community has recommended.
Surveys (site satisfaction, UX surveys, mod surveys)
In our monthly Site Satisfaction Survey, Dark Mode came up often. For example. in the first round of site satisfaction survey analyses, design feedback made up 9.8% of responses to the open-ended answers to the question: “What do you find most frustrating or unappealing about using Stack Overflow?” In round two of the Site Satisfaction Survey, design made up 5.7% of the responses, requests for Dark Mode made up a fair amount of those responses within the design category.
Feature enhancements
When we considered the enhancements to add to this feature, we took into consideration how people experience the site now. Especially when it comes to experience customizations, like advertisements, keyboard shortcuts, and navigation. Our users appreciate the ability to fine tune their Stack Overflow experience. Like those other features, we made Dark Mode something that only registered users have access to and added a setting on their profiles where they could toggle between Light Mode, Dark Mode, whatever their default system setting is, and (for April Fools) Ultra Dark Mode. We also worked to make it available on as much of the Stack Overflow ecosystem as we could right from the start, including on Teams and our company pages with plans to expand it to job postings and Teams Enterprise in the near future.
Data Team analyses
Since we’ve launched Dark Mode, our data team has been heads down on analyzing the effect to the network. Not only has it led to the most signups Stack Overflow has experienced in a month, we’ve also seen a rise in engagement around the network. We’re attributing some of that to the fact that users on Stack Overflow are signing in and then going to our sister sites on Stack Exchange. We’re keeping a close eye on the effects this new feature has on the network moving forward and will use that research as historical data as we think of new features down the road.
Thanks for reading this write up! A lot of work went into this feature, and we’re so glad it’s shipped and that so many of you love it. Keep reading The Loop for more insight on how we manage feedback and incorporate what we hear from you into how we build at Stack Overflow.
Have questions about this post or feedback? Please share them here.