This must come as no surprise to you, but the way developers—of all ages and experience levels—are interacting with knowledge has changed in the age of AI. As AI tools have become more popular, they’ve completely transformed how many technologists are choosing to consume information, ask questions, and learn new skills. And while there will always be a place for how-to articles and forum discussions, we know that how many developers seek out information has changed.
At Stack Overflow, we want to go where the developers are: to remain the always-open-tab of programmers around the world. So we created AI Assist, now available on Stack Overflow, to meet the changing needs of our lifelong users as well as the next generation of developers. A fast and efficient learning tool that prioritizes content from our expert community, AI Assist is a brand-new entry point to our public platform that combines the power of human-verified answers with generative AI to give developers the answers they need with less friction. Here’s how we built it—and what we plan to do next.
The idea
Gone are the days of developers doing keyword searches, digging through pages of search results, and opening multiple tabs for content from disparate sources. Tab- and context- switching have always been a pain point for developers, and the release of AI tools has lessened the friction of knowledge discovery for many of them. Our team at Stack Overflow knew that any modernization of our user experience would have to include AI.
For years, we’ve been the first place that developers go for knowledge and community, but we know that getting your questions answered is not without challenges. Whether it’s not knowing the community rules, struggling to find relevant content, or worrying about asking duplicate questions, there are many barriers that users may face when first accessing our sites. We needed to create a new way to use Stack Overflow that would address these barriers, providing users with guidance and direction so they can feel at home in the community. We also wanted this experience to be a friendly one—a conversational interface for problem solving and content discovery, where users can easily learn from the 17 years worth of expert knowledge already available on our platform.
Testing an AI for Stack Overflow
Our product team started their research by speaking to users, using both qualitative interviews and surveys to see where AI might fit into Stack’s user experience. We spoke to both power users and occasional users of AI tools to better understand how a wide range of individuals might interact with a feature like AI Assist.
We found that users employed AI tools across diverse use cases, often using them in combination with more conventional tools like a traditional search engine. As well, many of our respondents found that AI tools have transformed the ways they work, even as AI’s actual output can be a mixed bag in terms of accuracy and relevance. Ultimately, the users we interviewed wanted answers they can trust and for AI tools to become more seamlessly integrated into their workflows in order to alleviate friction and—especially—the need for context switching.
Because the team wanted to build quickly, we created our AI Assist tool on its own domain, allowing us to experiment and build in a space that was unobtrusive to the main site. Our Alpha build was used to test the viability of the tool’s infrastructure, allowing us to get feedback and refine the tool.
After our first round of testing of the product, which was focused on providing answers to users via an LLM experience and enriched with related Q&A from Stack Overflow, we moved onto Beta testing. Here, we included a mechanism for answers from the community, a RAG + LLM experience that would source answers from Stack Overflow and Stack Exchange sites when available, and an updated interface for easier use. In order to make sure our AI Assist tool—which is built to be model-agnostic and uses different models to surface the best answers for our users—would be on par with other AIs, we also integrated ProLLM benchmarks that rank LLM models.
Ultimately, because of our company’s ethos and community feedback, we knew that citation, attribution, and human-validated answers would be non-negotiables. Because trust in AI has decreased this last year, even as AI usage increases, making sure that AI Assist leaned on the trusted knowledge of our community would be paramount. “We have trust signals for those people who care about them, for the creators,” Product Manager Ash Zade said on an episode of the Stack Overflow Podcast. “[That] is a really important piece and one reason why we've put such a huge emphasis on attribution and sourcing…The first thing you see is, here are all the sources, and we tell the user this answer is comprised of human content augmented by AI content."
AI Assist would also include a pathway into the community to ask questions when the tool was unable to surface an exact answer, or when the user wanted to dive deeper. Through this, we are providing a way to engage with Stack Overflow with less friction than traditional search and Q&A.
How we continued to build AI Assist
We wanted to launch our next iteration of AI Assist within our public platform, fully on Stack Overflow. But before doing so, we wanted to improve speed, accuracy, and consistency. To balance these three necessary aspects of our AI tool, we ran several experiments with different models, prompting strategies, and output styles. In our mission to prioritize the accuracy of answers, we tweaked our search relevance and reranker, and made sure the latest model with the most up-to-date information would be the final step in the pipeline for augmentation. In this way, AI Assist is designed so users would receive correct answers supported by Stack Overflow’s community-created knowledge base, and that the LLM called to provide an answer would be the most up-to-date available for use.
To improve consistency and speed, we updated prompts for each of the three steps of our RAG + LLM pipeline:
- Utilize RAG to search for answers across Stack sites,
- Pulls the top results with attribution,
- Use an LLM to “audit” the answers for alternatives, structure, and completeness, and if necessary, supplement the answers with the LLM’s knowledge.
This maximized our tool’s compatibility with new models, and made it so that answers were the same or similar when asking the same question in the correct format. It also improved response speed by at least 35%.
We also made a few tweaks to the UX that would utilize Stack Overflow’s original content more and improve citations. We switched from inline quotes to blockquotes so we could highlight larger chunks of community-validated content, as well as longer code snippets with the ability to copy them. These code snippets have syntax highlighting for easier parsing and the copy code button includes attribution, which helps maintain code.
One of the major improvements we brought to AI Assist was bringing it on-platform to Stack Overflow. We did this with an HTTP proxy in the monolith to the underlying microserve. Because AI Assist originally lived on its own domain, we also needed to tweak the layout to make it work inside the Stack Overflow design. Finally, we passed a JWT from the monolight to the service so that we were able to authenticate users.
By integrating AI Assist into the public platform, we were able to enable authentication, allowing for more features and opportunities for personalization, like saving or sharing chats. These new features allow developers to jump back into their workflow and pick up where they left off, or share their conversation to boost team problem-solving by sharing chats that turn private insights into collective knowledge.
At its core, we want AI Assist to be a learning tool that breaks down barriers to access for our community’s expert knowledge base.
Now, AI Assist is widely available to anyone wanting to quickly find community-verified answers, and those wanting to learn, or connect with the community on Stack Overflow!
Our constant feedback loop with the community
As we built AI Assist, we were constantly gauging the responses of the community. Traffic to the independent AI Assist site has steadily increased as we released improvements and iterations. This reveals a curiosity from our community. Our analysis of traffic also found that AI Assist attracts a different demographic than our traditional Q&A site, with more emerging technology questions being asked on AI Assist than on the traditional Stack Overflow site.
With each version of our AI tool, we’ve seen sentiment shifting between positive and negative depending on the underlying architecture of the tool, with the latest iteration that uses the most up-to-date models having a primarily positive response.
Meanwhile, we’ve had resounding positive feedback for our attribution system, which roots answers in content that comes directly from Stack sites. This response from users has validated our human + AI approach to AI Assist, which prioritizes human-validated knowledge while still utilizing the power of AI. Users also expressed appreciation for how the tool pushes them towards learning and curiosity by adding code snippets and tips and alternatives in responses. The conversational interface was also noted because it allows users to prompt the tool with natural language and easily drill down into particular topics in one conversation.
AI Assist has already been visited by more than 285,000 technologists around the world, using it for a variety of tasks ranging from understanding error messages, to debugging code, to architecting apps. Our most engaged users are creating up to 6,4000 messages a day, with 75% of their conversations being focused on highly technical content.
What comes next for AI Assist?
AI Assist is a powerful tool that can help both new and lifelong users of our site learn, engage with the community, and dive deeper into our knowledge base. However, because the experience is unstructured, casual, and conversational, users may not recognize all of the ways it can help them, from debugging, to explaining concepts, to overcoming technical hurdles. Our next goal is to bring AI Assist deeper into our platform, meeting users where they are - like on individual Q&A pages to provide timely assistance to users.
The future of AI Assist is going to gain a lot more context, making this tool better equipped to proactively help users learn based on their interests and activity.
Finally, since it’s always been our mission to be where the developers are, we plan to bring AI Assist into our users’ IDEs, chat platforms, and wherever else they work.
The ways developers learn and consume knowledge has changed, but Stack Overflow is evolving with them. We’re building this tool for you and with you, so check out AI Assist today and let us know what you think.