While GenAI has been dominating the tech news cycles for a while, does all the AI hype translate into usage for professional developers? We tapped the Stack Overflow community for answers (I hear they like answering questions). Over 1,700 people told us what code assistant tools they are using, how they feel about them, and whether they feel more productive as a result. We also asked when and how often CodeGen tools fall short, what challenges developers face with these tools, and what they are doing with all of the free time these tools purport to offer.
More quality (work) time
We found that most of those using code assistant tools report that these assistants are satisfying and easy to use and a majority (but not all) are on teams where half or more of their coworkers are using them, too. These tools may not always be answering queries accurately or solving contextual or overly specific problems, but for those that are adopting these tools into their workflow, code assistants offer a way to increase the quality of time spent working.
The majority of respondents (76%) let us know they are using or are planning to use AI code assistants. Some roles use these tools more than others amongst professional developers: Academic researchers (87%), AI developers (76%), frontend developers (75%), mobile developers (60%), and data scientists (67%) currently use code assistants the most. Other roles indicated they are using code assistants (or planning to) much less than average: data/business analysts (29%), desktop developers (39%), data engineers (39%), and embedded developers (42%). The nature of these tools lend themselves to work well when trained well; a tool such as GitHub Copilot that is trained on publicly available code most likely will be good at JavaScript for frontend developers and not so good with enterprise and proprietary code scenarios that business analysts and desktop developers face regularly.
When we asked what CodeGen tools got wrong, developers relayed they struggle with context, complexity, and obscurity. For example, one user wrote that code assistants have trouble with “[m]ost high-level or architectural questions. Or questions regarding niche components, or resources behind NDAs”. Another user wrote that their coding tool of choice provided incorrect responses when working with lesser known “programming tools (or tools of [a] specific language).”
There are more code assistant tools being released everyday, though two are dominating the space so far: ChatGPT and GitHub Copilot. Professional developers and those learning to code are equally likely to be using ChatGPT, but those learning are less likely to be using GitHub Copilot (29% vs. 49%). Given that ChatGPT offers a popular free option and GitHub Copilot offers a time-limited free trial, this makes sense. Visual Studio IntelliCode is more popular with those learning (16% vs. 11%), as there is a free version of this IDE for those not using an enterprise license.
The productivity conundrum
Beyond just grinding out code, GenAI tools may be a way to just feel good. The tools that are more satisfying to use also rank high for being easy to use: the top-box scores for easiest to use are Codeium (84%), GitHub Copilot (76%) and ChatGPT (61%) respectively and the same three in order were ranked for top-box satisfaction scores (86%, 72%, 65% respectively). In our 2023 Developer Survey, one third of respondents agreed that the top benefit of AI was increased productivity. Nearly a year later, do these easy, good time CodeGen tools translate to actual productivity? Microsoft reports that power users of AI tools at work are much more likely to use them to boost creativity and focus on higher-level, strategic work. A little help from a CodeGen tool could be what turns an exhausting day into an empowering day, even if the same amount of work gets done.
The reality of code assistants is that accuracy remains an issue: 38% of developers report code assistants provide inaccurate information half of the time or more. The inaccuracy doesn’t affect satisfaction with code assistant tools, but the productivity reported for each does have a positive relationship with satisfaction and most users (95% of those answering both questions) report “a little” to “a great deal” of additional productivity. Of course, productivity remains an elusive metric: respondents indicated that most don’t know or are unsure how their organization measures productivity. This lack of clarity was more pronounced for developers working in smaller organizations: 77% are unsure of what productivity metrics are for those in companies of less than 500 people, 72% in companies of 500-4,999, and 62% in companies for 5,000+.
Slow to adopt
The process of adopting AI tools into a complex and thoughtful workflow will ultimately affect how useful and satisfying these tools will be. We asked code assistant users to tell us what is preventing their team from adopting these tools and respondents indicated an inability to handle a higher level of complexity and lack of trust in the tools as the top two reasons. Those on teams with lower adoption rates cite the lack of usage as a self-perpetuating cycle: 11% of users on teams with 50% or more adoption say team members not using code assistants is a challenge for their team/company to integrate AI, while a higher 19% of those on teams with less than 50% adoption say lack of teammates using AI is a challenge. What’s more, nearly three quarters of our respondents (73%) indicate they don’t know or are unsure if their companies have an AI policy, which could be further slowing adoption of code assistants within teams as well. In the meantime, those on teams with lower adoption are indicating more time is being spent on high-level strategy or personal projects while those on higher adoption teams are spending more time on administrative tasks or job-related training. The processes and systems may still need time to catch up, and those able to use these tools without a process in place can enjoy a little more time for creativity in their job.
Coding assistants: Raising the bar on the next phase of the developer experience?
The nature of working as a developer is complex and dynamic. Even if productivity isn’t yet clearly articulated as a business KPI, developers are highly satisfied with AI tools that are easy to use and make them feel productive, and teams are slowly beginning to incorporate these tools into their workflows. Complexity and inaccuracy remain challenges when it comes to widespread adoption and use of code assistants, but perhaps developers on teams with lower adoption rates may actually be using the time they gain from coding assistants to engage in more creative work. This also could be a turning point where the long-term investment of an elevated experience at work becomes more important than near-term improvements on productivity metrics.
We have more questions on AI and programming trends for developers that you may be interested in weighing in on - Take our annual developer survey here while it is still open!