Measurable and meaningful skill levels for developers

Understanding and supporting developers’ skills is something I think about a lot. Let me explain why.

First, I spent many a good hour of my youth learning BASIC on a Commodore 64 (shout out to cassette tape memory drives!) and then a 30 pound IBM “Portable.” I remember so vividly renumbering my lines of code by hand only to realize I didn’t understand there was an automatic renumbering function, and wondering why I hadn’t learned that before.

Second, as a leader in tech, over the course of my career, I’ve interviewed hundreds of candidates and seen thousands of resumes. And I’ve had the very good fortune to work with amazing recruiting teams who’ve reviewed hundreds of thousands of resumes on my behalf.

Third, as the CEO of a software company, the success of our company lives and dies on the strength of our engineers, our ability to remove blockers to their effectiveness, and our collective ability to help them grow in their knowledge and skill.

Fourth and finally, over the last year I stopped being just a consumer of open source code to understanding how it is created and maintained. And reader, I. was. Blown. Away. The collaboration, the ability for complete strangers to build something far more than the sum of their parts, the collective levels of trust is on the level of one of history’s greatest collective accomplishments, and certainly among the top volunteer-organized efforts, along with sharing results of scientific experiments. So I became a huge fan.

All of these experiences have led me to think about—OK, obsess about—how to improve developers’ skill.

The importance—and the challenge—of measuring developer skill

To improve anything, the first step is to be able to measure it accurately. And yes, “accurately” here comes with a huge caveat, but we’ll come back to that in a moment…

If it is possible to measure developer skill effectively, then:

• Individuals who are interested in their own growth of skills and knowledge (i.e., almost every developer I’ve ever met), have a roadmap to understand where they can focus in order to get to the next level
• Hiring managers and recruiters can efficiently match applicants to the right roles. This is good for them, their organizations, and their candidates—nobody wins when someone is under or overqualified for the position. Years of experience with a technology—especially unrealistic ones like "15 years of React experience"—don’t always tell the story.
• • In particular, engineers need a better way to convey their level of expertise across many languages and frameworks. Resumes that list "AWS / TypeScript / Terraform" and ten other items are not seen as credible by hiring managers—there’s not enough context to determine a candidate’s capabilities.
• Software organizations can effectively customize professional development opportunities to each engineer.
• Finally, open source projects can better match their projects to developers. In my involvement with open source projects, I observed that newcomers needed a minimum level of skills and knowledge to optimally contribute code—both for the sake of the busy open source maintainers and their project and for their growth and professional journeys. When skills are matched to requirements, contributors, maintainers, and projects all win.

Those are the benefits of measuring developer skill. Now let’s return to the challenge of measuring skill accurately. It is not easy.

Code is a craft, not a competition, and that means that many of the ways we measure success or improvement are simply not applicable to software engineers.

Hit more home runs this year? You are learning how to hit better (or the Major League Baseball balls are weird this year… story for another time.).

Company achieved more profit this year? The team may have collectively learned how to run a more effective business.

What about code… a developer adding more lines of code to a repository this year? It might be a good idea, it might be a bad idea, but it sure as hell is not a good indicator of skill.

No, for devs, the best indicator of developer skill is the subjective assessment of other skilled engineers. That’s inherent in the definition of the craft.

And so with that in mind I built the Developer Skill Matrix to help.

A matrix of developer skills and abilities

The Developer Skill Matrix is a structured, subjective framework of the coding and non-coding characteristics of software engineers, accessible to technologists and non-technologists alike.

Let me explain:

• Structured = there are five levels of Developer Skill: Beginner, Advanced Beginner, Intermediate, Advanced, and Expert
• Subjective = based on the wisdom and knowledge of engineers. It can be administered by the engineer themselves, their peers, or their manager or instructor.
• Coding and non-coding characteristics = the Matrix includes skill levels for core software skills for languages and frameworks, including data structures, style and semantics, and other general skills. But it also includes levels for infrastructure, including source control and application lifecycle management. And the third category of for the core/ soft skills of communication including giving and receiving feedback. Almost all coding is a team effort, and so communicating about the code with other humans is fundamental to the success of the project.
• Accessible to technologists and non-technologists alike = it is the job of technologists to use their wisdom and judgment to assign themselves or their team to skill levels. Once complete, having clear levels makes the results instantly understandable to others in the ecosystem—yes, I’m thinking of the millions of hours recruiters spend skimming resumes, trying to make their lives a little easier.

Examples of two levels:

A Beginner developer is just getting started and has little or no coding knowledge or skill. They can be self-taught developers who are just getting started, or programming or computer science students in their first two years of study.

With regard to core software skills, they can implement greedy solutions to algorithm problems but misses corner cases. They have no or low understanding of unit testing. They are familiar with a small number of frameworks, some only by name.

Advanced Beginners provide more value than they require from a dev team. Most developers who have been coding for a few months to a few years are in this category.

On core software skills, they can implement basic solutions that may not be optimal and catch simple corner cases. They will build tests if assigned, but need patterns to build off of. They are strongly familiar with at least one framework.

I’m a big believer in matrices, aka frameworks, aka rubrics.

Think about the example from human languageproficiency rubrics. Knowing your level of English, Spanish, or any one of the 7100+ languages on Earth (!) can help the student understand where they need to improve, and also help their colleagues and managers to understand what training and support is necessary to get them to the next level. Here’s one such example of language proficiency, the Common European Framework of Reference for Languages. A language learning starts with A0, Absolutely No Knowledge, and can progress through seven total levels, up through C2, Advanced Native Speaker.

After deciding on a matrix format, I then had to determine the correct levels and categories. Five levels seemed the right balance of meaningful (which would lead to more levels) and actionable (which calls for fewer).

Then I needed to come up with categories. Of course there needed to be a category for core software skills… but I wanted to provide a slightly broader view too. Infrastructure is quite important, especially one the early side of the Matrix: a coder could come of university as an Advanced Beginner for a software language but have no idea about Git. That person is not ready to contribute code to an open source project (but could become ready quickly).

Finally, communication—including being able to give and receive feedback through code reviews and other methods—is an essential component of an engineer’s ability to succeed professionally and contribute to their team. I wanted that in the rubric particularly so that engineers could get proper feedback on their communication skills from their managers and colleagues.

After the basic structure was set up, the Matrix went through many, many revisions through discussions and line edits with technologists passionate about the craft of coding. The strengths of this Matrix are due to their contributions; the shortcomings are all mine.

And a Matrix like this can at best only be partially right in the first version, and so it is freely useable and improvable by the community as an AGPL-3.0 Open Source project.

Putting the Developer Skill Matrix to work

There are a couple of ways that the Developer Skill Matrix can be used for initial assessment:

• Self-assessment: Engineers should apply their own best judgment to themselves. Even for beginner developers, this is well worth doing. One of the fundamental skills of engineering is to develop one’s estimation abilities, and so no better place to start than by estimating one’s own skills.
• Peer-assessment: Engineers ask their peers for their perspective on their skills. Like any subjective area of measurement, more perspectives are better than one to triangulate – though this does not mean that each person’s opinion should be equally weighted.
• Manager / instructor assessment: Bosses and coding teachers apply their own judgment to assess their employees / students.
• Expert assessment: Highly-trained observers perform independent assessments of coders through qualitative interviews and code reviews (in the informal sense). I have not tried this yet, but the evidence is strong from other crafts that it could be one of the most powerful and effective ways for an organization to do this.

With all of these methods, variance is your friend. What do I mean by that?

In self-assessment, if you give yourself the same level for all nine sub-categories, it is likely that you are not able to see clearly the variance in your own skill. The only exception would be a brand new developer, who could be Beginner on all levels.

In manager / instructor assessment, if every one of your team members has the same overall score, too, you are not likely seeing the variance across individuals.

In both cases, if you find yourself with consistency all across the board, bring in a second, third, or fourth person for calibration.

How to move between levels

Once you can assess skill, then you can work on increasing it.

There are three categories of methods to improve developer skill: more practice, more feedback, and more stretching.

More practice. Perhaps you’ve heard of the famous study of two groups of potters, one tasked with making great pots and the others with as many pots as possible. In the end, the group focused on quantity created higher quality pots too.

As a craft, coding gets better with more experience. So engineers looking to get better should find as many opportunities to code as they can.

This is one of the reasons why I’m so passionate about open source—it’s a 24 x 7 x 365 opportunity to keep working on your craft in domains that you care about.

Engineering teams should also be hyper focused on removing blockers to engineers actually coding. How many meetings do they have? How long do builds take? Keep a close eye on what is preventing devs from actually working on their craft.

Side note: I mentioned above that measuring quantity of code like lines of code is not a good way to assess developer skill; it is however a reliable way to increase skill.

More feedback. The subjective assessment of one’s skill as an engineer (including the non-coding categories) is one of the best accelerators of growth.

Thankfully, software development has a built-in method to deliver that feedback: code reviews. Teams should treat code reviews with the highest reverence—prioritizing code review time over new coding, recognizing great code reviewers, setting up the time and structure for proper reviews.

Code reviews should be supplemented by hands-on training in communication—giving and receiving feedback. Most humans do not naturally know how to give or receive feedback well… it is a skill too.

More stretching. This is not about doing more yoga. Though I recommend it!

Stretching is about working on assignments that are not too easy and slightly hard (but not too hard). Shout out to Yerkes and Dodson, rest in peace.

If work is too easy, it loses your interest.

If work is too hard, it leads to the “overwhelmed zone” or the “helplessness zone.”

Good community leaders and managers are constantly reassessing their colleagues’ skill levels and encouraging them to take on the next challenge.

Learning more and getting involved

The release of this Matrix is only the first step. I’d love to get your help on the journey.

1. Use the standards in the Skill Matrix in your resumes, portfolios, and interviews. Remember the craft of coding is built on trust—be as accurate as you can. Try to find a person or two who you trust, maybe a previous manager or a mentor, to review your results for calibration.
2. You manually assess your skills, or you can install an open source Discord Bot that will assess your skills, or you can join our Discord community and use the Discord Bot here.
3. Apply the skill levels to yourself for where your skills are currently. For example, to be an Advanced Beginner, you should already be performing at the Advanced Beginner level.
4. Remember, it is likely that you are at different levels for different skills. Perhaps you are Advanced on Core Coding Skills for JavaScript, Intermediate for React, and Advanced Beginner for Communication. Don’t be afraid to describe your full capabilities—having different levels is both more honest and more credible.
5. Give us feedback on how it can be better. You can reply here, create issues in GitHub, or join us in our Discord server.
6. Help out! We want to build open source versions of the assessment for Slack, the web, and other tools. Ping our contributor channel in Discord to volunteer.

I can’t wait to see what you think, and how you use it. Onward!