[Ed. note: While we take some time to rest up over the holidays and prepare for next year, we are re-publishing our top ten posts for the year. Please enjoy our favorite work this year and we’ll see you in 2023.]
In my last role as a technical writer, I took on a lot of developer experience responsibilities, since knowing and documenting the client-facing APIs meant I had become pretty knowledgeable about how the overall system (and engineering org) worked. It was a larger engineering org that encouraged autonomy, and as a writer with a technical aptitude, there were a lot of things that I could apply myself to.
To track down the people I needed to talk to about specific APIs, I created a spreadsheet that listed all services, their owners, code repos, docs, and more. This grew into the document of record for services. I tried to create a central hub for all of our internal tools, onboarding docs, and information. And I helped set up an internal documentation site, while trying to enforce templates and get all of the development teams to add their information to the site.
Backstage, a framework for building developer portals, would have handled all of these functions—and more—far better than my ad-hoc solutions. Spotify open-sourced their homegrown developer portal, Backstage, because when they used it at conferences, developers couldn’t pay attention to anything else: “A few folks went to one of the cloud conferences, and they were demoing some tool from within Backstage,” said Helen Gruel, [Senior Engineering Manager]. “It wasn't the tool itself that got the attention. It was Backstage, the surface where we would do the demo from, and everybody was like, what is it?” My thoughts exactly.
In this article, I’ll talk about the problems I faced, how I solved them, and how Backstage solves them better and for free. I also reached out to a former colleague, Omar Delarosa, now a senior backend engineer at Spotify who works on their ML plugin, to see how his developer experience changed from one role to the next.
Who owns this service?
My first project when hired into my last role was to document the API we gave to integration partners. I was given the list of APIs we exposed and even had some early documentation done for me. But finding information about each API (beyond playing with cURL requests) was difficult. Modern service-oriented architectures like the one I documented tend to fragment engineering departments, especially when each team is given the freedom to perform their own work the way they see fit.
So when I had questions, I needed to track down who owned the service, where their code lived, where their Jira ticket project was, which Slack channel they lived in, and where in the wiki their internal documentation was. Keeping track of this for 100+ services was a pain, so I ended up creating a spreadsheet. It turns out that everyone else in the org needed this information, so this spreadsheet that I created for myself became the document of record for services.
This may sound like bragging, but it isn’t. Anyone who has solved a similar problem in this way knows that creating the initial spreadsheet is only the start of the work you’re doing. As more people found it and found it useful, they added their own fields and sheets to it so that it became a massive, sprawling beast. Because a large engineering org has a lot of turnover, promotions, and cross-team movement, this spreadsheet needed to be regularly updated, I just created a quarterly reminder to nag the entire engineering team.
Backstage was originally created to handle this exact problem. We spoke with two members of the Backstage team at Spotify for the Stack Overflow Podcast. Tim Hansen, senior engineer, said of Backstage’s origin: “Spotify got into this stage of hyper-growth and we were just hiring like crazy. All these new developers were coming on and you kind of lose that institutional knowledge. Everyone doesn’t know all the pieces anymore or who owns what. So it started off just as a catalog of ownership—who owns this microservice that’s breaking prod right now, who do I contact about this?”
This is the Software Catalog. It’s not just services; services, websites, libraries, data pipelines, and more can be discoverable there. Each team that owns one of these things owns the configuration of their Backstage presence. They make changes when the API is updated, owners change, or dependencies are added. Once you have this information centralized with the services, you might as well start adding more.
Delarosa verified that the difference in the two approaches was real. “In my previous role, searching for a library or service's owner involved looking at Google Sheets or maybe asking around in Slack channels that referred me to other Slack channels. It took me sometimes one or more hours just to find an answer to a simple question about another team’s code. Meanwhile, searching for a service’s owner on Backstage couldn’t be simpler. As long as you know the name of the service, just search for its overview page and right there you have all the relevant links to tech docs, owners, repos, etc. in one place. I typically find an answer to a question within minutes.”
Docs as code
As the lone technical writer in a large engineering organization, once I had the public-facing API docs in good shape, it made sense to improve the internal ones. We had a wiki, but like many wikis, it was poorly organized and poorly maintained. So we decided to spin up a static site in MkDocs. It would pull markdown files from each service repo and automatically add them to the docs site. This seemed like a clearly superior process: keep the docs alongside the code so they’ll be updated together.
The downside was that the doc files were treated as code. Every change, even typos, had to have an associated ticket, be reviewed as a pull request, and go through the build process. The new site was a net positive, but it was still a cumbersome process. Even after a long process of getting every team to use a template for the main service page, I’d turn up onboarding docs, architecture specs, and other documentation that was created ad hoc in whatever format teams wanted them in.
As with many large organizations, information can be scattered across a number of wikis, emails, and documents unless there’s a deliberate effort to bring them together. Spotify found the same issues as I did. They came up with a similar solution: docs like code built on MkDocs. But they solved a problem that I couldn’t: getting it all in one place.
Another way to keep information together is for developers to specify the Stack Overflow for Teams tags that apply when they set up a docs site for their service. If there are relevant questions for those tags, they’ll automatically show up on the documentation pages. Teams is a great compliment to a robust internal documentation effort; Backstage just tightly couples that complementary relationship with a plugin. Like Backstage, it was built to solve their own internal needs first. “As we built out search as part of the Backstage experience, we wanted to include those Stack Overflow for Teams questions and answers,” said Hansen
One platform to rule them all
One of the significant problems that I saw at my old job was the vast number of tools, services, and information sources available. There was no central way to access them (other than setting up bookmarks to the individual sites). Bookmarking only worked if you knew that these resources existed in the first place.
When I started at my previous job, I got a document from my manager with links to several (but not all) of the tools and resources that we used. This included things like the office floor plan, expense processes, wikis, and more. But there was plenty of stuff that I had to find on my own. This is a pretty common problem as companies scale up: teams get new tools and processes that don’t always filter down to their colleagues. The problem gets worse when you empower teams to work independently.
I eventually found myself looking at onboarding processes. There was nothing formal at the time, and teams had spun up ad-hoc onboarding docs. When I wanted to know how certain systems worked, I was directed to individuals who sat down with me in front of a whiteboard. But that personal touch doesn’t scale well when an engineering org starts growing. “Everyone struggles with growing their workforce, extending their teams, growing their product development capabilities, and maintaining the developer productivity level,” said Greul.
Gathering everything a new hire needs into one place—their tools, their Stack Overflow for Teams knowledge base, and information about the owners of all the projects they will touch—gives them a roadmap to becoming a productive member of the team. And it scales. It’s not just useful for new hires though, I found myself referencing documentation and looking up tools I’ve used a few months ago but have since forgotten the details.
Simpler developer experiences
Anyone who has seen how large engineering organizations operate knows that a person or a tool needs to wrangle all the tools, services, docs, and information into a space where they can be useful to everyone. I’ve seen more roles appear that have “developer experience” as part of the title, so it’s clear that companies are starting to put resources behind improving developer experience. I’m grateful that Spotify created something that helps so much for free. I just wish I’d had it sooner. Watch our recent webinar to see how Backstage and Stack Overflow for Teams work together.