Open source won. In the battle between software created openly by anyone who wants to contribute and those rigidly coded in closed shops, it was no contest. If you used closed source software and you wanted new features included or bugs fixed, you had to wait for the product to change, complain about it, or develop a workaround, hoping that the change made it into the product someday. With open source, you can participate by offering changes and improvements yourself, so long as you can show that you are willing to be part of their community.
Those closed software shops are starting to take notes and incorporate open source in their process. Not only are corporations open sourcing more and more pieces of their custom stack, they’re incorporating the techniques and processes that make open source so successful.
This is called InnerSource: open source software development within a corporate engineering organization. It’s an exciting shift in how software gets made and frees up individual contributors to make significant changes outside of their silo. To get a better sense of what these practices look like in the real world, we talked to Danese Cooper, founder and chairperson at InnerSource Commons and long time open source advocate.
Open processes behind closed doors
When open source software started becoming popular, many feared that it would lead to more bugs, security holes, and other issues. But open source codebases have been found to create bugs at a slower rate than closed ones. “The secret sauce of the open source method is the fact that it's transparent. “Many eyes make small work of all bugs,” said Cooper.
If you’ve spent a lot of time reviewing your own text, be it writing or code, you probably had the experience of missing simple errors because you’re too familiar with the text. Reading it over, you see what’s in your mind, not on the screen. When open source working styles get incorporated into a closed shop through InnerSource, you get a lot of fresh eyes on your work, and those eyes end up seeing the problems you’ve missed. Beyond spotting bugs, new perspectives can also introduce fresh solutions or optimizations. This is possible because InnerSource favors an open contribution model. Anyone on any team can put some changes into a pull request and have the product owner review them for inclusion in the product. Just as in many open source projects, this doesn’t mean it’s a free-for-all. Each codebase—be it a service within a larger service-oriented architecture, a feature within a monolith, or a product within a company’s suite of products—has a set of trusted committers, people who review all commits and approve them into production.
Trusted commitership is key to the process. At first, you may have to put your best developers on the role of mentoring and reviewing/approving commits instead of writing code, which may seem like you’re letting them sit idle. “I recommend 10% of your engineering workforce should be conscripted to trusted committership,” said Cooper. “If you don't have that 10%, you will be in trouble when InnerSource starts happening.”
Another key to successful open source projects that is being ported over to closed shops through InnerSource is what Cooper calls “the automatic accretion of actionable documentation.”
“[Apache] made an early decision that anything that is important needs to land on the mailing list so that it can be archived so that people can discover it later and research into it. If you have a meatspace conversation that's material to the development of this software, it never happened unless you memorialize it on a mail list, period.”
Apache’s projects, if you haven’t heard, become pretty widespread. But your documentation storage doesn’t have to be a mailing list—it’s no longer 2004 and better tools exist. With every bit of conversation about a project automatically being visible and searchable, new contributors can both search for answers and step through the log of development decisions that is the documentation center.
Speaking of better tools, Cooper and her foundation have been talking with folks from Stack Overflow about how to use our Stack Overflow for Teams product as the documentation hub in InnerSource implementations. “I do think that your engine (Stack Overflow for Teams), as I have interacted with it in the open source world, would be a pretty good bet,” said Cooper.” As GitHub has taken over development—I think it is a brilliant tool—and as the new crop of kids come in and want to make everything their own and do it slightly differently, they don't realize they're throwing the baby out with the bath water when they have half the conversations on Slack.”
In fact, that’s exactly what Intuit has been able to do. They used Stack Overflow for Teams as their documentation log—if an Intuit engineer has a question, they were encouraged to get it out of the black hole of email or chat and get it into the permanent record of Stack Overflow for Teams.
“If Intuit has been able to use it,” said Cooper, “I imagine there's some others who've been able to use it.”
A cultural shift towards collaboration
Implementing these changes company-side takes a fair bit of work, but the rewards are great. In 1999, Tim O'Reilly, Brian Behlendorf, and Bill Portelli founded CollabNet with the intention of doing two things:
- Create a brokerage service between open source developers and anyone who wants work done.
- Teach companies to use open source methods internally.
Sounds a lot like InnerSource, right? Like many projects in technology, it may have been a good idea that arrived too soon. CollabNet failed to get traction and pivoted to products to support DevOps and Agile processes.
“When I looked at why companies couldn't get there, it was always cultural,” said Cooper. “It was always their prejudices about how engineering was supposed to happen.” Traditional engineering, especially back around the turn of the millennium, was focused on everyone building their piece of the product, then crunching in the two weeks (or, heaven help us, longer) before release. People were invested in their piece of the process because they worked at it and made sure it shipped with as few bugs as possible.
In 2014, Cooper saw that clash happen in person when she took a job at PayPal with the intention of bringing some open source magic to their internal development lifecycle. “When I got inside the company and started listening to where they wanted to go, I realized that they had no business doing open-source engagements until they understood collaborative development.”
That traditional engineering culture respects owning your little piece of the product from beginning to end. “If you have an excess of ownership culture, you have silos, and that means you have knowledge that is held by few people,” said Cooper. “If those people all go away, you have orphaned code.” Companies are left with broken, hard to maintain code, and new hires have to spend a lot of time parsing the mysteries left by their predecessor.
“A big problem with the silo system is that people's ownership makes people sort of Gollum-ish, right?” said Cooper. Getting senior devs to part with their precious is the first step. But it’s in their best interest—I don’t know any engineer that likes being bombarded with chat messages because they’ve kept the secrets to their fiefdom siloed away. For companies, breaking down walls between codebases means bottlenecks go away.
Rocio Montes, Staff Software Engineer at Intuit has seen the benefits from collaboration. “We want to communicate through the work that we're doing and not create more meetings, more time spent figuring out where to look or who to ask. Stack Overflow for Teams plays a big role in InnerSource because it helps us document all these answers that are needed for engineers to move quicker.” Collaboration allows knowledge and code reuse; think of it as avoiding boilerplate code on a large scale. If you’ve already solved a problem, share it so everyone can get their brains around the unsolved mysteries of the business.
This is the pathway to innovating new ways of working, solving new or existing problems, and letting your engineers do the interesting work that they were hired to do. Cooper talks about what happened at HP with InnerSource. “HP used to have a separate printer driver for every single device they made, sometimes more than one, if they were addressing more than one operating system. Using InnerSource, they were able to get down to under 50 from several hundred.”
Engineers are happiest when they are engaged with interesting problems. The duplicated effort that comes from trudging through hundreds of similar printer drivers—or whatever it may be in your organization—depletes morale. “The current system often will leave some of those people down in the salt mines because they don't have any way to make themselves known,” said Cooper. “They're under the control of a manager who has other ideas about where they should be. InnerSource allows people to take agency and actually do things that influence their career inside a company without going out to open source projects.”
So how do you get your engineers to actually give up control and share the hard won knowledge they’ve collected? We like to point to the public Stack Overflow and Stack Exchange sites. People give thousands of answers every day for free on a wide range of topics. “People do Stack Overflow in the public spaces because they like knowing the answer at the end of the day,” said Cooper. “That's why Jeopardy works. You always have to find that carrot, and you have to do it for every party at every level of engagement to make it actually work.”
Succeed together or fail alone
As open source becomes the de facto standard, companies looking to keep up with it need to adopt its methodologies. Most engineering processes have gotten fully onboard with making code readable and easier to maintain; it’s a short leap to allowing anyone with code access to push suggested changes and work towards the privilege of maintaining. InnerSource doesn’t need to disrupt how companies prioritize new features and fixes; the philosophy behind InnerSource assumes that teams already have ways to organize their time and priorities. It just provides a way to allow guest contributions from others familiar with the inner workings of another part of the codebase.
Good things come from cross-pollination like this. With so much ink spilled to talk about how to measure developer productivity, InnerSource lets your processes get out of engineers way to let them produce code.
Want to learn more?
Listen to Intuit’s Journey to InnerSource.
Hear from Stack Overflow’s director of product, Vasudha Swaminathan, as she sits down with the members of Intuit’s engineering team to discuss all things InnerSource.
What you’ll learn:
- What is InnerSource?
- The benefits InnerSourcing brings to organizations.
- How to gain support and adoption.
- How to enable collaboration across decentralized teams.
- Tools to support InnerSourcing.