DevOps has become one of those buzzwords with many conflicting definitions. What’s for certain is it’s on the rise. In our 2020 developer Survey, around 80% of the respondents believed that DevOps is at least somewhat important. We take a look at the phenomenon, some definitions, and talk to our engineers about some of the trends our survey stats are reflecting.
What is DevOps and why is it so popular?
The problems DevOps tries to address are in the name. It aims to overcome the institutional divide between the team that writes the code (Dev) and the team that manages the infrastructure and tools used to run and manage the product (Ops). As software development life cycles get more complex, it has gotten harder and harder to track responsibilities for the outcome and easier to blame other departments for bottlenecks, delayed deadlines or shortcomings in the quality of the product.
Dev, QA, and the often further fragmented Ops groups may be unaware of each other's roadblocks, have little insight into the business context, or even aim for opposing goals. Mistrust is harming the organization and leaves everyone involved unhappy.
Tom Limoncelli, Site Reliability Engineering Manager at Stack Overflow put the pain points short and simple: “The software development life cycle in most places is broken. DevOps helps to fix that.”
The term DevOps embodies the solution to one of the most critical questions of our time: How can software teams deliver business value best? The answer? Developers and Operations working together (DevOps) instead of in isolation
If DevOps holds to what it promises, the benefits can include improved deployment frequency, faster time-to-market, lower failure of new releases, and shorter time between fixes. With the benefits of greater predictability, efficiency, security, and maintainability popping up along the full life-cycle. Or to put it simply, like Teresa Dietrich, Stack Overflow’s Chief Product Officer, “The right DevOps culture ultimately makes the product you deliver better.”
With 44% of this year’s Stack Overflow Developer Survey respondents working at organizations with at least one dedicated DevOps employee, many engineering teams around the world are trying to lessen some of the pains of modern software development.
So how can DevOps deliver on its promises?
Which problems is DevOps trying to solve?
At a high level, DevOps has three pillars: Communication, Automation, and Measurement. The State of DevOps report found that companies succeeding at DevOps shared these common threads.
Let’s break that down a bit more and take a look at some DevOps practices together with the problems they are trying to tackle.
From silos to One-team-thinking
Devs primarily work with devs, operations engineers primarily work with operations engineers, infrastructure engineers focus on their infrastructure problems. It is easy to end up isolated.
Different departments and experts need to start thinking as one team, towards one goal. In order to make teams work together without friction there needs to be a cultural shift and the tools in place to encourage transparency.
This idea is described by the widely-cited Gene Kim as a smooth transition between all the SDLC steps, from identifying the requirements, to actually building the application(s), then code hand over to IT Operations, and ultimately delivery to the client.
No more ‘hell month’ during releases
When silos prevail, most companies will feel greater pain between Devs and Ops around application deployments.
Limoncelli remembers from his time at other companies, when new releases meant days or weeks of manual deployment it was not uncommon for this to involve working on the weekend. Many places have ‘hell month’: a stressful period of time when everyone is rushing to finish a release and deploy it. At some companies it happens more than 12 times a year.” With good DevOps practices, those ‘hell months’ disappear. Automation is a big part of eliminating hell month. This prevents burn-out, reduces attrition, and makes businesses more sustainable.
In the same vein, the State of DevOps report shows a need for delivery teams to standardize their patterns and components. The report’s authors acknowledge that the complexity of this task varies widely. “Some teams do this without much thought. Others, particularly those who inherit code from all over an organization, have to take a methodical approach towards eliminating variables and achieving standardization.”
Nevertheless, Limoncelli is convinced that change is happening. He is pleased to see changes in culture. From a hiring perspective, he tells me that the lack of "hell months" used to be a "perk" only a few companies could offer. “Eventually”, he predicts, “companies that do not have good DevOps practices will find it difficult to hire at all.”
Standardizing tech stacks and lazy paths
There are many historical or sometimes political reasons for unnecessary complexity and variation. So standardization as part of the process makes automation easier.
Andi Mann, Chief Technology Advocate at Splunk, writes in the State of DevOps report that successful companies show a ‘concerted effort to normalize the stack and get rid of outliers or snowflakes that need to be maintained, tested, and managed as one-offs.’ The authors include building a standard set of technology, keeping application configurations in version control, testing infrastructure changes before deploying, and making source code available to other teams as essential steps for the early stages of DevOps.
For a fresh perspective on how to make DevOps a success, Limoncelli suggests striving for ‘low context’.
A system of low context, as opposed to one with high context, requires little a priori knowledge. The simple example Limoncelli gave me goes as follows: airport signage is easy to understand for everyone—travelers from all around the world may need to know how to get to the bathroom (if you are lucky). While the customs at a family dinner, and the complicated relationship between all attending, are learned over decades without being written down.
Limoncelli believes DevOps teams are more effective when they strive to create a low-context work environment. For example, he advocates that the best way to get people to adopt a recommended practice is to make it the ‘lazy path’. “If your default for foundational tools like CI/CD and Git matches your recommended practices, you are making doing the right thing the same as doing the easy thing. You want people to fall into the pit of success.“
If you want to learn more about creating a low context DevOps environment, click here to view Tom’s webinar (free, registration required).
Repeating a process and perfecting is also expressed by Gene with “repetition and practice [as] the prerequisite to mastery.”
The State of DevOps talks about the reuse of building blocks in this context. It points out that this not only saves the team who ‘consumes’ a building block time and effort, through feedback on shared patterns it also improves on the tooling used throughout the organization.
Speaking of feedback: this is another skill to master for DevOps.
From finger pointing to feedback loops
If you give your coworkers the benefit of the doubt—which you should—then you can assume a problem only gets kicked down to the next team down the development cycle due to lack of information or communication. The answer to this is: share more.
As Gene points out “With sharing comes trust and with trust comes greater levels of collaboration.” In his holistic theory of DevOps, feedback loops are one of his three "Way of DevOps" principles.
As a Stack Overflow user writes about Gene’s loop, “this is achieved through continuous Integration/Delivery/Deployment and shared monitoring and alerting.” The authors of the State of DevOps Report echo this. They see self-service monitoring and alerting as one of the antidotes to the anti-pattern of dev and ops working in silos. “Simply opening access to these key metrics enables a sharing culture, populates feedback loops, enables continuous feedback, and promotes a culture of continuous learning across teams.”
According to Gene, this feedback loop from Ops back to Dev means every developer should work towards understanding all their customers’ feedback. Here he sees departments down the line as internal clients.
Limoncelli also sees DevOps heralding the end of finger pointing culture, “It takes two to tango,” he likes to put it. “The improvements can’t be made by just Devs or Ops (operations engineers, sysadmins, production engineers, SREs) on their own. DevOps is about having shared responsibility and therefore cooperation.”
Making friends with failure
The DevOps mindset not only involves getting better at passing on feedback about mistakes. It means getting better at making mistakes in the first place.
One of the pillars described by Gene is continual experimentation, and he stresses the importance of taking risks and learning from that failure. Tim Hunter, whose take on Gene’s three ways has entered official canon, explains that DevOps folks need to “practice outages and [to] find innovative ways to deal with them.”
Both write about DevOps as giving teams the confidence to push against limitations, because there is a process to fall back on and handle the outcomes. More frequent iterations and feedback result in a better product.
DevOps as self-service
Note that the State of DevOps Report includes self-service in their fifth and last step of archetypal stages of DevOps success. So you could consider it as reaching higher stages of DevOps.
Here the implementation of the org philosophy should allow everyone along the life-cycle to get access to internal services including development, operations, security, ITSM, and other functional areas.
Making these self-service allows teams and individuals to work at their own speed, without having to wait for things like tickets to be approved, license keys to be obtained, required configurations to be installed, or Sara to come back from vacation.
An important requirement for this, according to the State of DevOps Report, is that Ops delivers systems and configurations to be used by QA and Dev that match the final production environment. Without that, Software is often tested in an environment that is dissimilar from the production. Self-service infrastructure helps to have the two environments be as similar as possible.
Tools: What are the defining tools?
Tools don't magically solve problems. However, better practices require the right tool. You were hoping for some names to be named?
On Stack Overflow’s sister network we have a dedicated Stack Exchange DevOps community. Most DevOps engineers spend their time with us on this site as well as Stackoverflow.com. These tags are a fairly good indication of what DevOps engineers are working with right now.
Without further ado, the 30 top topic tags on DevOps:
Your DevOps tool kit will most likely include:
- Work Tracking: Asana, Monday, Jira,
- Source code control: Git, Github, GitLab, BitBucket, TFS
- CI/CD: Jenkins, Team City, Github Actions
- Source Code Analysis: SonarQube
- Artifact management: Artifactory, Docker Container Register
- Configuration Management: Terraform, Ansible, PowerShell/DSC, Puppet, (who are behind the State of DevOps report) and Chef
- Container orchestration: Docker, Kubernetes
- Monitoring: prometheus
- To enhance automation tools further, virtual infrastructures allow organizations to configure a server in an automated fashion. Amazon Web Services and Microsoft Azure are examples of virtual infrastructures.
Note: Most of these DevOps tags have huge knowledge resources on Stackoverflow.com. As with many things DevOps, don’t think of it as a rigid tech stack. The reality most likely will be tools that are loosely coupled with adjacent ones in the application life cycle.
DevOps everywhere?
It has come a long way since the day at Toronto’s Agile Conference in 2008, when the discussion around how Ops and Dev could improve collaboration famously only attracted an audience of one. In contrast, this year’s Stack Overflow developer survey 80% of the respondents believed that DevOps is at least somewhat important.
Furthermore, 44% report to work in a place with at least one dedicated DevOps employee. We also see the growing demand reflected in the price tag on experts in the field. The specialization ranks at the top of the payscale for individual contributors.
Not only that, if you account for experience, we see that DevOps command a disproportionately higher salary compared to developers within a similar level of experience in different roles.
Even though this year around 12 % of our Developer Survey participants self-identified as DevOps specialists, the process of adopting a DevOps mindset can never mean hiring one of those well-paid specialists with DevOps in the job title and being done with it. DevOps engineers should build tools and coach people on DevOps practices. The wrong way is to have a dedicated DevOps team that is expected to do the work for others, and by this creating a new silo. Exactly what DevOps is set out to tear down.
Leadership in the space comes from some of the usual suspects but is not limited to them. How the DevOps mindset is interpreted on a team by team basis may differ. And how it will evolve to incorporate security teams into the holistic approach to software development as best practices is exciting to watch.
It is said only a few unicorns (think. Netflix, Amazon, or Google) ever reach the Continuous Deployment ideal of "deploying all the way into production without any human intervention.
I was curious about where Stack Overflow employees would put us on the spectrum. The people I spoke to (mostly SREs) all seem to agree, we are somewhere in the middle.
Tom Limoncelli, Site Reliability Engineering Manager at Stack Overflow, sees the company at currently three out of five stars for DevOps excellence. He also has some ambitious projects in the pipeline to improve our operation further. “Ask me again in a year!” he exclaims.
Introducing DevOps is a huge organisational shift. There are no doubt risks involved. So it required a certain curiosity and commitment. Spafford at Garner once put it as “DevOps challenges conventional IT thinking, its constant evolution, and its requirement for acceptance and management of risk.”
With the lack of a standard approach at this point, there is one definition, however, that will stick with me. It comes from talking to Chris Hunt, Site Reliability Engineer at Stack Overflow: “DevOps is like the Spoon in the Matrix. It's not the DevOps that bends, but yourself.”