Fulfilling the promise of CI/CD
[Ed. note: While we take some time to rest up over the holidays and prepare for next year, we are re-publishing our top ten posts for the year. Please enjoy our favorite work this year and we’ll see you in 2022.]
A Morality Tale
Once upon a time there was a large and terrifying pirate ship, and it ruled the high seas despite being a bit leaky. The crew often had to pump seawater out of the lower decks after a storm or a fight. One day, after a particularly vicious battle, they realized that cannonballs had breached the hull below the water line. They weren’t sure how to fix it without tools or carpenters. So they just hooked the pumps up to run constantly, day and night. For a while this kept pace, but soon they had to buy twice as many pumps, and bring on more sailors to staff the pumps while others sailed. Before long, they had more sailors running pumps than actually sailing the ship or looting and pillaging. This wasn’t what they signed up for, which made everyone pretty cranky. Before long they became experts in everything that had to do with pumping water out of ships, but their other skills of sailing, looting and pillaging suffered, and their best people left in frustration. Before the year was over their ship had sunk.
The moral of the story is, don’t treat the symptoms: treat the cause.
Speaking of which, let’s talk about CI/CD.
CI/CD (continuous integration and Continuous Delivery/Deployment) is part of the fabric of our lives. We go to conferences about CI/CD, we write articles and blog about CI/CD, we list CI/CD on our LinkedIn page. Nobody even debates about whether it’s the right thing to do or not. We are all on the same page. Right?
Except … if you listen closely to what people are saying, they aren’t talking about CI/CD at all—they may say “CI/CD,” but they are only talking about continuous integration. Nobody is talking about (or practicing) continuous deployment. AT ALL. It’s like we have all forgotten it exists.
Don’t get me wrong, it is wonderful that we have gotten better at CI over the past decade. But it’s not enough. It’s not even half the battle. The point of continuous integration was never to feel smug about the syntactic and logically integrated correctness of our software, although that’s a nice bonus. The point of CI is to clear the path and set the stage for continuous delivery, because CD is what will actually save your ass.
Just like on that pirate ship, you need to treat the causes instead of chasing after symptoms. And what is the primary source of those leaks and holes in our (arr!) production pirate ship? It is everything that happens between the moment you write the code and the moment that code is live: your too-long, leaky, distended, bloated, lumped-up, misfiring software digestive system.
Err…what is CI/CD, really?
Great question. Glad you asked. CI/CD is a process and a methodology designed to make sure that all the code you merge to main is deployable at any time by testing it and deploying it. Automatically. After every diff.
The goal of CI/CD is to reduce the lead time of software changes to an interval short enough that our human learning systems (adrenaline, dopamine, etc.) can map to the feedback loops of writing and shipping code.
If any engineer on your team can merge a set of changes to main, secure in the knowledge that 15 minutes later their changes will have been tested and deployed to production, with no human gates or manual intervention required, then congratulations! You’re doing CI/CD.
Very few teams are actually practicing CI/CD, but nearly all should be. This is basic software hygiene.
The software development death spiral
The time elapsed between writing and shipping is the room temp petri dish where pathological symptoms breed and snowball. Longer lead times lead to larger code diffs and slower code reviews. This means anyone reviewing or revising these nightmare diffs has to pause and swap the full context in and out of their mind any time they switch gears, from writing code to reviewing and back again.
Thus the elastic feedback loop of the development cycle begins to stretch out longer and longer, as people spend more and more time waiting on each other—to review, to comment, to deploy, to make requested changes—and more and more time paging state in and out of their brains and trying to remember where they were and what they were trying to do.
But it gets worse. Most deploys are run by hand, at some indeterminate interval after the code was written, not by the person who wrote the code, and—worst of all—with many developers’ changes batched up at once. It is this, above all else, that severs the engineer from the effects of their work, and makes it impossible for them to practice responsible ownership over the full lifecycle of their code.
Batching up changes means you cannot practice observability-driven development; you can’t expect engineers to watch their code go out and ensure it is working in production. You don’t know when your code is going out and you don’t know who is responsible for a change in production behavior; therefore, you have no accountability or autonomy and cannot practice software ownership over your own code. Ultimately, your engineers will spend a higher percentage of their time waiting, fumbling, or dealing with tech debt, not moving the business forward.
Since so many of your existing engineers are tied up, you will need to hire many more of them, which carries higher coordination costs. Soon, you will need specialized roles such as SRE, ops, QA, build engineers, etc. to help you heroically battle each of the symptoms in isolation. Then you will need more managers and TPMs, and so on. Guess what! Now you’re a big, expensive company, and you lumber like a dinosaur.
It gets slower, so it gets harder and more complicated, so you need more resources to manage the complexity and battle the side effects, which creates even more complexity and surface area. This is the death spiral of software development. But there is another way. You can fix the problem at the source, by focusing relentlessly on the length of time between when a line of code is written and when it is fully deployed to production. Fixate on that interval, automate that interval, track that interval, dedicate real engineering resources to shrinking it over time.
Until that interval is short enough to be a functional feedback loop, all you will be doing is managing the symptoms of dysfunction. The longer the delay, the more of these symptoms will appear, and the more time your teams will spend running around bailing leaks.
How short? 15 minutes is great, under an hour is probably fine. Predictability matters as much as length, which is why human gates are a disqualifier. But if your current interval is something like 15 days, take heart—any work you put into shortening this interval will pay off. Any improvement pays dividends.
This is where lots of people look dubious. “I’m not sure my team can handle that”, they may say. “This is all very well and good for Facebook or Google, but we aren’t them.”
The way you are doing it now is the hard way
This may surprise you, but continuous deployment is far and away the easiest way to write, ship, and run code in production. This is the counterintuitive truth about software: making lots of little changes swiftly is infinitely easier than making a few bulky changes slowly.
Think of it this way. Which of these bugs would be easier for you to find, understand, repro, and fix: a bug in the code you know you wrote earlier today or a bug in the code someone on your team probably wrote last year?
It’s not even close! You will never, ever again debug or understand this code as well as you do right now, with your original intent fresh in your brain, your understanding of the problem and its solution space rich and fresh.
As you write your code, you should be instrumenting it with an eye to your future self half an hour from now: “How will you be able to tell if your code is doing what you wanted it to or not?” You write the code, you merge to main, wait a few minutes, and pull up your observability tool to view production through the lens of your instrumentation. Ask yourself, “is it doing what it’s supposed to do?” and “does anything else look … weird?”
Teams that get this rhythm down usually find >80% of all bugs right then and there. It’s fresh in your head, you’re primed to look for the right things, you’re instrumenting on the spot. If anything goes wrong or doesn’t look quite right, you can churn out another diff on the spot.
Conversely, teams that don’t have a practice of swift deploys and aren’t used to looking at their changes in production, well, they don’t find those bugs. (Their customers do.)
By the way, yes, it is not exactly easy to take an app that ships monthly and get it shipping ten times a day. But it is astonishingly easy to start an app off with continuous delivery from the beginning and keep it that way, never letting it slip past 15 minutes of lead time. Like Alexander the Great picking up his horse every morning before breakfast, it hardly feels like work.
So if moving your legacy apps to CD feels too heavy a lift right now, could you at least start any new repos or services or apps off with CD from the start? Any CD is better than no CD. Having even some of your code deploy automatically after changes will enforce lots of good patterns on your team, like preserving backwards compatibility between versions.

The #1 challenge for technical leadership in 2021
Why do so few teams make continuous delivery a priority? The virtues of a short lead time and tight feedback loops are so dramatic, widely known and well-understood that I have never understood this. But now I think I do.
The issue is that this reads like a people problem, not a technical problem. So this gets classed as a management problem. And managers are used to solving management problems with management tools, like asking for more headcount.
It can also be a hard sell to make to upper management. You’re asking them to accept a delay on their feature roadmap in the short term and possibly less reliability for some indeterminate amount of time in order to implement something that runs absolutely counter to our basic human instincts, which tell us to slow down to increase safety.
By and large, engineers seem to be aware of the value of continuous delivery, and many are desperate to work on teams that move as swiftly and with as much confidence as you can with CI/CD. If you’re trying to hire more engineers, it makes great recruiting catnip. More importantly, it makes your team use your existing engineering cycles way more efficiently.
I would like to end with an appeal to the engineering managers and directors of the world. Would you like your teams to be passionate and engaged, working on all cylinders, outputting at peak capacity, spending minimal time firefighting or tending to overdue technical debt payments or waiting on each other’s reviews?
Would you like your team members to look back and remember you wistfully and fondly as someone who changed the way they thought about developing software—someone who forever raised the bar in their eyes? Our fates are in your hands.
The teams who have achieved CI/CD have not done so because they are better engineers than the rest of us. I promise you. They are teams that pay more attention to process than the rest of us. Great teams build great engineers, not vice versa.
Continuous delivery helps your engineers own their code in production, but it also has many other side benefits. It forces you to do all the right things—write small diffs, review code quickly, instrument your original intent just like you comment your code, use feature flags, decouple deploys and releases, etc.
You can either spend your life nagging people to clean up their code and follow all the best practices separately, or you can just laser focus on shortening that feedback loop and watch everything else fall into place on its own.
Is it hard? Yes, it is hard. But I hope I’ve convinced you that it is worth doing. It is life changing. It is our bridge to the sociotechnical systems of tomorrow, and more of us need to make this leap. What is your plan for achieving CI/CD in 2021?
Tags: CD/CD, continuous deployment, continuous integration, top 10 2021
47 Comments
How can one approach the problem if one can not deploy directly to production?
To be precise, we write a software , let us call it an engine, that is integrated into our customer’s products which they use mostly internally. This means we do not run the production servers ourselves but customers run their production server with our software installed. Every deployment automatically has to go through the human gates of the customer and adapt to their release cycle. Not to forget “never touch a running system” syndrome which makes customers reluctant to update our software even if they deploy a new increment of their product.
CD sounds all well if you own the full stack cloud service, but for many products there is no full stack. One may call this obsolete in the age of cloud services but that are a whole lot of products. How can those products go beyond “just running some tests on the CI” and use the potential of CD concepts?
In general I agree; going all in on CD seems like a worthwhile direction, but not necessarily a goal that can or should always be accomplished.
Maybe the goal should be CI/AD (automatic deployment) or CI/OCD (one-click-deployment) where the the CI does everything right up to but not include the actual deployment (push-to-prod, add to the download page and send email, whatever) and then the next step is fully automated, but gated on some human saying “GO!” after they are sure that the product is in a good sate (which CI should in theory[1] do) and that all the “soft requirements” are satisfied.
[1] but in practice I’d never want to release something that hasn’t had some human look at it: https://blog.revolutionanalytics.com/2017/05/the-datasaurus-dozen.html
That’s the difference between Continuous Delivery and Continuous Deployment 🙂
The idea is to keep the software in a production-ready state (being able to deploy master to prod at any time). Whether or not you do deploy every commit on master to the prod environment automatically, is not that important.
Drone.io and Harness.io have been under my radar lately.
This is great stuff, I’m going to be thinking about it for a while.
> You write the code, you merge to main, wait a few minutes, and pull up your observability tool to view production through the lens of your instrumentation. Ask yourself, “is it doing what it’s supposed to do?” and “does anything else look … weird?”
Where can I read more about observability and instrumentation? Have you written about that elsewhere? For me it’s hard to imagine a system that gives me confidence in a change, even a small one, that hasn’t been regression tested by a human being. What kind of data is a system like that gathering and displaying? Really anything that would help me understand the quoted section better would be fantastic.
To answer my own question, here are some articles that seem like good follow-up reads:
https://charity.wtf/2020/03/03/observability-is-a-many-splendored-thing/
https://www.honeycomb.io/wp-content/uploads/2020/08/SKILup-Days_-Observability-2020-Charity-Majors.pdf
https://thenewstack.io/observability-a-3-year-retrospective/
Se are working on CD, New concept of CD, circle deployment…. https://charlescd.io/
I’m curious what you mean by “production”?
For instance, I view it as “code our customers are running”. So if there is a bug in production, customers find it.
Are you thinking of something else?
This. Is some misguided version of CD the explanation for why companies like Facebook always seem to be testing in production?
I had the same doubt. Maybe she means “pre-production” in this specific case. Some environment where the developers can test their diffs live, production-like.
It all sounds great, in theory! How you’re supposed to have code reviews then? Or you’re suggesting there should be none, and everyone on the team should be able to deploy to production whatever he wants?
While having all senior devs on the team is awesome, with bigger projects it’s not always the case. Not to mention clients that cannot afford someone testing his new feature on their production and have 10 deploys in an hour (zero downtime costs money and increases complexity).
How would you approach it then?
Why would code reviews be a problem for CI/CD? If anything code reviews become even more important if you’re actually going for CD and you should absolutely limit the people who can approve changes into the release branch.
You simply work on your own branch and when trying to merge into master you do the code review. You can also test your branch in a staging environment just fine with this approach.
Where the CD approach really breaks down is if you simply don’t control the target environment or where you have required regulatory tests for specific product versions.
Code reviews happen before merging into the branches that CI operates on.
You can replace the code review process with pair programming, working in small batches, doing TDD and having effective test automation. There is no need to use branches, other than working remotely/cross time zones (hello Covid).
I was wondering the same thing. Where do code reviews fit in, if at all?
> You can fix the problem at the source, by focusing relentlessly on the length of time between when a line of code is written and when it is fully deployed to production….How short? 15 minutes is great, under an hour is probably fine.
If the target is 15-60 minutes from the time a line of code is written to the time it is fully deployed to production, it seems like that requires eliminating code reviews. And also no manual QA (no “human gates” as the author says).
Great overview!
> “Nobody is talking about (or practicing) continuous deployment. AT ALL. It’s like we have all forgotten it exists.”
I don’t think that is true – how come you think that?
I agree. CI/CD is *not* like teenage sex (anymore) furthermore, reading this I wonder whether the author actually drinks the potion herself, as this does not sound like a developer view on CI/CD. Merge to main and *then* it gets tested?!?? That really is the wrong way around.
And as some have said already, Continuous Deployment and Continuous Delivery are two separate things. We do continuous deployment (to staging) and then the delivery follows in a pace comfortable to business…
Can I ask a question about how you handle changes that take a long time (say a week)? Do you break that change into sub-changes or just wait for for the change to be completed? I worked on a client’s application which used CI (CD to a test application, not production) and I was adding a major performance enhancement. I kept it in my branch (with merging in of changes from the other dev as they came) and then merged it into the production when it was finished.
That worked fine with the small team of two devs, but what do you do with a bigger team?
Exactly the same thing, just with more than two levels of branching hierarchy.
Hey Jon at our company we use feature flags to allow larger changes to be broken up and deployed incrementally to production.
So for example if you are building a feature that may take weeks or months it lives behind this feature flag turned off for users until it is code complete, reviewed, and finally tested in production. But pieces of it get deployed (after review and testing of course) out under the feature flag which is turned off. So the first chunk of work is diligent planning then creating the appropriate feature flag and wiring it up to an application. If you are using .NET Core this tutorial might be useful https://docs.microsoft.com/en-us/azure/azure-app-configuration/use-feature-flags-dotnet-core
I think this article does a good job explaining feature flags https://launchdarkly.com/blog/what-are-feature-flags/
Your example didn’t seem like the work spent a lot of time waiting which is the main thing our team is focused on reducing time code is waiting to be deployed. Our team focuses on once something is in code review it is deployed (not released) within an hour after approval.
It’s always entertaining to read from people who think all software runs on a web server they control and where introducing bugs isn’t a problem because they can just update to a new version 15 minutes later and everything will be alright.
In reality there’s a great deal of software out there where that’s simply infeasible for a multitude of reasons (I’m looking forward to your proposal on how to do CD for embedded development on industrial machines) or where introducing bugs can have dire consequences up to human deaths.
Claiming there’s only one way of doing it and everybody else simply hasn’t seen the light just shows that the author only thinks about it from their own specific experience. That said for your usual website CD is a perfectly fine way to approach things. CI on the other hand is simply applicable for *all* software (this might show my limited experience in some situations though, it’s hard to make generalizations).
Exactly. We can release down to a staging environment automatically, but pushing to production without a human-approved gate is crazy in a great many critical software implementations.
Aye, it’s almost like most software developers these days work on websites with limited consequences (Facebook example from above)
On my first day at my current employer many years ago I was introduced to one of our users who was quite integrated into the department.
He was a specialist nurse in the topic we were writing software for, he pretty much made sure I understood that giving our users which were doctors and nurses in a specific medical discipline duff information was really not a good idea (court appearance bad idea, jailably bad idea).
Can certainly do Continuous Delivery through to various gateways before getting approval and if processes are “sorted” there’s no reason Continuous Deployment techniques can’t be used down through those gateways and deploy to live.
I could not agree more. CD for embedded software is just an entire difference scope than “push it to the cloud and let kubernetes do the rest”. When that embedded software has functional safety requirements that make lives depend on it, then it is much more complex than this.
And this is case where the size of the tests might surpass the size of the code itself.
I think this article relates more to people saying they do CICD and are only doing CI rather than those who don’t need it.
100% – do we want any developer pushing code and within 15 minutes it runs our electric grid? life support system? ECG monitors (our business)?
I don’t think it was the purpose here, but one can understand this article as proposing “shift right”, let’s just push all our problems to production and deal with them there (hence “OBSERVABILITY”).
I’d just like to find a company that appreciates the value of CI. I’ve never been given the time to implement it, and in some cases have been expressly forbidden to waste time on it.
This intro “When people say “CI/CD,” they are only talking about continuous integration. Nobody is talking about (or practicing) continuous deployment. AT ALL. It’s like we have all forgotten it exists. It’s time to change that.” seems very sensational.
And no data to back it up?
I’d like to second some of the other comments here. What does continuous deployment look like for a desktop app? Deploying a new desktop app every fifteen minutes is insane, because you’re increasing the number of versions people have in the wild. When your customers can potentially hold on to their specific version forever, you’re multiplying your support burden when every customer can have a slightly different version. Continuous Integration is a must-have, obviously, but continuous deployment doesn’t make much sense.
Now there’s nothing wrong with deploying often as long you’ve got the tooling client-side to support automatic updates, like say, VS Code, which updates monthly automatically. And I do mean automatically, all the user has to do is close the software. That vastly reduces the amount of versions that exist in the wild, you’ve got maybe three relatively close versions live at any one time. That’s doable. On the other hand, there are environments, like most enterprise software or high performance computing clusters, which really really really don’t like changing things, and have a rabid distrust of any kind of automatic updates. And usually your biggest clients have exactly those kinds of environments.
There are more kinds of software development than just web development.
This is the exact scenario for my company. CD is not automatic but human driven, on production. And big customers disable automatic updates because before chaning even the slightest thing and bring it to production stage, they want to test the whole system in their own testing stages, for weeks.
@CI/CD approval disable
“Great teams build great engineers, not vice versa.” is so true, also in mechanics projects.
Your pirate ship is a nice tale to understand history of many products.
Thanks.
A lot fo the CD comments point out the dangers of going straight to “production”. I would suggest that the better way to handle this would be to have “stable production” and “production” branches. Your battle tested “stable production” branch becomes the default for live client updates to check. While your CD changes can go straight to the “production” branch, which is basically your latest release possibly with bugs. You can add a disclaimer for anyone that might care to know. This will solve the majority of the issues I saw mentioned.
That’s great when users choose the version they want to run, but in enterprise systems there is only one version that thousands of users can run. If there is a bug in it, then those thousands of users suffer until it is fixed. If it is still broken after it is “fixed”, then the thousands of users lose all confidence in the product and start screaming for a different product or development team. Testing in production does not work in this environment.,
Being able to know that the latest version of the system can be built and deployed, say to a test team or beta group, may be a good discipline for the people who are creating it, but that doesn’t mean that each version should be released to users in general.
You wouldn’t want the manufacturer to come round to your place and swap your car for a different model without asking, and the same goes for software and web services. This applies to individuals as well as enterprise settings.
When all you have is a hammer everything starts looking like a nail. Continuous deployment is not a one-size-fits-all solution as many people have already pointed out, and what this article preaches is a bit narrow minded at best.
Even for web development (where CD is mostly applicable), there will always be external constraints like policies, legal requirements, inherent risks, etc. that make continuous deployment unfeasible or even non beneficial.
I believe most dev teams, where possible, should strive for both CI and Continuous *Delivery* (i.e. frequent “one-click” deployments), but Continuous Deployments are not for every organization and preaching that as a solution for all kinds of organizations and environments, and condemning teams that don’t do it as dysfunctional just reveals naiveness and limited exposure outside of web development.
Thank you for writing this up, Charity. It resonates soundly with both me and my team. Keep fighting the good fight.
While I like many paragraphs in this post, I am still skeptical about the need to deliver 15 minutes after the code has been pushed. (We do it on a daily basis if there is new code, and it seems to suffice.)
Why do we need such short feedback loops for writing code? Who is giving feedback on the other side (do we want the customer to be the first tester)? Do we need that hourly or every quarter of an hour?
Looking at changes in production (after 30 minutes or so) – would we really do that? We test on local machine and by someone else on the same day on stage. Why putting possibile bugs to live?
And last my first thought reading this post, which stems from my skepticism about the efficacy of CD:
What if the leaky pirate ship metaphor is this: setting up and maintaining CI/CD (pumping), instead of writing code (sailing)?
Great article! My company, Def Method, is hosting an event on February 19th, on How to Ship Software Quickly, Safely and Reliably, which touches upon several themes discussed in your article.
We’ll discuss how to ship software quickly, reliably, and safely. Whether you’re an engineering leader or a member of an engineering team and your team is struggling to release regularly, join to understand the foundations for making regular releases a reality. If your team is already releasing software regularly, join us to learn new ideas on how to further improve workflows.
We’ll discuss factors that influence the frequency of releases, what enables more frequent releases, common deployment issues and how to address them, how system complexity impacts deployment ease, and much more.
It’s a free event, for anyone who’s interested, you can rsvp here!
https://us02web.zoom.us/webinar/register/2816129086356/WN_TUswSv6dQ7O-JZgMT2QcKQ
Not all software should be continuously deployed, but all software must be continuous delivered. The idea is every commit/diff is a release candidate. Some fail, some pass, but the entire process should be automated. For software that clients use, agility means you don’t have to follow their release cycle. With agility you can always be ahead. With software that is continuously delivered and deployed on demand, you can react quickly, and always be earlier that downstream release cycle. With techniques like feature flags, behaviour can even be changed without deployment, allowing backward compatibility and change on demand.
CI/CD is good guidance for teams deploying static web sites, and maybe a few sites with small amounts of data or unimportant data. But this is not good guidance for every software team, nor even most. In fact, it is not even possible for software embedded onto chips and placed into airplanes, cars, refrigerators, rockets, and many other devices. Nor is it practical for most desktop applications and applications delivered to phones and other devices through an “app store”.
CI is generally good, though not always.
CD is not good if your software manages the heart rate of your patients. You don’t want a junior developer’s code deployed into that environment 15 minutes after they coded it. The same is true for many systems. I sure hope my stock broker doesn’t let junior developers, or even senior developers, deploy code changes to my stock trading app within 15 minutes to let me find the bugs in it. That software, like a lot of software, needs to run through lots of integration testing scenarios. A lot of software requires human user testing before it is deployed because our tools just don’t allow the automated tests to be sufficient.
I hope readers of this article realize that it very well may not apply to their type of software development. If the author of the article spent some time working on the Funds Transfer software for the Federal Reserve Bank she would understand that deploying to learn if you got it right is unacceptable. There are many environments that would rather spend an extra thousand hours to make sure the code is correct rather than deploying bugs to production once a year.
https://www.jetbrains.com/lp/devecosystem-2021/#Main_for-which-platforms-do-you-develop
Keep in mind that only 7% of developers develop for hardware and IoT. The vast majority are doing things related to the web or backend.
Also, it’s not true that for high stakes financial systems, you can’t do continous development.
Maybe there’s a different way to look at this statistics: some desktop (32%) and server/infrastructure (26%) applications maybe as mission critical as IoT or embedded…
https://www.youtube.com/watch?v=bHKHdp4H-8w
This video demonstrates how company uses CI/CD to deploy changes for brokerages. This is very high stakes situations where the translating APIs must be highly available and correct. Yet they still manage to have CD.
… Before the year was over their ship had sunk near a Northern European coastline and they founded a new country which has become to be known as the Netherlands.
Fighting the symptoms is not always a bad thing, esp. if you know there is no chance to find the root cause of a problem or you simply don’t care. It also is the driver to invent new means of observability and other innovative tools helping to mitigate symptoms.
Please don’t get me wrong, I agree with the author‘s text but I wanted to point out that nothing is really black or white.