DevOps

What a release should actually do

CI/CD pipeline design starts with one question: is this change safe to release? Give each stage a defined job and avoid the mistakes that cost the most.

What a release should actually do

If a deploy at 9pm on a Tuesday needs three people watching a chat channel, review what those people are checking. Your pipeline should test whether a change is safe to release and give the team a way to stop or reverse it.


The one question a pipeline answers

Strip away the stages and a release pipeline exists to answer one thing: is this change safe to put in front of customers, and if it isn't, how fast can we stop?

For each stage, identify the failure it detects. Compare those checks with the last three incidents to find missing coverage and stages you can remove.

What we'd do

Ask which failure each stage detects. Remove checks you cannot justify, then use the time for missing coverage. Long waits can encourage developers to batch changes, making failures harder to trace.


Give each stage a defined job

A release that answers the question
  1. Build onceOne artefact, promoted through environments — never rebuilt
  2. Fast checksLint, unit tests, static analysis. Seconds, not minutes
  3. Slow checksIntegration tests against real dependencies
  4. Deploy to stagingSame mechanism as production, or it proves nothing
  5. Deploy to productionGradually, with a health signal that can halt it
  6. Watch, and be able to stopRollback as a routine action, not an emergency

Each stage fails faster and cheaper than the one after it. That ordering is the whole design.

Build once, promote the artefact

If you build separate artefacts for staging and production, the staging tests do not verify the artefact you ship. Promote the tested artefact instead.

Build one artefact. Promote that exact artefact through each environment. Configuration changes between environments; the thing being deployed does not.

Order the checks by speed, not importance

Run fast checks before slow ones. If a linter takes four seconds and integration tests take eleven minutes, run the linter first so developers receive that feedback before switching tasks.

Staging has to deploy the same way as production

If staging is deployed by the pipeline and production is deployed by a person with SSH access and a runbook, staging is not testing your deployment. It's testing your application, which is the half that was probably fine. The deployment mechanism is the part that breaks, and it's the part you've left untested.

Rehearse rollback before release

Rehearse rollback while the team can investigate failures without an incident in progress. Record the procedure and confirm who can run it.

Common trap

If you plan to fix a release by writing more code during the incident, record that as a roll-forward procedure. Prepare and test a rollback option where the application supports it.


The mistakes that cost most

  • Environments that drift. Production doesn't match staging, so bugs only appear after release. This is the root cause behind most "the pipeline didn't catch it" incidents, and no amount of extra test stages fixes it. Reproducible infrastructure does.
  • A test suite nobody trusts. One flaky test teaches the team that red doesn't mean broken. After that the pipeline is decoration — it goes red, someone re-runs it, and the signal is gone. A suite that's 95% reliable is worse than one that's 80% as long as everyone knows which.
  • Manual approval as a substitute for confidence. A human clicking approve on a change they can't assess isn't a control. It's a signature on a decision that was already made, and it's there to spread blame rather than reduce risk.
  • Secrets in the pipeline. Credentials in environment variables, in the repo, in someone's shell history. It works right up until it's an incident report.
  • One person who can deploy. The pipeline exists and yet deploys still happen through someone's laptop, because that's where the credentials are.

What "boring" looks like

We reported a four-minute mean deploy time, zero open Sev-1s and 99.98% uptime over a 90-day period. Those figures describe our environment, rather than a target for yours. A short deploy makes it practical to release ten small changes in a day instead of batching them into a large release.

Shorter deployments let developers release smaller changes. If your team waits an hour for each deploy, check whether that delay encourages large batches that make faults harder to trace.

We built exactly this for a global cybersecurity enterprise under SOC 2: baseline automation with Chef and Ubuntu Pro producing CIS-compliant environments, and a CI pipeline on top of it. We made the environments reproducible before building the pipeline so its checks covered the configurations used in production.


Where to start if yours isn't there

Find out what a deploy actually involves

Not the documented version. Watch one. Count the manual steps, and note who has to be awake.

Fix environment drift before adding stages

If staging and production differ in ways nobody can enumerate, extra test stages test a fiction. Reproducibility first — it's the prerequisite for everything else.

Make one artefact, promoted

Stop rebuilding per environment. This single change makes every later stage mean something.

Rehearse a rollback in daylight

Do it on a normal Wednesday with no incident. If it doesn't work, you've learned that at the cheapest possible moment.

Cut the stages that stop nothing

Then use the time you got back to add the one that would have caught your last incident.


Frequently asked questions

One that answers a single question: is this change safe to ship, and if not, how fast can we stop? Practically that means building one artefact and promoting it, ordering checks fastest-first, deploying to staging the same way you deploy to production, and treating rollback as a rehearsed routine rather than an emergency.

Because then you tested a different artefact than you shipped, and every test result downstream is evidence about something that no longer exists. Build once, promote that exact artefact, and let configuration vary between environments instead.

Aim for a process short enough that developers can release small changes without delaying other work. Compare a four-minute deploy with the pressure to batch changes when each run takes one hour.

Use approval steps where the approver can assess the change. Define what they must check and support that decision with test results and a recovery procedure.

Infrastructure. A pipeline deploying to environments that drift is a faster way to be surprised. Make the environments reproducible, then the pipeline is worth building.

Make releases boring

Pipelines that answer whether a change is safe — and let you stop when it isn't.