Lesson 14 of 17Advanced16 min

Environments and Approvals

By the end of this lesson

Promote a build through environments with appropriate gates.

An environment is a complete running copy of your system with its own configuration and its own data. Three is the common arrangement: development, where changes land continuously; staging, where a build is exercised before anyone commits to it; and production, where real users and real data are.

The value is not in having three copies. It is in one artifact moving through them in order, gathering evidence at each step, with the checks between steps chosen deliberately. An environment that nobody uses to make a decision is infrastructure you are paying for and learning nothing from.

The terms, used consistently for the rest of this lesson:

Environment: A running copy of the system: its compute, its database, its storage, its configuration and its secrets. Production is one of them, and treating it as a different kind of thing is how the others stop resembling it.
Promotion: Deploying the artifact that already ran in one environment to the next, unchanged. If anything is rebuilt, it is not a promotion, and the evidence from the previous environment no longer applies.
Gate: A condition between two environments. Automated gates are checks — smoke tests, a health endpoint, an error budget. Manual gates are approvals. Both belong, in different places.
Smoke test: A small set of checks run against a deployed environment to confirm the obvious things work: the service answers, it can reach its database, one representative request succeeds end to end. Minutes, not hours.
Configuration drift: Environments diverging over time because a change was made in one and not the others. Usually discovered when a deployment that worked everywhere else fails in production.
Synthetic data: Generated records that resemble real ones in shape and volume without being real. This is how staging gets a realistic dataset without holding a copy of personal data.

The promotion path for the employees API, and what each step is actually for:

Build once, on a merge to the shared branch
One artifact, tagged with the commit, identified by digest. Every environment below deploys this exact artifact. Nothing downstream compiles anything.
Deploy to development automatically
No gate. This environment exists to catch the failures that only appear once something is deployed: a missing configuration value, a dependency injection registration that only fails at run time, a container that will not start. Those are worth finding within minutes of the merge.
Promote to staging automatically, then run smoke tests
Migrations run here first, against a dataset that resembles production. The smoke tests are the gate, and they must be able to fail the pipeline — a check whose result nobody acts on is decoration.
Hold for approval before production
A named group approves, and the approval request shows what is being promoted: the version, the commit, and what changed since the last production release. An approval without that information is a button, not a decision.
Deploy to production, then verify
The same artifact, the same deployment mechanism used in staging, followed by the same smoke tests. Using a different mechanism for production means the one you rehearsed is not the one that matters.
Record what landed where
Which version is in which environment, and when it arrived, somewhere a person can read during an incident. Reconstructing this from pipeline history takes longer than you will want to spend.

Staging earns its cost by catching problems before users meet them. It is worth being honest about which problems it can catch, because assuming it catches all of them is how confident teams get surprised.

	Staging can tell you	Only production will tell you
Start-up and configuration	Whether the artifact starts, reads its configuration, and reaches its dependencies	Little more, as long as staging is configured the same way
Migrations	Whether a migration applies cleanly, and roughly how long it takes on a similar dataset	How long it takes on the real one, and which rows break assumptions your generated data never had
Performance	Whether a query is catastrophically slow, if the data volume is comparable	Behaviour at real concurrency, with real cache states and real contention
Data edge cases	What you thought to generate	Names with apostrophes, records from 1998, the employee with no manager, and the duplicate nobody knew about
Integrations	That your code speaks the protocol correctly, usually against a sandbox	How the real third party behaves under load, and what it returns on a bad day
User behaviour	Nothing. Staging has no users	Everything: which paths are hot, what people do that you did not design for
What makes it useful	Same artifact, same deployment mechanism, comparable data volume, equivalent configuration	Monitoring good enough to notice quickly, and a rollback you have rehearsed

One artifact, three environments, one approval

YAML

jobs:
  build:
    # Produces the artifact and writes its digest to digest.txt.
    uses: ./.github/workflows/build-and-test.yml

  deploy-dev:
    needs: build
    environment: development        # no protection rules: this one is automatic
    runs-on: ubuntu-latest
    steps:
      - name: Deploy the built digest
        run: ./scripts/deploy.sh employees-api-dev

  deploy-staging:
    needs: deploy-dev
    environment: staging
    runs-on: ubuntu-latest
    steps:
      - name: Apply migrations, then deploy
        run: ./scripts/migrate.sh staging && ./scripts/deploy.sh employees-api-staging
      - name: Smoke test
        run: ./scripts/smoke.sh https://employees-api-staging.internal.example

  deploy-production:
    needs: deploy-staging
    environment: production        # required reviewers are configured on the environment
    runs-on: ubuntu-latest
    steps:
      - name: Apply migrations, then deploy
        run: ./scripts/migrate.sh production && ./scripts/deploy.sh employees-api-prod
      - name: Smoke test
        run: ./scripts/smoke.sh https://employees-api.example.net

The needs chain is the promotion path. Production cannot run until staging has succeeded, so the order is enforced by the pipeline rather than by someone remembering it.
Every deploy job runs the same script against a different target, and each one uses the digest produced by the single build job. Nothing in these jobs builds anything, which is what makes the word promotion accurate.
The approval lives on the environment, not in this file. That separation matters: a reviewer requirement stored outside the repository cannot be removed by editing a pull request, so the gate is not bypassable by the change it is gating.
Environment-scoped credentials come from the same place. The staging job cannot obtain production credentials, because they are attached to an environment it never enters.
Migrations run as their own step before the deployment rather than from application start-up, so they execute once instead of racing across replicas. They run in staging first, which is the only rehearsal a migration gets.
The smoke test after production is not optional politeness. Deploying and then not checking means the pipeline's last word on a bad release is success.

What may differ between environments, and what must not:

Must be identical: the artifact. Same digest in development, staging and production. This is the rule the whole arrangement rests on
Must be identical in shape: the deployment mechanism, the migration process, the platform service types, and the runtime version. If production uses a different mechanism, staging rehearsed the wrong thing
May differ: scale. Two replicas in staging and eight in production is reasonable and honest, as long as you know that scale-related faults will not show up before production
Must differ: secrets. Each environment gets its own credentials, its own store, and access granted separately. A shared secret means a development machine holds production access
Must differ: data. Production data does not belong in staging. Use generated records that match production in volume and in awkwardness, and keep real personal data in one place
May differ: third-party integrations, where a sandbox exists. Note what the sandbox does not simulate — rate limits and slow responses are the usual omissions
Should not differ quietly: anything changed directly in one environment's portal. Make the change in code and let it apply everywhere, or the environments diverge and the divergence is invisible until it breaks a deployment

Common mistake

Common mistakes when starting out

Staging with a few hundred rows and production with several million. A query without an index is instant on small data, and the plan the database chooses changes with volume. Staging then reliably approves the change that takes production down.

Baking environment configuration into the artifact, which forces a rebuild per environment and breaks promotion. Read configuration at run time instead.

Approvals that are rubber stamps. If an approver is shown a version number and nothing else, they cannot evaluate anything, and the gate adds delay without adding judgement. Show the diff, the commit range, and the migrations included.

Changing a setting directly in the production portal during an incident and not putting it back into code. That value now exists only there, survives until someone redeploys, and disappears without explanation.

Discovering production-only configuration at deploy time. If a value exists in production and nowhere else, the first environment that exercises it is the one with users. Give every environment the same set of keys, with different values.

A hotfix straight to production, bypassing the path. Sometimes justified, and it needs to end with the same change going through the normal route afterwards. Otherwise production is running something no environment has ever validated, and the next promotion overwrites it.

Copying production data to staging to make it realistic. This is a privacy problem, not a shortcut, and a staging environment usually has weaker access controls than the one the data came from.

Trade-off

Environments and gates both cost more than they look like they do

Every environment costs money continuously, and it costs attention: another set of secrets, another database to patch, another place a deployment can fail. Three well-maintained environments are worth more than six that nobody trusts.

Making staging resemble production is where most of the cost sits. A production-sized database in staging is a real line on the invoice, and it is also the single change that makes staging capable of catching performance and migration problems. Scaling it down is a defensible decision if you are honest that you have traded away that coverage.

Approval gates add delay measured in hours or days, and they can become theatre. A gate is worth keeping when the approver can act on what they are shown and would sometimes say no. If nobody has ever declined, you have a queue rather than a control.

More environments means slower feedback, because each one is another step before a change reaches users. The arrangement to aim for is the smallest number of environments that each answer a question nothing earlier could answer.

Summary

An environment is a full running copy with its own configuration, secrets and data, and production is one of them
One artifact moves through development, staging and production unchanged; if anything is rebuilt, the earlier evidence no longer applies
Gates belong between environments: automated checks where a machine can decide, approvals where a person genuinely might decline
Staging only catches what it resembles, so data volume and configuration parity decide how much it is worth
Secrets and data must differ per environment; the artifact, the deployment mechanism and the migration process must not

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Think about it

What did staging fail to catch?

A release passes every check in staging and causes timeouts in production within ten minutes. Staging runs the same artifact and the same migrations, with 5,000 employee records. Production has 1.8 million.

List the candidate causes that staging could not have found, and say which single change to staging would have found the most of them.

Show solution

Candidates: a query with no usable index, which is instant on 5,000 rows; a different query plan chosen by the database once the table is large; a migration that locks a big table for minutes; pagination that loads far more rows than intended; a cache that is effective at small volume and thrashes at large; and connection pool exhaustion at production concurrency.

Most of that list is data volume, not concurrency. Loading staging with generated data at production scale would surface the index, the plan change, the migration duration and the pagination fault — four of the six — without needing production traffic.

Concurrency effects remain. Pool exhaustion and cache contention need load, so either a load test against staging or a careful canary in production is the honest answer for those.

It is worth naming the decision this exercise exposes. A production-sized staging database costs real money every month, and it is the difference between a staging environment that can catch this class of fault and one that cannot. Choosing the cheaper option is legitimate; believing it protects you is not.

Challenge

Design the gates for one system

For a system you know, write the promotion path: which environments, what gate sits between each pair, and who or what decides. For every manual approval, state what the approver is shown and what would make them decline.

Show solution

A defensible answer for a small team: automatic to development with no gate; automatic to staging with migrations and smoke tests as the gate; manual approval to production, shown the commit range, the migrations included, and the result of the staging smoke run.

The question about declining is the one that does the work. If you cannot describe a realistic reason to decline, the approval is not a gate — it is a delay, and removing it would lose nothing. Common genuine reasons: a migration that needs a quiet window, a release that has to follow a customer communication, or a change that a specific person needs to see first.

Automated gates are usually undersold. A smoke test that fails the pipeline catches more real problems than an approval does, because it looks at the deployed system rather than at a description of it. Prefer adding an automated check over adding another human step.

There is a defensible opposite answer too. A team with strong automated checks, fast rollback and good monitoring can remove the production approval and deploy continuously. That is a higher bar than it looks and it is a reasonable destination, as long as the checks that replace the approval exist first.

Saved in this browser only.