Secrets in Pipelines

A pipeline cannot deploy without credentials. It has to push an image to a registry, tell a platform to run it, and often apply a migration to a database. Every one of those actions needs something that proves the pipeline is allowed to do it.

A pipeline is also the most thoroughly recorded place in your system. Every command's output is captured, stored, attached to a build number, and left readable by anyone who can see the build history. That is the whole point of a pipeline — you want to know what happened — and it means the place holding your deployment credentials is also the place that writes everything down.

This lesson is about holding those two facts at once. The earlier secrets lesson covered where an application keeps its credentials at run time. This one covers the automation, where the failure mode is different: not a value committed to a repository, but a value printed by a script that was trying to be helpful.

name: Deploy employees API

on:
  push:
    branches: [main]        # only merged code deploys; a fork cannot trigger this

permissions:
  contents: read
  id-token: write           # lets this job request a token about itself

jobs:
  deploy-production:
    runs-on: ubuntu-latest
    environment: production # reviewers and any environment secrets live here
    steps:
      - uses: actions/checkout@v4

      - name: Sign in without a stored credential
        uses: azure/login@v2
        with:
          # Identifiers, not secrets. Placeholder values shown here.
          client-id: 00000000-0000-0000-0000-000000000000
          tenant-id: 11111111-1111-1111-1111-111111111111
          subscription-id: 22222222-2222-2222-2222-222222222222

      - name: Deploy the digest that passed staging
        run: ./scripts/deploy.sh employees-api-prod "$(cat digest.txt)"

• The single added permission is what makes this work. The job asks the pipeline platform for a signed token describing itself — which repository, which branch, which environment — and hands that to the cloud provider, which trusts it because you configured the trust in advance.
• The three identifiers can sit in the file in plain text. They name an application, a directory and a subscription. Presenting them without a valid token gets you nothing, which is why this job has no secret to protect.
• The credential the job ends up holding is valid for about an hour and was issued for this run. A leaked build log from last month contains nothing usable.
• The environment line does two jobs. It attaches reviewers, and it scopes credentials: any secret configured on the production environment is unavailable to jobs that do not target it. A staging deploy job literally cannot read them.
• The trigger is a push to the shared branch, so a pull request cannot run this workflow at all. That is deliberate, and it is the second line of defence rather than the first — the first is that this workflow holds nothing worth reaching.
• AWS is the same shape with different names: aws-actions/configure-aws-credentials with a role-to-assume and no access key, backed by an OIDC identity provider in the account. Azure DevOps has workload identity federation for service connections. The mechanism differs; the absence of a stored key is the part that matters.

# 1. The safety net working. A value the platform injected is replaced in log output.
echo "$DB_ADMIN_PASSWORD"                      # prints ***

# 2. The same secret, revealed three ways. None of these is masked, because the
#    bytes written to the log no longer match the stored value.
echo "$DB_ADMIN_PASSWORD" | base64             # a readable encoding of it
echo "$DB_ADMIN_PASSWORD" | fold -w 1          # one character per line
curl -v -u "api:$DB_ADMIN_PASSWORD" https://employees-api.example.net/health

# 3. Not log output at all, so masking never looks at it.
cp ~/.config/deploy-settings.json ./upload/    # kept as a downloadable artifact

# 4. A value fetched during the run is unknown to the masker until you register it.
MIGRATION_PW=$(az keyvault secret show --vault-name anvi-employees-prod \
  --name employees-migration-password --query value -o tsv)
echo "::add-mask::$MIGRATION_PW"               # replaced in output from here on

# 5. To check a credential is present, measure it. Never print it.
if [ -z "$DB_ADMIN_PASSWORD" ]; then echo "password is not set" >&2; exit 1; fi

• Masking is a substring replacement on the log stream. It looks for the exact value the platform was given, which means anything that changes the bytes walks straight past it: encoding, escaping, compressing, or printing the value one character at a time.
• Nobody writes block two on purpose. It arrives during a bad afternoon, as a base64 step while debugging an authentication header, or as a verbose curl added to find out why a call was rejected. Basic authentication puts the credential in a header, and verbose mode prints headers.
• Block three is the gap people miss entirely. Masking applies to console output, not to files. A settings file, a core dump, a test report or a database export uploaded as an artifact is stored as-is and downloadable by anyone with access to the build.
• Block four matters once you start doing the right thing. Reading a credential from a cloud secret store at run time means the pipeline platform never saw it, so it cannot mask it. Registering it costs one line, and the earlier line that fetched it has already been logged if command tracing is on.
• That last point deserves stating on its own: shell tracing, which set -x turns on, prints every command with its arguments expanded. It is a useful debugging tool and it will print your secret with no involvement from you.
• Block five is the habit to build. Almost every accidental disclosure starts as a check that a value arrived, and a length test answers that question completely without putting the value anywhere.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:anvi/employees-api:environment:production"
        }
      }
    }
  ]
}

• This document answers one question: whose token is accepted. The subject condition pins it to one repository and one deployment environment, so a job in any other repository presenting a perfectly valid token is refused.
• Leaving that condition out, or widening it with a wildcard, is the mistake worth understanding. Without it the role trusts the identity provider rather than a specific caller, which means any repository on the platform can assume your production role. The trust is the control here, so the condition is the control.
• Pinning to an environment rather than a branch is usually the better choice, because the environment is where your reviewer requirements sit. A branch condition is fine too, and both beat trusting the whole repository.
• A fork cannot produce this subject. The token describes the repository the job ran in, not the code it checked out, and a fork's workflow runs under the fork's own identity.
• This is only half of least privilege. The permissions policy attached to the role is the other half, and it should list the specific actions the deployment needs — update this one service, pull from this one registry — rather than a wildcard. A deployment credential that can also read the employee database is a credential doing two jobs.
• On Azure the same two halves exist under different names: a federated credential on the identity, holding the issuer, subject and audience, and a role assignment scoped to a resource group rather than the subscription. Write both down when you set it up, because the scope is the part nobody revisits.