Rollback

Rollback means putting the previous version back. It is the cheapest thing you can do during an incident, because it does not require understanding what went wrong — only knowing that things were better twenty minutes ago.

It is also the procedure most likely to be attempted for the first time while a service is down. That is when teams discover the previous image was pruned from the registry last week, that nobody wrote down which version was running, that the deployment script only works from one person's laptop, or that the person on call does not have the permission the command needs. None of those is difficult to fix. All of them are expensive to find out about at the wrong moment.

So the work in this lesson is mostly done in advance, on an ordinary day. Rehearse the rollback, time it, and write down what you needed. The number you get becomes the basis for a decision you will have to make under pressure: is it faster to go back, or to go forward?

-- Release 1. Additive and invisible to the running version.
ALTER TABLE employees ADD COLUMN mobile_number varchar(32) NULL;

-- Release 3. Batched, so the table is never locked for long. Run until it
-- reports zero rows updated.
UPDATE employees AS e
SET    mobile_number = e.phone
FROM  (SELECT id
       FROM   employees
       WHERE  mobile_number IS NULL
         AND  phone IS NOT NULL
       ORDER  BY id
       LIMIT  5000) AS batch
WHERE  e.id = batch.id;

-- Release 6, weeks later. The only statement here you cannot undo.
ALTER TABLE employees DROP COLUMN phone;

-- What a rename is, underneath. A drop and an add in one statement, with no
-- moment when both columns exist. Never in the same release as the code change.
-- ALTER TABLE employees RENAME COLUMN phone TO mobile_number;

• NULL in the first statement is what makes release 1 safe. The previous version inserts rows without mentioning mobile_number, and a nullable column accepts that. A NOT NULL column would reject every insert from the code currently in production.
• If the column genuinely has to be NOT NULL, that constraint is a later release of its own: add it nullable, backfill, then add the constraint once every row has a value. Three releases instead of one, and each is reversible.
• The batched update exists because one statement covering the whole table holds locks for as long as it runs. On a large employees table that is minutes, during which the API waits, which looks exactly like an outage to whoever is using the field staff app.
• Run the batch in a loop with a short pause, and stop when it updates zero rows. Because the WHERE clause skips rows that already have a value, re-running it is harmless — which means an interrupted backfill can be resumed rather than reasoned about.
• The DROP is separated from the rest by releases and by weeks, and that separation is the entire safety mechanism. There is no technical reason it cannot run on day one, and that is precisely why it needs a written rule rather than a judgement call.
• The commented rename is the trap. It reads like a tidy single change and it removes both halves of your compatibility window at once: the old column is gone, so the old code cannot read it, and there is no release in which both shapes exist.

# 1. Flag off. Changes behaviour without deploying anything and without touching
#    the schema. Only covers changes you wrapped in a flag beforehand.
az appconfig kv set --name anvi-employees-config \
  --key EmployeesApi:UseMobileNumber --value false --label production --yes

aws ssm put-parameter --name /employees-api/production/use-mobile-number \
  --value false --type String --overwrite

# 2. Previous revision. Answer this question first, every time: what was running?
az containerapp revision list --name employees-api-prod --resource-group employees-prod \
  --query "[].{revision:name, active:properties.active, created:properties.createdTime}" -o table

az containerapp ingress traffic set --name employees-api-prod --resource-group employees-prod \
  --revision-weight employees-api-prod--r0147=100

# 3. AWS equivalent. Point the service back at the task definition revision that
#    was running before, by number.
aws ecs update-service --cluster employees-prod --service employees-api \
  --task-definition employees-api:214 --force-new-deployment

• The flag is first because it is the only lever that needs no deployment. A configuration change reaches the running instances within their refresh interval, which is usually seconds to a couple of minutes, and it works during an incident where the pipeline is also broken.
• Its limitation is absolute: a flag can only disable what you wrapped in one before you shipped. That is the argument for adding a flag around anything you would want to be able to switch off, and for keeping the check simple enough that its correctness is easy to see.
• The revision list is the step people skip, and it is the one that answers what was running before. Write the current version somewhere a person can read it at deploy time — the environments lesson makes the same point — because reconstructing it from pipeline history during an incident costs minutes you do not have.
• Shifting traffic weight is fast because the old revision still exists and is still warm. This is the blue-green style of return: no image pull, no start-up, seconds rather than minutes. It only works while the platform is still retaining that revision, which is a setting worth checking rather than assuming.
• The AWS command is the same idea with a different noun. A task definition revision is an immutable numbered record, so rolling back is a pointer change — and it is only quick if you recorded which number was live.
• None of these three commands touches the database. That is the boundary the rest of this lesson is about: code and configuration come back quickly, schema does not come back at all unless you planned for it.