Fire drill: practicing an incident before it happens
A fire drill is an incident we invent ourselves and then handle as if it were real. Here is what we prepared, how this one went, and what we forgot.
What a fire drill is
A fire drill is an incident we invent ourselves and then handle as if it were real. We find out what happened, tell the users who are affected, and put the system back into a state we can trust.
The point is to check that we know what to do before we have to do it for real. A written procedure is not the same thing as a procedure someone has actually walked through. On a real outage there is no time to work the steps out from scratch, so the useful part is everything we can prepare in advance.
What we could prepare before the fire drill
Working in the AWS console during a real incident is slow, and it is easy to miss a step. Before the drill we prepared GitHub Actions for the recovery steps we expect to need most often:
- Rollback returns the application to a version we have already shipped
- Point-in-time restore returns the database to its state at a chosen moment
- Secrets rollback returns a secret, such as our encryption keys, to its previous value
How this one went
The scenario was damaged data in the database. A user had already noticed it in the application and told us roughly when it started. That was enough to see that something was wrong. It was not enough for the restore, because a point-in-time restore needs a specific time.
We went through the logs and compared them with what the application was showing, and with the time the user had given us. That gave us the moment when the data stopped being trustworthy.
We ran the point-in-time restore. It updated the application's connection settings and rotated the database password, so the old password no longer works on the restored database.
The data had been damaged on purpose. Putting the database back would not have been the whole response. In a real incident we would also have had to change access to AWS. During the drill we only noted that. We did not change it.
What we forgot
We did not tell the users of the application that something was wrong.
We were reading the logs and waiting for the restore to finish. Users could see the damaged data, but they did not know that we were already fixing it. During an incident, that information should go out at the beginning, not after the restore has finished.
Conclusion
After the drill we talked it through. That was as useful as the restore itself. We wrote down the gaps we found, such as a missing naming convention and not telling users what is going on while the recovery is still running. They are in the procedure now, so the next outage does not depend on someone remembering them.
