Building an Incident Response Plan From Scratch

It is 2 AM. Your monitoring dashboard lights up. The application is down. Customers are tweeting about it. Your phone is ringing. What do you do?

If your answer is "figure it out in the moment," you are not alone. Most startups do not have an incident response plan until after their first major outage. But by then, the damage is done: lost revenue, angry customers, and a team that is burned out from an all-night firefight with no playbook.

Here is how to build an incident response plan in one afternoon.

Define severity levels

Not every incident is the same. A minor UI bug is not the same as a complete outage. Define three or four severity levels so your team knows how to respond to each:

• SEV 1 (Critical): Complete outage or data breach. All customers affected. Revenue impact. Requires immediate response from the entire engineering team.
• SEV 2 (Major): Significant feature degradation. Many customers affected. Requires response within 30 minutes from the on-call engineer plus one additional team member.
• SEV 3 (Minor): Partial degradation. Some customers affected, workarounds available. Requires response within 2 hours during business hours.
• SEV 4 (Low): Minor issue with minimal impact. Tracked as a bug and fixed in the normal development cycle.

Establish roles

During an incident, everyone needs to know their role. Define at least three:

Incident Commander (IC): Runs the response. Makes decisions about what to investigate, what to communicate, and when to escalate. The IC does not debug the problem. They coordinate the people who do.

Technical Lead: Leads the debugging effort. Investigates the root cause, identifies fixes, and implements them. This should be your strongest engineer for the affected system.

Communications Lead: Manages external and internal communications. Updates the status page, responds to customer inquiries, and keeps stakeholders informed. This is usually someone from customer success or the founding team.

Create a communication plan

When an incident happens, you need to communicate with three audiences:

The team: Use a dedicated Slack channel (#incident-YYYYMMDD) for real-time coordination. Keep all technical discussion in this channel. Post status updates every 15 minutes for SEV 1 and SEV 2 incidents.

Customers: Update your status page within 10 minutes of confirming an incident. Send email updates for SEV 1 incidents. Be honest about what is happening. Customers forgive outages. They do not forgive silence.

Stakeholders: Notify your CEO, board, and investors for SEV 1 incidents. They should hear it from you, not from Twitter.

Build a response checklist

When an incident is declared, the IC follows this checklist:

1. Confirm the incident and assign a severity level.
2. Create the incident channel in Slack.
3. Page the on-call engineer and assign the Technical Lead role.
4. Assign a Communications Lead.
5. Post the first status page update.
6. Begin investigation. Check dashboards, logs, and recent deployments.
7. If a recent deployment is the cause, roll it back immediately.
8. Post status updates every 15 minutes.
9. When resolved, confirm resolution and update the status page.
10. Schedule a post-incident review within 48 hours.

Set up on-call rotation

Someone needs to be reachable 24/7. Use PagerDuty, Opsgenie, or even a shared phone to route alerts. Rotate on-call weekly so no one person bears the burden. Pay your engineers extra for on-call shifts, either through additional compensation or time off.

The on-call engineer should have access to all production systems, a laptop, and a stable internet connection at all times during their rotation. No exceptions.

Run post-incident reviews

Every SEV 1 and SEV 2 incident gets a post-incident review (often called a postmortem, though we prefer the less dramatic term). The review covers what happened, why it happened, what the impact was, and what you are going to do to prevent it from happening again.

The most important rule of a post-incident review: no blame. The goal is to improve systems, not to punish individuals. If someone made a mistake, ask what about the system allowed that mistake to have such a big impact. Then fix the system.

Document the review and share it with the entire company. This builds a culture of transparency and continuous improvement.

An afternoon spent building this plan will save you countless hours of chaos during your next incident. And there will be a next incident. The question is whether you will be ready for it. If you would rather not be the one running the response at 2 AM, an incident response retainer puts someone experienced on the other end of the page.

The Plan Has to Work When the Plan Is Unreachable

Almost every incident response plan written in a startup assumes Slack, the wiki, and single sign-on are all working. The incidents that hurt most are the ones where they are not. If your identity provider is the thing that is compromised, the plan that lives behind SSO is inaccessible, and the channel where you were going to coordinate is the channel the attacker is reading.

Fix this with three cheap artifacts. A one page contact sheet with mobile numbers for everyone who might be needed, stored somewhere that does not depend on your corporate systems, and refreshed whenever someone joins or leaves. A named out-of-band channel decided in advance, whether that is a group text, Signal, or a separate workspace on a different provider, so nobody spends the first twenty minutes debating where to talk. And break-glass credentials for your cloud accounts and your DNS registrar, held offline with hardware tokens, with a written record of who holds them and a rule that any use triggers a rotation afterward.

The DNS registrar deserves specific attention because it is the account most companies have never inventoried. It usually belongs to whoever registered the domain years ago, sometimes a former contractor, often secured with a shared password and no second factor. Check who has access this week rather than during your first real event.

Test the reachability, not the plan. Once a quarter, page the rotation at a random weekday hour with a test alert and record how long a human took to answer. That number tells you more than any document review, and it surfaces the expired phone number and the escalation path that goes to someone who left.

What Your Insurance Policy and Your Lawyer Require

Cyber insurance changes how you are allowed to respond, and most engineering teams have never read the policy. Policies commonly require notification to the insurer within a short window of discovery, and many require that forensics, legal, and breach notification work be performed by firms on the insurer's approved panel. Hiring your own responder first, then claiming afterward, is a well documented way to have costs denied. If you carry a policy, extract three things from it and put them directly in the plan: the notification phone number and deadline, whether pre-approval is required before engaging outside help, and the panel vendors you may use.

Legal counsel matters for a related reason. Where investigation work is directed by counsel, there is at least an argument for privilege over the resulting analysis, which affects what a plaintiff or a regulator can later obtain. That is a decision for your lawyer, not for us, but the plan should name who calls counsel and at what trigger rather than leaving it to be improvised at 3 AM by an engineer who has never spoken to the firm.

The same section should record your notification obligations in plain language. Under PIPEDA you must report breaches of security safeguards that create a real risk of significant harm, notify affected individuals, and keep records of all breaches regardless of severity. Quebec's Law 25 layers its own duties and register on top, and if you serve the EU or certain US states the clocks differ and some are shorter. The plan needs a table of who decides, who is notified, and how fast, not a legal treatise.

Then there are contractual clocks. Enterprise customers negotiate notification windows into security addenda, often 24 or 48 hours from detection, occasionally shorter. Keep a list of which customers you have promised what, held by whoever runs the response. Companies routinely discover the commitment exists after they have already missed it.

Set Up Access Before You Need It

The most expensive hours in an incident are the ones spent on procurement and access rather than on investigation. A forensics firm that has never seen your environment needs a contract signed, a scope agreed, an account provisioned, and a log export before they can begin, and that easily consumes the first day. Doing the paperwork in advance turns a day into an hour.

Three things are worth arranging while nothing is on fire. A pre-signed engagement letter with whoever will help you, so the commercial conversation is already done. A read-only investigative role in your cloud accounts, defined in code and left disabled until needed. And a documented log inventory: what you collect, where it lives, how long it is retained, and who can export it. That inventory is what tells you, in advance, that your authentication logs roll off after fourteen days.

Retention is the quiet killer. Most compromises are discovered weeks after they began, often by an outside party. If your retention window is shorter than your detection lag, the honest answer to "when did they get in" is that you cannot tell, and that answer shapes both what you must notify and what your customers conclude. Extending retention on identity, network, and database audit logs to a year is mostly a storage bill and one of the highest value security spends available to a small company.

Decisions to Make Before an Extortion Event

Ransomware and data extortion force a set of decisions under time pressure, in public, with an audience of customers. Making them in advance is the entire point of having a plan.

Decide who has authority to authorize a payment and what your default position is, understanding that sanctions rules can make payment to certain actors unlawful and that your insurer will have views. Decide whether you would take the affected environment fully offline to contain, and who approves that, since the decision is commercial as much as technical. Write your four hour holding statement now, for the point when you know almost nothing, because drafting it during the event produces either silence or overclaiming. And confirm the restoration path is not reachable from the compromised environment, which is the specific reason immutable or offline backup copies exist.

Finally, decide what "recovered" means. Restoring service is not the same as evicting an intruder, and companies that declare victory on the former have been re-encrypted within the week. The plan should require confirmation that the initial access path is closed and credentials are rotated before the incident is closed, with a named person who signs that off.

When Not to Build This Yourself, and When Not to Buy It

The honest position is that most early stage companies should write this plan themselves. The template above is not proprietary, the decisions are yours rather than a consultant's, and a plan written by the people who will execute it gets executed. If you have four engineers, one product, and no regulated data, spend the afternoon and keep the document to four pages. Paying someone to produce a forty page plan you will never open in an emergency is the most common waste of money in this category, and the length is actively harmful because nobody reads forty pages at 3 AM.

Skip the outside help entirely if what you actually lack is monitoring. A response plan for a system with no detection is a plan to react to customer complaints. Fix the alerting first.

External help pays when you need someone experienced reachable during an event, which is what an incident response retainer provides with the access and contracts already in place, when a customer or auditor requires the plan to hold up under sampling rather than merely exist, or when you are in the middle of something now and have never handled one. If you want the plan, contact sheet and incident register somewhere your team will actually find them, the free Workspace is there, and our pricing is published so you can judge whether in house is the better trade.