Why the first 15 minutes matter

Most outages aren’t solved in the first 15 minutes — but most outages are either stabilized or made worse in the first 15 minutes.

The difference is usually context: what changed, what’s impacted, and where the blast radius stops.

What we’re optimizing
Faster context → faster hypothesis → fewer wrong escalations → fewer customers affected.

What good triage should produce

  • Impact summary: who is down, where, and how badly.

  • Probable scope: single device vs node vs upstream provider.

  • Recent change context: configs, deployments, maintenance windows.

  • First hypothesis + next action: what we’re testing first and why.

Inputs you can auto-collect

Before anyone starts guessing in chat, your system should attach the same basic context to every incident. Start simple: one structured incident note containing the essentials.

  • Affected services (PPPoE, DHCP, DNS, transit, core, last-mile nodes)

  • Top alarms in the last 5–15 minutes (deduped by device + trigger)

  • Device status snapshot (CPU, memory, interfaces, optical levels where available)

  • Last change (config diff, last commit, last maintenance action, last deploy)

  • Topology context (upstream/downstream neighbors, site/node mapping)

Keep it actionable
Avoid dumping raw logs. Summarize + link. Your goal is speed and clarity, not completeness.

A simple first-15 workflow

  1. Declare incident + owner (one person drives the checklist).

  2. Capture impact in one sentence (region/service/customer count if known).

  3. Attach auto-collected context (status snapshot, top alarms, last change, topology).

  4. Pick first hypothesis (based on evidence, not preference).

  5. Run 1–2 fast checks to confirm/deny (ping/trace, interface errors, BGP state).

  6. If unclear: narrow scope (single site? upstream? node?).

  7. Update stakeholders with impact + next update time (every 10–15 mins).

Copy-paste incident note template

INCIDENT: [Short name]
OWNER: [Name]
SEVERITY: [S1 / S2 / S3]
START: [Time]
STATUS: Investigating / Mitigating / Monitoring
IMPACT (1 sentence):

  • [Who / where / what is affected]

CURRENT SIGNAL:

  • Top alarms:

  • Affected nodes/sites:

  • Customer reports:

AUTO-CONTEXT LINKS:

  • Device snapshot:

  • Last change / config diff:

  • Topology view:

FIRST HYPOTHESIS:

  • [What we think is happening + why]

NEXT ACTIONS (next 15 mins):

  • Check …

  • Verify …

  • Escalate to … if …

Common pitfalls

  • Too many people “driving” — assign one owner early.

  • No timeboxes — always state the next update time.

  • Log dumping — summarize, then link to details.

  • No change context — most incidents correlate with a recent change.

If you can’t explain the impact clearly, you’re not ready to troubleshoot yet. — NOC principle

Next steps

Start with one automation: create an incident note and auto-attach device snapshot + last change + topology.

Once that’s reliable, add deduped alarm summaries and a lightweight “suspected scope” label (single device / node / upstream).

#Automation#Incident Response#ISP Operations#NOC