Outage - Do you have an unplanned outage process?

During your course of being a SysAdmin, you will come across many unplanned outages. Some of them will impact BAU (Business as usual) and others will just be minor service outages. Do you know what to do in the event of these outages?

For planned outages, see Outage - Do you have a planned outage process?

Below is a process for these types of outages. Some amount of common sense is required here, an outage would be if services that would affect BAU work are disrupted and/or some hardware has failed.

Hardware Outage:

Firewall
Switch
Blade Servers
SAN Storage
UPS

Service Outage:

Active Directory Domain Services
O365 Services; Teams, SharePoint, Exchange, OneDrive
File Servers
SQL Servers
IIS Servers

Determining what services are disrupted

Many services can be used for device monitoring e.g. WhatsUp Gold, Solarwinds, SCOM. You would do the following in any of them:

Login to monitoring service
Check to see what services are down

First contact

After you have determined what services have been disrupted it is time to call your SysAdmin team and organize a quick conference call. This will allow you to have a discussion prior to making any changes/fixes that could cause the outage to become worse.

Key discussion points:

What services have been disrupted?
What is the impact of these services?
Is an email to everyone in your company required?
What are your next steps?

What if you cannot reach anyone?

If you cannot reach anyone move on to the Email section.

Email

If from the previous discussion you have determined that an email needs to be sent to your entire company, or you have decided this is necessary if you cannot contact anyone above, send an email in the following format:


To:	SSWAll
Subject:	SysAdmins – Outage Notice

Hi All

We are experiencing an outage and the following services have been affected:

We are working on restoring these services and will keep you updated.

Thank you, SysAdmins

A separate email needs to be sent to SysAdmins outlining what was discussed on the call. If no one was contactable, please proceed with what you have determined on your own.


To:	SysAdmins
Subject:	SysAdmins – Outage Notice

Hi Team

As per our conversation,

The following services are disrupted:

The impact of these services disrupted are:

We have decided that an email to ALL is/is not required.

To myself

The next steps to resolving this are:

Thank you, SysAdmin

Next steps did NOT resolve the issue

If you have completed your tasks but the issue has not resolved, please try to make contact with the SysAdmin team again and send an updated 'To Myself' email.