How to Prepare for a Disaster: Lessons Learned from Hurricane Sandy

By Curtis Hampshire

Disasters happen. And you simply can’t plan for every single scenario. But you certainly can plan based on what you know about, what you’ve lived through and what you can imagine will happen.

Hurricane Sandy furnished an example of a recent learning experience, as a little over a year ago it caused unprecedented destruction to businesses and IT operations on the Atlantic coast. The lessons from the storm provide a library of information about plans and processes, checklists and execution strategies. The following are a few of the key takeaways I learned post-Sandy.

How businesses can prepare for a disaster

  • First, leverage your partners such as technology vendors, third-party data hosting providers or disaster-recovery partners. Ask the following questions:
  • If my business is impacted in an event, what capabilities can my critical suppliers or partners bring to bear to help restore my operations?
  • Are my critical suppliers and partners also ready for significant events? If they are impacted, what are my business alternatives?
  • Does our business need a secondary failover site for critical staff and IT operations?

If we do need a failover site, have we chosen a secondary site that is outside the region of potential impact? What is the balance between local convenience, data network latency and the potential area of impact for my business?

Does my data management strategy facilitate recovery of operations and continuity of operations?

Second, review your contingency plans, team assignments and critical staffing in-depth. Consider what-if scenarios and plan these through to full conclusion. For instance, if a storm should approach, consider where you’ll place your employees. The day before the storm made landfall we deployed our chief operations officer to our Carlstadt, New Jersey, facility. This put him right in the heart of the action, and his on-site leadership and experience proved to be a key component to our operation.

Recognizing that our local staff would be impacted similarly, we also mobilized our out-of-region employees to perform roles that might normally be handled by staff in the path of the storm. We organized our data centers and staff in locations such as Denver, Houston, Scottsdale and Minnesota, so we would be prepared if we needed to switch operations to those centers. In all, we had employees prepared in more than15 states, five countries and three continents.

Given the unpredictability of the storm and the potential for the wide area of impact it presented, our teams made storm preparations as far south as Atlanta and as far north as Boston, with key personnel ready for most contingencies in those areas. All sites began execution of emergency operation plans. Teams executed against pre-defined site checklists. Key response personnel were identified, response teams were established and all clients who were serviced out of sites that would be potentially affected by the storm began receiving emergency-planning notices. The communication began two days prior to the storm’s arrival. All sites made certain that they had completed all emergency preparations three days prior to the storm’s arrival.

Third, leverage every bit of information you can get your hands on. Establish sources of data before a disaster strikes and leverage them prior to and during the event. Essentially, begin storm watching. Develop emergency scenarios and follow them through to their likely conclusion. Here’s what we did:

  • We looked at timetable charts for storm surge predictions and other information from the commercial weather sources and the National Oceanic and Atmospheric Administration (NOAA).This gave us a sense of the storm’s strength and where it would potentially make landfall.
  • We evaluated the elevations of various properties and their proximity to water to see if the storm surge could potentially affect them. The team researched NOAA storm scenario planning documents and used those documents to assess impact and response plans. The team pulled tide tables for nearby locations to gauge timing. The team also accounted for additional variables -that the storm was hitting during a full moon, and that the storm would make landfall within one hour of the high tide in the local New York and New Jersey areas.
  • We watched areas of landfall between our locations and the storm; we knew, for instance, that what happened to Battery Park at the southern tip of Manhattan would probably happen to us. As a result, we were able to make decisions and pre-stage key elements in the 30-40 minutes between the time the storm hit those locations and the time the storm hit our locations. We staged staff in locations where they were able to take measurements on critical elements, such as the real-time tidal and surge information.
  • We considered additional events such as flooding and tornadoes. We also made certain we had contingency plans, staff and customer evacuation/shelter plans and communication plans for any potential scenario we could envision.

Test your disaster preparation plans

It is critical to test your disaster preparation plans at least once a year (or more if possible). As part of this, make sure your employees are well-trained; and test your plans in as realistic a situation as possible. This means don’t tell those who will be involved that there will be a test. Plan for “what-if” scenarios; you need to work through the realistic types of problems that would arise in a disaster.

Learn from others’ experiences during events like Sandy. The first day of Sandy, businesses had to deal with the storm itself. There were wide-spread flooded facilities and power failures. Build a response team – empower them. Bring in members of the senior executive leadership team when needed and have everyone on alert and standing by. We had everyone, including our CEO, on alert and standing by to support the response activities.

Figure out what is a priority when situations are hectic. Determine what you’ll need based on common scenarios of how a disaster will affect your employees. Make sure safety is a top priority; constantly determine when staff should start for home or make other arrangements. In the case of a significant event, expect that your team will be impacted and make alternative arrangements early.

During Sandy’s second night, many enterprises were online again, but logistical troubles prevailed. Fuel was a real problem; the storm disrupted fuel suppliers and fuel distribution over a wide area. Back-up generators began to shut down for a lack of fuel. Our team worked with our national providers outside the region to redirect fuel shipments to our facilities in the impacted regions. We were also able to leverage our fuel suppliers to provide a tanker outside our Carlstadt data center. This ensured our employees and others could get the fuel they needed to get home and back.

The third day saw some business and service providers actually shut down because they hadn’t prepared or properly maintained their infrastructure. If fuel was available, business that hadn’t effectively maintained the infrastructure began to have problems. Some diesel generators started to go offline due to service issues. Many businesses found themselves operating under conditions and on a timeline that they hadn’t considered. We learned that instead of planning for a three-to-five day period, businesses should consider plans for a week to two-week scenario. We also learned that when everyone in an area has issues at the same time, competition for resources increases. Businesses that plan ahead of these scenarios will be better off than those that do not.

Communications planning is critical

Your communication plans should be well defined, include who will be communicating what and their respective timelines. Determine how you will communicate with staff, customers and vendors before, during and after a disaster. Consider every possible mechanism you can use to communicate, including social media. Transparency with customers is vital. In our case, we outlined possible scenarios for our customers and made certain we would immediately be in communication should they be impacted.

In many respects, you are preparing for disaster resiliency as much as disaster recovery. If you plan correctly and train your teams you shouldn’t have as much to recover; your resiliency will have prevented serious damage. Tap businesses and others who deal with possible disasters regularly for advice, such as business in the southeast during hurricane season or the southwest during tornado season. They have many lessons learned that they gained through tough situations. Learning from others who have had such experiences is always much less painful than learning these lessons first-hand.

Again, disasters happen. Those who plan, test and train for these scenarios will be prepared. If you take all the necessary steps, you will be able to successfully serve your business, your customers, your employees and your local communities.

Curtis Hampshire is vice president, Managed Services, for SunGard Availability Services.