BlackswanHunting the Black Swans in Your Continuity Program

This is the fourth in the DRG ongoing series regarding hunting and mastery of the black swans in your continuity program. Look for it on the first Wednesday of each month.

“Black Swans” in your Continuity Program are those events that remain outside the range of your normal expectations, and may well produce a significant negative impact when they occur. For reasons of budget, culture, or simple lack of awareness, we just do not see or deal with these potentially devastating exposures in our enterprise continuity capability. This series discusses some of the most common of these “black swans” in business continuity programs, those that are really staring us in the face and screaming for attention.

Already published:
Quarry 1: Employee Availability for Response Activities.
Quarry 2: The Level of Individual Employee Commitment to BCM
Quarry 3: Exercising Your Plans

Quarry 4: Exercising Your Plans: Objectives and Annual Programs

Today I am going to talk about objectives for individual exercises and how all of this fits together into an annual exercise program. If you have not yet looked at last month's article on exercise types, take a look there first. Exercises will vary hugely in costs and benefits depending on what you set as the objectives for those exercises. And no, I am not talking here about the objective of "passing the test". There are many different types of exercise objectives, and these will vary based on the maturity level of your program as well as mandates from external regulatory authorities. Those regulation-driven exercises will always have the same objectives, right? Yes and No. Performing these exercises always in the same way with the same staff does nothing to grow and develop your continuity capability….even if you fully meet the letter of the regulation.

Once you have established that your recovery strategy does respond to your requirements, and that the recovery documentation is largely correct and complete, and that your recovery people are minimally competent to execute their recovery duties, THEN you can begin to perform much more useful exercises. You will probably need a few years or more of diligent exercising just to get to this point.

There are three main objectives around which you should be building your BCM longer-term exercise program:

  • Plan documentation correctness and completeness
  • Staff and/or vendor training
  • Increasing verisimilitude

I have seen BCM Exercise Programs where Black Swans so contaminate the programs that they are metaphorically covered in black, so dense is their population of Black Swans. If these BCM Programs are ever called upon during an incident, those Black Swans will probably seriously harm the ability of the organization to deliver the expected recovery result.

The first of these is outdated, incomplete, or just plain incorrect documentation. Completeness and correctness of recovery documentation cannot be verified by those who wrote it and who run the business operation or IT system on a daily basis. If these are the only staff members that you use for exercising, how do you know how much information is inside their heads and how much is in the recovery plans? Such plans are replete with Black Swans that are unseen and whose effects cannot be anticipated. And you do not want to encounter these Black Swans for the first time when your "A-Team" members are unavailable to assist with an interruption event. Then is not the time to find out that your documentation is incomplete…and your backup staff DOES NOT KNOW the necessary information. Varying your exercise scenario will also produce new and different challenges. A good one to try is re-synchronization of interlocking data when only some of it has been damaged and therefore recovered from backup.

The second objective is training. This means that you need to train your suppliers as well as your own people. And that you need multiple teams with depth and breadth of knowledge, rather than one expert team that becomes a single point of failure. Try this simple exercise:

  • Put pins on a local map charting were your key people live.
  • Look at where damage has occurred in the past – wind, flooding, power outages. Color the affected areas on the map?
  • How many of your key people live in areas susceptible to being cut off by floods or by other conditions that might prevent them from getting to your recovery site expeditiously and without warning? What about toxic spills that close arterial roads and highways?

If all of your key people for a given skill set live in the same neighborhood, get some diversity. Or you can of course choose to feed that Black Swan!

Verisimilitude is a fancy word that means "like reality" in this context. It means that sometimes things go bump in the off-hours and with no warning when your most experienced people are not available. It means that you do not always have the luxury of verifying the contact numbers and availability of key staff just before the test. And so it means that you need to rehearse under conditions that are more rather than less realistic. It means that you need to ensure that your "B Team" and "C Team" can cover when your "A Team" is on vacation in the Pacific or out sick or hurt in the event. It means that you need to define the event scenario in much more detail and with more realistic detail than is usually done.

It means that you need to start the RTO and RPO clocks when the event "happens" and that you cannot prepare for the exercise ahead of time. It means that you cannot order your backups from your offsite storage vendor to be delivered to your alternate site the day before the exercise. How will you know how long it will take during a real interruption if you never exercise that part of your plan?

Injecting realism through the development of detailed event scenarios where the most experienced staff is not available, where events occur in the wee hours of weekend mornings, where weather may have disrupted physical transport wrought havoc with people's homes: this is the way to achieve greater degrees of verisimilitude. Of course you need to do basic validation of your strategy through scheduled tests in order to do basic training and documentation verification. But if after a few years, you are still doing these same tests over and over...well you are sustaining a healthy population of Black Swans within your program.

Annual Exercise Program Objectives

Like any business department, you will need to set your budget for the following year. This is the perfect time to define what your exercise program will look like for the next year. You can work along multiple axes – training, documentation verification, and verisimilitude. You can employ a variety of exercise types. Let's say that you set your objectives for next year as follows:

Exercise the ability of staff to receive and respond to emergency messages. This addresses training and the quality of documentation (if you don't verify that contact information is correct just before running the exercise). You can easily run at least two or more notification tests per department or division per year if you are using an automated notification system that keeps an audit trail of system activity.

Do an unscheduled callback of backup tapes from your vendor at unusual times: try 3 a.m. on Sunday morning, or midnight on Tuesday. You will find out if your vendor can deliver as promised when they don't know about your needs ahead of time. And so this tests both your supplier's ability to meet contractual requirements, as well as your ability to meet your promised RTO's in a much more realistic way than if you had scheduled delivery a week ahead of time. If it will be impossible to meet those RTO's when you request tapes on an emergency basis, your business should know that.

Plan a small relocation test either to an alternate work location or to an IT alternate site….but do not tell those who will be participating ahead of time. Yes, these are more advanced exercises, with higher levels of verisimilitude. But guess what? No interruption event is planned ahead of time, so you need to know what is your ability to deal with an unplanned interruption. This kind of exercise will help you to get you there.

So when you set your specific annual exercise objectives, you should design the exercises you will be performing that year to advance those objectives. And once you have selected and designed the exercises you will be performing during that budget year, you can request the resources to conduct those exercises. Since you have already tied these exercises to specific objectives that advance your program to more reliable levels of operational resilience, you have already justified them. Over time, your management will learn the value of this approach and will be better able to understand the value of your exercise program to THEIR OBJECTIVES of operational resilience.

What you are really doing when you approach your BCM Exercise Program in this way is systematically identifying and eliminating the Black Swans in your program. Business management does not like surprises … and nor should you. This approach is the surest way to develop a high level of confidence that your plans and strategies will work as planned when you need them. And isn't that the true purpose of your Business Continuity Program?

About the Author:
Kathleen Lucey, FBCI, is President of Montague Risk Management, a business continuity consulting firm founded in 1996. She is a member of the Board of Directors of the BCI, and the founding President of the BCI USA Chapter. IBM chose her as the first winner of its Business Continuity Practitioner of the Year Award in 1998. She speaks and publishes widely in both North America and Europe. Kathleen may be reached via email at kathleenalucey@gmail.com.