A Guide to ISO 22398 for Crisis Management Testing

The importance of exercises and testing for ensuring business resilience

If the pandemic has taught us anything, it is that you can’t just plan for crises; you have to test plans consistently, under conditions that approximate the real-world crisis scenario. The failure to exercise and test resilience plans regularly is often given as a reason for the breakdown of business resilience processes during critical events.

The issue predates the pandemic. In the 2018 Deloitte study, Stronger, fitter, better: Crisis management for the resilient enterprise, 90 per cent of organisations reported confidence in their crisis management capabilitiesⁱ. Only 17 per cent of those organisations, however, had performed simulation exercises. 

Despite the central role of communications in crisis management, companies didn’t fare much better when it came to crisis communications. A 2016 Nasdaq public relations services study found that a majority of corporate communicators said that their company either lacked a crisis communications playbook (48 per cent) or were unsure of whether they had one (12 per cent)ⁱⁱ.

When looking at the best-practice measures organisations failed to take to prepare themselves for crisis, the picture only got worse. Sixty per cent of organisations did not role play or were unsure if they did. Fewer than half (48 per cent) were actively using a media monitoring platform. Only 24 per cent of company CEOs and other spokespeople were receiving annual media training. 

What should organisations be doing to prepare, instead? The expert consensus, here, is for organisations to “make maximum use of the controlled, risk managed environment of exercises and testing.” After all, the practice that builds familiarity and comfort with business resilience practices is only possible in such an environment. When real crises come, the time for practice is over.

In addition to breeding confidence in the crisis management system, program, and the overall competence of the organisation to protect and maintain its prioritised human, physical, and environmental assets, what other roles do exercises and testing play? Exercises and testing also: 

• Help crisis leaders to identify problems with and solutions to latent issues with crisis management programs and practices
• Reinforce the culture of crisis competency and the value of exercises and testing

Of course, any number of ways exist to conduct exercises and testing. Organisations might not understand which is right for them.

They are in luck. The resilience community came together, developing international standard ISO/DIS 22398, which lays out a best-practice framework for performing resilience testing and exercises. In turn, this guide outlines the most important aspects of the international standard, informing organisations how to get their own best-practice testing and exercise program up and running. 

Introducing ISO/DIS 22398

International standard, ISO 22398 describes the procedures necessary for planning, implementing, managing, evaluating, reporting, and improving exercises, as well as the testing designs needed to assess the crisis-readiness of an organisation. The standard itself consists of seven sections, in addition to a forward, introduction, and multiple informative annexes.

The introduction sets up fundamental principles for crisis management exercises and testing, such as the need for performance objectives. Objectives, here, include: 

• Orientation/demonstration.
  A simulated experience of an expected situation with the intent of increasing awareness of vulnerabilities and the importance of effective action in response to the simulated conditions.
• Learning.
  Acquiring knowledge, skills, or abilities by individuals or groups with the goal of mastery of specific competencies.
• Cooperation.
  Providing an opportunity for people to work together to achieve a common end result.
• Experimenting.
  Trying out new methods and/or procedures with the intent of refinement.
• Testing.
  Evaluating a method and/or procedure in order to assess which components are sufficiently developed. 

Further, the standard argues that organisations should codify specific policies stipulating that exercises, testing and, implementation procedures should lead to corrective action. To this end, organisations should:

1. Develop exercise performance objectives to define the direction and scope of exercises and testing.
2. Implement the procedures that trigger a review based on the critique of an exercise, test, and actual events. Scenarios should reflect the objectives of the exercise. 

Establishing the foundation and other key standard sections

Understanding theory is good, what matters, though, is practice. The standard excels in making pragmatic recommendations for tangible actions, too, e.g., what organisations need to do before performing tests and exercises. 

In the establishing the foundation section, the standard instructs complying organisations that they need to conduct a needs and gap analysis; the purpose of this analysis is to establish the need for exercises and testing in the first place.

Beyond that, pre-testing analysis effectively signals the role of exercises and testing in managing business risks. The practical import in that is it helps stakeholders (including senior leaders) understand that conducting exercises and testing is needed to manage risks. 

What questions might organisations ask to get started with this planning stage of the testing process? Common questions include: 

• Does the exercises and testing plan address requirements for exercises and testing?
• Can this plan promote consensus with interested parties?
• Does the plan offer an opportunity to reach and interact with its target group(s) and potentially address their interests?
• Does this plan provide an opportunity to address multiple issues in depth?
• Does this plan focus on key issues?
• Does the plan provide information tailored to the target group(s)?
• Is this plan practical and relatively easy to implement?
• Does the plan provide for information transfer at relatively low cost?
• Is this plan easy to update?
• Is the effectiveness of this plan measurable?
• Is this plan a good vehicle for education?
• Is this plan creating a constructive and supportive atmosphere?
• Is this plan an effective way to get publicity or increase public awareness?
• Does the plan conform to the organisation’s constraints? 

Indeed, the genius of the ISO standard, here, is that it enables organisations to move away from generic exercises to a more customised testing program better suited to managing their specific business risks. 

From that vantage, the gap analysis not only helps make the case for such a best-practice testing program, but it also indicates what kind of exercise (out of the many available options) that that program should be deploying.

Exercises companies might undertake include: 

The standard offers even more room for exercise customisation than that. Besides type, exercises themselves can be broken down into discussion or operations based. The former helps participants familiarise themselves with existing plans, policies, agreements, and procedures. 

Operations-based exercises, on the other hand, help stakeholders validate plans, policies, agreements, and procedures. They also allow for the clarification of roles and responsibilities as well as the identification of resource gaps in an operational environment.

Of course, even these two categories include multiple sub-categories, examples of which include: 

What are the testing stages?

So far, the standard has counselled the importance of testing and exercises for ensuring business resilience. It has also advised organisations to perform a needs and gap analysis to determine the kind of exercise that makes the most sense for their resilience needs. As mentioned, exercises consist of broad types (discussion- and operations-based), with multiple sub-categories falling under each. The remaining question, though, is what should organisations do once they have determined the type of exercise they need to conduct?

The standard doesn’t provide a play-by-play for each specific type of scenario. It does, however, give organisations a set of six generic stages through which exercises go through. Those stages include: 

1. Run-through.
   The organisation should carry out a joint exercise run-through prior to the start of the exercise in order to ensure that all members of the exercise team receive the same initial information. This review should be brief and contain only information that is vital to ensure that the participants can perform as planned during the conduct of the exercise. The lead evaluator should be a participant in this process. It is also critical that a similar review occurs with the control team to remain synchronised with scenario changes, and to facilitate the implementation of the exercise director’s guidance as the exercise proceeds.
2. Start-up briefing.
   The organisation should organise a start-up briefing, an integral part of the exercise hazard control. If a hazard is identified and cannot be eliminated, the first technique in hazard control is awareness. If the participants are not aware of the hazard, it is difficult to avoid it or “control the hazard” by maintaining the distance from the hazard, minimising the exposure to the hazard and maintaining a “shield” from the hazard. The organisation should clearly communicate the reasons for an exercise intervention (both crisis and non-crisis) to all participants. The start-up briefing should be used to avoid confusion between simulated and actual events.
3. Launch.
   The organisation should check the communications that will be used to launch, stop (temporary), and terminate exercises and testing prior to the scheduled launch. The methods for communicating launch, stop, and terminate exercises and testing should be explained during the start-up briefing.
4. Wrap up.
   The organisation should use the same communications for launching and temporary stop at the end of the exercises and testing. The start-up briefing should be used to ensure clear communication with the intent of avoiding confusion between simulated and actual events.
5. Post-exercise briefing.
   The organisation should organise a post exercise briefing in order to gathering information from actual exercises and testing. Critique of actual incidents and near-incidents will provide valuable information concerning the validity of the plan, the resources that were available, how the resources were used, and the transfer of behaviour learned in training. Every actual incident should be subjected to a critique and a review by key decisionmakers. The same format for the critique of an exercise or test will be used for an actual incident. During the post-exercise debriefing, special attention should be given to the functioning of the exercise organisation and the exercise planning process.
6. Observation.
   The evaluators of the exercise should have knowledge of the expected performance. They should have prepared observation forms, which should contain the exercise performance objective and allow for notes to be taken during the exercise. 

A closer look at what to do once the exercise is completed

The primary purpose of exercises and testing is to inform stakeholders which business resilience practices are working as planned and which are not. That is why the after-action report, the natural terminus of the (cyclical) testing process, is perhaps the most important deliverable of all. 

Most organisations would have heard of the after-action report, a staple of post-crisis analysis. The post-testing afteraction report does something similar, in that it (a) gives organisations an overview of the exercises and testing performed; (b) reports on any successes against performance objectives; (c) elucidates what went well; (d) lays out the issues identified; (e) lists subsequent remediation actions to be taken and by whom.

Of course, post-testing after-action reports differ in substance from post-crisis after-action reports; the former, by definition, details what happens in the more controlled exercise environment. What, then, are discussion points one might see in the former but not the latter? Discussions might include:

• The set-up and staging of the exercise (project management versus crisis management)
  – Experiences of the participants with respect to the set-up (first impressions and the evaluation forms)
• Exercise aims or objective of testing
• Constraints on the exercises and testing process
• Exercise performance objectives
• Type of exercises and testing
• Choice of a location
• List of preparation participants
• Expert opinion concerning the quality of the exercise
• Conclusions regarding the validities of the exercise and the durability of the exercise aims
• Evaluation of the exercises and testing performances
• Recommendations for the next exercise
• Self-reflection of the participants, taking into account the adaptation of the exercise aims
• Operational performances, competencies, and learning experience of participants 

Finally, the COVID-19 crisis upped the ante on business resilience: crisis management planning is no longer enough. In order to be crisis ready, organisations will need to build and promote best-practice crisis testing and exercise programs, as well.

International standard ISO 22398 provides a framework for such programs. The onus now is on the individual organisations, starting with their senior staff, to do the hard work of implementing these programs, giving them all of the resources needed, including advanced crisis management software like Noggin Crisis, to manage all stages of the crisis management lifecycle. 

Citations

i. Peter Dent, Roda Woo, and Rick Cudworth, Deloitte Insight: Stronger, fitter, better: Crisis management for the resilient enterprise.

ii. Seth Arenstein, PR News. PR News/Nasdaq Survey: Nearly Half of Organizations Shun Crisis Preparation. Available at http://www.prnewsonline.com/pr-newsnasdaq-survey-nearly-half-organizations shun-crisis-preparation/.