Historic Information Breakdowns

Risk managers study causes of tragedies to identify control measures in order to prevent future tragedies.  “There are no new ways to get in trouble, but many new ways to stay out of trouble.” — Gordon Graham

Nearly every After Action Report (AAR) that I’ve read has cited a breakdown in communications.  The right information didn’t get the right place at the right time.  After hearing Gordon Graham at the IAEM convention , I recognized that the failures stretch back beyond just communications.  Gordon sets forth 10 families of risk that can all be figured out ahead of an incident and used to prevent or mitigate the incident.  These categories of risk make sense to me and seemed to resonate with the rest of the audience too.

Here are a few common areas of breakdowns:

Standards: Did building codes exist?  Were they the right codes?  Were they enforced?  Were system backups and COOP testing done according to the standard?

Predict: Did the models provide accurate information?  Were public warnings based on these models?

External influences: How was the media, public and social media managed?  Did add positively or negatively to the response?

Command and politics: Does the government structure help or hurt?  Was Incident Command System used?  Was the situational awareness completed?  Was information shared effectively?

Tactical: How was information shared to and from the first responders and front line workers?  Did these workers suffer from information overload?


“Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained, as among savages, infancy is perpetual. Those who cannot remember the past are condemned to repeat it.”  — George Santayana

I add that in since few people actually know the source and accurately quote it.  Experience is a great teacher.  Most importantly, remembering the past helps shape the future in the right direction.

Below are a list of significant disasters that altered the direction of Emergency Management.  Think about what should be remembered for each of these incidents, and then how these events would have unfolded with today’s technology – including the internet and social media.

Seveso, Italy (1976).  An industrial accident in a small chemical manufacturing plant.  It resulted in the highest known exposure to 2,3,7,8-tetrachlorodibenzo-p-dioxin (TCDD) in residential population.  The local community was unaware of the risk.  It was a week before public notification of the release and another week before evacuations.

Bhopal Methyl Isocyanate Release (1984).  An industrial accident that released 40 tones of MIC.  There was no public warning.  The exact mixture of the gas was not shared so the first responders did not know how to treat the public.

Chernobyl Nuclear Disaster (1986).  An explosion at the plant and subsequent radioactive contamination of the surrounding geographic area. Large parts of Europe and even North America were contaminated.  The Communistiic regime hid the initial information and did not share information until another country detected it.

Hurricane Hugo (1989).  At the time, this was the costliest hurricane disaster.  There was an insufficient damage assessment that lead to wrong resource allocation.  The survivors in rural communities were not located and responded to for many days.  Much of the response was dependent on manual systems.

Loma Prieta (1989).  A M7 earthquake that injured around 3800 in 15 seconds.  Extensive damage also occurred in San Francisco’s Marina District, where many expensive homes built on filled ground collapsed and / or caught fire. Beside that major roads and bridges were damaged.  The initial response focused on areas covered by the media.  Responding agencies had incompatible software and couldn’t share information.

Exxon Valdex (1989).  The American oil tanker Exxon Valdez clashed with the Bligh Reef, causing a major oil leakage.  The tanker did not turn rapidly enough at one point, causing the collision with the reef hours. This caused an oil spill of between 41,000 and 132,000 square meters, polluting 1900 km of coastline.  Mobilization of response was slow due to “paper resources” that never existed in reality.  The computer systems in various agencies were incompatible and there was no baseline data for comparison.

Hurricane Andrew (1993).  Andrew was the first named storm and only major hurricane of the otherwise inactive 1992 Atlantic hurricane season. Hurricane Andrew was the final and third most powerful of three Category 5 hurricanes to make landfall in the United States during the 20th century, after the Labor Day Hurricane of 1935 and Hurricane Camille in 1969.  The initial response was slowed due to poor damage assessment and incompatible systems.

Northridge Earthquake (1994).  This M6.7 earthquake lasted 20 seconds.  Major damage occurred to 11 area hospitals.  The damage made FEMA unable to assess the damage prior to distributing assistance.  Seventy-two deaths were attributed to the earthquake, with over 9,000 injured. In addition, the earthquake caused an estimated $20 billion in damage, making it one of the costliest natural disasters in U.S. history.

Izmit, Turkey Earthquake (1999).  This M7.6 earthquake struck in the overnight hours and lasted 37 seconds.  It killed around 17,000 people and left half a million people homeless.  The Mayor did not receive a damage report until 34 hours after the earthquake.  Some 70 percent of buildings in Turkey are unlicensed, meaning they did not get approval on their building code.  In this situation, the governmental unit that established the codes was separate from the unit that enforced the codes.  The politics between the two units caused the codes to not be enforced.

Sept 11 attacks (2001).  The numerous intelligence failures and response challenges during these three events are well documented.

Florida hurricanes (2004).  The season was notable as one of the deadliest and most costly Atlantic hurricane seasons on record in the last decade, with at least 3,132 deaths and roughly $50 billion (2004 US dollars) in damage. The most notable storms for the season were the five named storms that made landfall in the U.S. state of Florida, three of them with at least 115 mph (185 km/h) sustained winds: Tropical Storms Bonnie, Charley, Frances, Ivan, and Jeanne. This is the only time in recorded history that four hurricanes affected Florida.

Indian Ocean Tsunami (2004). With a magnitude of between 9.1 and 9.3, it is the second largest earthquake ever recorded on a seismograph. This earthquake had the longest duration of faulting ever observed, between 8.3 and 10 minutes. It caused the entire planet to vibrate as much as 1 cm (0.4 inches) and triggered other earthquakes as far away as Alaska.  There were no warning systems in the Indian Ocean compounded by an inability to communicate with the population at risk.

Hurricane Katrina and Rita (2005).  At least 1,836 people lost their lives in the actual hurricane and in the subsequent floods, making it the deadliest U.S. hurricane since the 1928 Okeechobee hurricane.  There were many evacuation failures due to inadequate considerations of the demographic.  Massive communication failures occurred with no alternatives considered.


Additional resources


EOC Technologies

An Emergency Operations Center (EOC) is central to coordinating the resources for a response to an emergency.  While the term can be used loosely throughout government, public, NGO and private organizations and modified to fit their specific needs, there is a NIMS definition.  From the National Incident Management System, December 2008:

“Often, agencies within a political jurisdiction will establish coordination, communications, control, logistics, etc., at the department level for conducting overall management of their assigned resources. Governmental departments (or agencies, bureaus, etc.) or private organizations may also have operations centers (referred to here as Department Operations Centers, or DOCs) that serve as the interface between the ongoing operations of that organization and the emergency operations it is supporting. The DOC may directly support the incident and receive information relative to its operations. In most cases, DOCs are physically represented in a combined agency EOC by authorized agent(s) for the department or agency.”

An EOC for a locality brings together the numerous agencies into one spot for communications and coordination.  While it includes the fire department and law enforcement agencies, it also includes public works, health, utilities, transportation, volunteer organizations active in disaster (VOAD) and representatives from different private sectors.  Each of these entities brings valuable resources to an emergency response.  The EOC provides a physical location equipped with the tools and people necessary to manage the external resources for an incident.

Incident Command System (ICS) uses the EOC as the point of contact for any needs outside what is assigned to the operation.  An EOC may serve in this role for multiple incidents simultaneously.  The facility that houses the EOC may also be home to other centers used day-to-day or only during large operations, such as the Emergency Communications Center, Joint Information Center (JIC), Joint Operations Center (JOC), Multi Agency Coordination Center (MACC) and so on.

The Building

Given the core nature of the EOC to both daily first-responder work and crisis / disaster response, the building itself must be designed to function even in the worst circumstances.  Multiple redundancies are needed to cope with failures of the primary systems.

Electricity is usually the first redundancy that is thought of so there are many options there.  Positioning the building between two major power grids will mitigate the failure of one grid (or substation).  This limits the physical locations of the building since finding a location that can be served by two or more power grids may be challenging.

Redundant power through an on-site generator is a very common method for backup power.  A power transfer switch is installed in the building that will automatically switch from the power grid to generator power and back.  The power transfer switch isolates the generator from the transmission line which prevents a very dangerous situation called backfeeding.  Backfeeding is when electrical current goes backwards through the lines.  An electrical line worker may think the power is off on the line being worked on, but a facility creating a back feed will energize the lines creating a fatal situation.  The transformers that normally step down the power down to lower voltages will work opposite in backfeeding situations and step the power up to a high voltage.  I’ve experienced where the power transfer switch failed, and it took three days for it to be repaired.  Meanwhile, the building was not able to operate on either generator or the power grid.  Look for all bottle necks and single points of failure in the system design.

On-site generators can have multiple fuel sources.  Diesel engines are traditionally viewed as better to run at constant speeds for longer periods of time then gasoline engines.  Propane is another fuel source.  The downside is the need to store a sufficient quantity of fuel on-site to power the generator.

Natural gas generators can be installed and these run off the natural gas system in the community.  Natural gas systems have a high reliability because turning the system off would require trained professionals to turn it back on at every juncture.  It is expected in earthquake and flood catastrophe planning for these to be overwhelmed, but no worse then any other utility.

Alternative energy, such as wind and solar, is a way to offset a portion of the energy usage required by the facility.  It is rare that these can meet the peak demand on the site for an extended period of time.

Water and sewer systems may not be required to run the technical systems in the EOC, but are certainly required to operate kitchens, sinks and toilets used by the people in the EOC.  Drinking water may be stored at the site as a backup.  Contracts may be put in place to arrange for portable toilets.  Keep in mind that commercial kitchens require running water by local health code, if the EOC is that equipped.

Telecommunications — last but definitely not least.  An EOC should be a hub and contain at least one of every communication type used in the community and agencies represented in the EOC.  This can include terrestrial, cellular, two-way radio, broadcast radio, and satellite technologies.  It is important to be able to communicate with individual units, other sites, as well as other counties, the state EOC and Federal points.  The main land-line phone numbers in the location need to be re-routable remotely should the site go dark.  Only being able to forward a phone when you are at the site doesn’t help when the site is a smoldering hole in the ground.  This is especially true of the public safety access point (PSAP) phone number where 9-1-1 calls are routed.  Ensure that you have the direct line to the office in the telephone company that can reroute a phone number somewhere else.  And test this.  Often.  You’ll also need verification information such as the account number, circuit number, authorization codes and so on.  Have those written into your procedures.  Yes, you need written procedures so any authorized person can pick up and make things happen.

In short, the building should be ready to operate as an island unto itself in the worst situations.  Once this structure with all the redundancies is completed, it needs to be replicated.  Ultimately, even the EOC could be in the path of destruction and require a backup.

A Dark Room

As a side note, when an EOC goes active and will be staffed around the clock for many days, consider where workers can take a break.  While I was at the Pentagon for the response to the September 11, 2001 attacks, a tent was setup with cots and not much light.  This was a quiet place away from the noise and sights where individuals could go to disengage for a few moments.  It was frequently used during the first week when many people were working long shifts.  This is also helpful for permanent facilities.  During the Hurricane Katrina response, a few cots were setup in an out of the way conference room that allowed staff to take a moment for themselves too.  Keep the human in humanitarian response.

Pretty Sparkly Things

Technology must facilitate decision making or else the value is limited.  Technology that slows down decision making hinders response.  All this occurs at the user interface.  The user interface is what the workers see.  Everything else is like the wizard behind the curtain — most people won’t even know it is there.

Consider for a moment how many EOCs (and similar facilities) have a large radar or satellite image of a hurricane, yet they don’t employ a weather forecaster who can interpret the image.  There is a certain amount of technology in an EOC that lends to the visual appeal, otherwise known as the theater of disaster.  Admit and accept the fact that some technology will be installed in an EOC to make it more appealing for tours and media.  One aspect of an EOC is to be a known location for media conference which helps tell the story of how they are helping the people in the community.  The works in the EOC know who employs them; the big sign on the wall with the location name and graphic is for the visitors.

Every permanent site should have a list of all technology being used in a rank-ordered prioritized list with the most critical aspects listed first.  Get this list approved by all stake holders.  When system failures occur, this list will be the order that items are restored at the site or brought up at the alternate site.  Having this determined ahead of time will mitigate some of the ways that you and your team will be pulled during system failures.  It provides a common planning sequence which allows independent operation when a leader is not there to give the orders.  Remember, technology doesn’t fail users; technology fails to meet the users’ expectations.  Set the expectations correctly and you’re more likely to be considered a success even when the technology crashes.

IS-775, EOC Management and Operations

Now is a good time to mention that FEMA’s Emergency Management Institute has a self-study course on EOC Management and Operations.  It is IS-775.  This course gets into more details about the role, design and function of an EOC.  The course can be found on EMI’s website at http://training.fema.gov/EMIWeb/IS/is775.asp.

The Basics

At the core, the EOC needs to be equipped with systems that allows C3: Command, Control and Communications.

As mentioned earlier, the technology must assist in decision making and increase the effectiveness of a disaster response.  Therefore, the systems must be easy and efficient to use.  Success in this area is obvious because the system will be used.  Failure in this area leads users to make work-arounds, after-the-fact data entry, or simply not using the system at all.  A good user-interface is critical for data to be accurately input, and the right information to be returned at the right time.

The system should be able to identify trends in real-time that lead to failures, and provide notifications with recommendations so timely action can be taken to mitigate the failure.  This could be anywhere from a critical failure of life-safety services to the “minor” issue of accurate records.  Automation of data collection can create robust data sets that can be used for analysis and ease the burden on the workers.  These systems become the “system of record” that will be used during audits and legal proceedings so ensure the information is captured consistently and accurately, and then backed up using current best practices.  It must produce the documentation that may be requested after a disaster to assist with any sort of claims, reimbursement, and research.

Systems kept behind glass with “break in case of emergency” writing will fail when used.  Odds are good that the users will not be familiar with the systems or know when to shift from the daily systems to the crisis systems.  Require the system to be scalable so it can be used on routine daily events and grow to the massive response.  The system will need to track multiple incidents at the same time using resources from the same resource pool.  A process that will allow the same unit to be dispatched to different incidents at the same time creates confusion and therefore increases risk.

Sharing the data will be important when other governmental units get involved.  For instance, allowing data to pass to the financial systems will make it easier to capture costs, categorized accurately, adjust budgets and handle overtime pay.  Passing data to the maintenance section will highlight which resources have been used (abused) more than expected recently which could shorten the time to overhaul and inspection.

The data captured should be geotagged to allow analysis and view on a GIS system, and integrate with predictive models.  Example: capture the real-time location of all resources (facilities, people and equipment) using GPS or similar technology, then include that in all overlays of hazard modeling.  It will instantly show which units may no longer be in a safe zone and need to be relocated.

Make it mobile

Now that all the data is at the EOC — or the EOC is the hub for a community-wide network of information gatherers — it is time to push the data back to all the users.  How will this information be passed back to all the departments, units and individuals that need it?  Who needs what information?  There is a good chance that a lot of data will be transmitted wirelessly to vehicles with onboard computers.

Will it be a push model that data will be automatically pushed to the user’s attention, or will it be a pull model where the user needs to request the information?  The system will be both a push and pull in reality.  Life-safety and other critical data must be pushed out to the users as soon as it is known.  Other data may use a pull format so the user isn’t constantly streamed information when they are not ready to receive it.

A third way to handle data is with triggers.  When a particular action occurs then push the data.  For instance, when a unit marks itself as at the end of a shift, the system may produce a list of reports that need to be completed.  In another situation, when a unit is assigned to a particular type of response, a safety reminder specific to that response may be generated.

Make it virtual

The final step is to make it virtual.  If all the systems at the EOC are technologically based then it should be just a matter of programming to make the systems operate in a virtual environment.  With the right tools and resources, the EOC can go virtual and mobile in a command vehicle.  The Chicago Office of Emergency Management built a Unified Command Vehicle.  The “doomsday” scenario included this vehicle taking over the EOC responsibilities.  More often, this vehicle is sent to large events to bring some of the EOC capacity to the site and provide redundant communications to relieve the burden from the operators at the EOC.

Make it upgradable

When designing and selecting an EOC system, design an upgrade path.  Interconnected systems should not be over limiting to the future expansion of the system.  A few years ago, a T1 circuit was the gold standard for business connectivity.  That same circuit today barely covers the needs for a single small unit.  The FCC moves to narrow banding or shifting radio frequencies should not require an entirely new system.  Portals in the system should be compatible with various implementations of the national broadband plan and social media integration.

The rate of technical change today is rapid.  Purchasing a proprietary system with a single upgrade path dependent on a single company’s ability to keep pace is probably not the best choice.

Bandanas and earrings: My memories of Sept 11, 2001

Camp Unity loglI have the utmost respect for the people in New York who worked in and responded to the towers. For the people near the Pennsylvanian field. For the people who were saved, and for the people who tragically lost their lives. I remember that 9/11 ended horrifically for many people in three locations, and the people who knew and loved them are all over the country and around the globe. For me, my memories of 9/11 is the Pentagon.

As write, it is late at night on September 11, 2009. I’ve been thinking on and off today about where was eight years ago. I passed someone in the hallway at work and she said “happy anniversary, we met eight year ago today.” Another friend reflected to eight years ago on Facebook and my simple comment back was “bandanas and earrings 🙂 .” I hear other people’s stories about their experiences and many center around watching TV for days and weeks straight. I didn’t watch much TV during September, and I didn’t see the constant replay of the tragedy. I believe am lucky in that respect. I don’t consider my experience of September 2001 special, yet do feel I’m one of small cadre of people with perspective different than most. I spent the day of September 11, 2001 in the Arlington County Emergency Operations Center. Then spent the rest of September’s nights at location that became known as Camp Unity until the response and recovery phase was completed and the American Red Cross left the Pentagon parking lot.

Continue reading Bandanas and earrings: My memories of Sept 11, 2001