Venyu Blog

In the wake of a deadly EF-5 tornado that took the lives of more than 140 of its citizens, Joplin, Missouri has taken its dubious place in the roll call of U.S. cities devastated by natural disasters.

From Joplin’s ground zero, however and for data administrators who have to plan for disaster recovery like scenarios, there are lessons learned (and retold) by Mike McCreary, chief of services for Mercy Technology Services, whose IT portfolio includes St. John’s Regional Medical Center in Joplin. St. John’s and its data center were among the many buildings and homes completely destroyed in the May 22 tornado.

Speaking recently with Information Management, McCreary credits proactive EHR-associated initiatives in the weeks prior to the tornado that resulted in the vast majority of patient data, hospital records keeping, registration scheduling and pharmacy applications being migrated to a shared hospital data center offsite.

Following frantic text messages to ensure everyone on his IT staff was safe, McCreary and his St. Louis-based department went onsite at the Joplin facility the next morning to take the first step in the data recovery process, including gathering patient records amidst the loss of the entire 1,200 square-foot data center.

As related by Justin Kern in the aforementioned Information Management newsletter the onsite data center that was ruined housed a few dozen legacy servers, imaging equipment and older patient information sources that were not mission critical and had not yet been migrated to the larger, shared hospital system center. The hardware is useless, but McCreary is hopeful the recovery process can turn up historical data from the backup tapes and drives from machines racked by winds and rubble.

As it turned out planning and a bit of good timing spared St. John’s hospital from what might have been additional tragedy. In April, the Mercy hospital system, which includes St. John’s, opened a state-of-the-art data center for mission critical applications and clinical data for its 28 acute care hospitals across a four-state region.

The new data center, in Washington, Mo. (and backed up at another location), is about 250 miles from Joplin and was unaffected by the violent weather. Three weeks before the tornado hit, St. John’s went live on its scheduled switch to its electronic health records system, the last major hospital in Mercy’s system to do so.

Kern blends in observations from select industry analysts who collectively suggest that it’s the small steps, considered cumulatively, that effectively becomes the default disaster plan in the face of unpredictable events, as was the case with the tornado that destroyed parts of Joplin.

This includes Jason Schafer, research manager of data center technologies for Tier1 Research, who says risks like weather and geography are obvious for data center and IT managers. What is less clear is the ability to maintain data operations and quickly recover multiple systems from the top down, something symptomatic of data plans that omit day-to-day users or hurry the training process.

Commissioning, integration and final testing by operational staff at a data center is critical to effective emergency procedures.

“You may not be able to prepare for everything, but if you run through 10 or 15 different emergency scenarios and you know how the system reacts in every case, your ability to respond to an incident outside of your typical [emergency] list is enhanced,” says Schafer.

While the cost of protecting mission critical services can be daunting, the plurality of data and applications can be covered by lower cost options like replicating disk-based backups or the cloud, says Rachel Dines, a Forrester Research analyst who follows infrastructure and operations. For data center operations, Dines says the most resilient sites have n+1 or 2n (redundant) infrastructure for power, network feeds and cooling. When mitigating for disruptive events, the major risks to data centers are usually less outwardly catastrophic, Dines says.

“For many companies, this won’t necessarily arrive as a natural disaster, but in more mundane events like a power failure or human error that can cause a major data center ‘disaster,’” she says.

Amid the successful continuation of hospital operations after the catastrophic tornado, McCreary has changed his outlook on the risk-versus-finance argument often integral in disaster planning.

“We have folks reevaluating our disaster planning to measure theories against realities, to now understand the gaps between what our disaster plan was and how it stacks up against the truth,” McCreary says. “Coming out of this, God forbid we ever have anything like this happen again, we will be more ready for it.”

For more information on disaster recovery planning, contact Venyu today.