Three Things the Oil Spill Can Teach Us About Disaster Recovery
- Date: 26 July 2010
- Author: broyer
- Category: BC/DR
Maybe it’s overstating the obvious but hidden in even the most dire of circumstances ― often events of devastating magnitude ― are lessons for IT professionals on making their already demanding environments more resilient, reliable and yes, disaster proof.
In a recent article in eSecurity Planet, Diana Kelley writes that it’s just not enough to pay what amounts to lip service to disaster planning and recovery when you produce a plan for it but never actually execute, even in a test environment, to ensure that it will truly work as expected. As Kelly contends the Gulf Coast oil spill was a convergence of the failure to put into place adequate controls for preventing a catastrophic event and a poor to non-existent business continuity and disaster (BC/DR) recovery plan.
Although the symmetry between the two may not be readily apparent, there are similarities. For example, we’re all familiar with backups until the point they become just another checked box at the end of the day. But how many of us actually validate the results on a regular basis to ensure the data is not only present, but also properly restored? It’s easy to make the rudimentary routine –until the data is required (for compliance purposes, for examples), and you discover not only is it not there but worse yet, you also have no idea where it went.
In taking BP to task Kelly’s observations fall along the same fault lines that are easily attributable to any IT professional finding themselves in a potentially problematic situation.
1. Get the Consequence Cost Right
The cost of implementing BC/DR controls should be directly proportional to the loss of business risked by their absence. In other words, if you don’t have adequate controls in place and you suffer a devastating loss there is almost nothing to gain by downplaying the potential consequences. As an organization be nimble and perceptive enough to assess the impact of your loss at benchmarks along the way that make sense to your organization. Evaluate your organization and the practical cost of consequence scenarios and then deploy a level of BC/DR controls consistent with the results of that evaluation.
2. Go for value, not volume
The BP response plan comes in at a “hefty” 583 pages and while that appears to be consistent with the level of response required given the devastation, Kelly believes that any BC/DR plan also needs to be able to articulate key decision points quickly so responders can act accordingly (e.g. a corporate policy that states shutting down the e-mail server if a virus has found its way in and is re-transmitting itself company-wide). And while Kelly concedes that even the most thorough BC/DR plan can’t anticipate unexpected circumstances, taking the steps to document what you can control in the face of a crisis is certainly preferable to being caught completely unawares.
3. Practice makes (almost) perfect.
As media reports surface about the level of readiness those on the oil rig had in place for just such a disaster, most especially the crew’s weekly evacuation drills, it’s clear that many lives were saved by practicing for the least likely but most dramatic outcome. Kelly likens this readiness to validating data backups, where practice not only confirms the data has been backed up correctly, but also enables an administrator to easily find specific data called up for business reasons.
The lesson in all this, and one I agree with conclusively, is to never be complacent. By bringing your BC/DR plan to life ― in other words not taking it for granted ― when a serious attack on the network does occur you won’t be scrambling to implement basic response activities, and too late after the devastation has already incapacitated your business.