Customer Profile:
A pharmaceutical plant producing anti-infective treatments against serious and often life-threatening infections.
Background:
During extreme weather, this facility experienced a power failure that started on Friday and carried into Saturday causing their control systems to be unresponsive. When the plant’s generator failed, logs revealed that their systems did not recover, and their access to the VMWare infrastructure was exceptionally slow. When one of their three servers would not respond due to the single boot disk failure with no backup drive, they called Avid for emergency support.
Upon arrival, the Avid team found more than 38 VMs in a panic state since their storage had been lost and had exceeded the VMWare host’s ability to cache drive operations. Our engineers quickly accessed the plant’s environment of 3 servers, 1 SAN, 2 NAS devices, 4 network switches. 2UPS rackmount devices, a BMS control converter and multiple workstations to come up with a viable disaster recovery plan.
Because both backup NAS servers were in the same room, a deployed water sprinkler could have destroyed the client’s production services preventing any recovery at all. And proper credentials were required to decrypt the backups, further complicating the recovery. Loss of these passwords or equipment damage could have forced a ‘business rebuild’ or ‘product recall’ event for this plant. Fortunately, all systems were restored and made operational after Avid was able to fully implement their recovery plan.
Takeaways:
A disaster recovery plan assessment can help to identify areas of concern before facing a catastrophic event. Avid’s experienced team of engineers understands the elements needed to properly design a system to backup the critical data to minimize unavoidable downtime.
