As some may have learned the hard way, being stuck at Airports, Train Stations, or other major transportation hubs around the world, a global IT issue wreaked havoc across the globe last Friday. From Airports, to Hospitals, to digital signage, and nearly anything else running a Windows computer which utilized protective software from a company called Crowdstrike, users across the world were greeted with the dreaded Blue Screen of Death.
What Caused the Bug?
Systems that were affected by this issue were specifically Windows based devices utilizing the Crowdstrike protection software called Falcon. Falcon is a Crowdstrike security platform widely used by organization’s of all sizes. Falcon integrates itself very deeply into the Windows OS, allowing itself to make changes to Windows kernel processes.
A logic flaw in a new version of the Falcon software caused it to crash. However, because the Falcon software is integrated so deeply into the Windows OS, it resulted in Windows itself crashing, hence displaying the Blue Screen of Death.
Microsoft estimates that roughly 8.5 million Windows devices were affected by the errant Crowdstrike Falcon update. Many of the systems affected were running critical tasks. Some of the affected industries included: Airlines, Airports, Public Transit, Healthcare, Financial Services, and Multimedia to name a few.
Recovering From the Outage
Crowdstrike was able to quickly identify and deploy a fix for the issue approximately 1 hour after the issue was discovered. However, the recovery process for businesses will be time-consuming. The main problem being that once systems running Falcon updated to the errant version, Windows would undergo a Blue Screen, making it impossible to automatically fix the issue.
IT Admins dealing with this bug had to manually reboot affected systems into Safe Mode. Once Windows was botted into Safe Mode, IT Admins then had to delete the faulty files delivered by the update, and then reboot to restore normal operations. It is estimated that it could take months for some organizations to fully recover all their devices.
How To Prepare for Outages Like This
Living in a world where technology is integrated, we can see how one small bug can take down entire industries for hours, days, or even longer! Preparing for when the inevitable outage occurs is tantamount for any organization that takes their business seriously.
Utilizing local and off-site cloiud backup’s for your systems can ensure that in the event that one or more of your workstations or servers becomes inaccessible, the data is still recoverable.
Utilizing a test base environment before installing updates can also ensure that you can prevent problematic updates from affecting the bulk of your systems. Using a system of update rings to manage the flow of updates to your systems can help ensure that updates that may cause problems with your systems are caught early on.
Disaster Recovery and continuity plans are critical for business to be able to continue operations even in the event of system wide outages or disaster. Having a cloud backup can ensure that users can still access their files and systems even in the event that their office workstations are unavailable due to an outage caused by a bug, or in case of disaster like fire or flooding.
Finally, partnering with a Managed IT Services provider like Natural Networks can help ensure you stay ahead of the curb, so you and your IT infrastructure can stay protected in cases like these. If you want to learn more about how Natural networks can help protect your critical business technology, give us a call today!
