It's going to be a long day, weekend, and likely weeks or longer for cybersecurity and IT professionals. In a significant disruption to operations across various industries, a recent software update from CrowdStrike has caused Windows systems to crash, resulting in the notorious Blue Screen of Death (BSOD).
The incident, which emerged today, has had far-reaching impacts, taking out computer systems at airlines, hotels, and other critical services.
CrowdStrike, which boasts having the "definitive AI-native cybersecurity platform" on its website, issued a routine software update to its Falcon product aimed at enhancing the security and functionality of its endpoint protection platform. However, shortly after deployment, users began reporting that their Microsoft Windows systems were crashing with BSOD errors.
The fallout from the update has been extensive, affecting a wide range of sectors around the world, including:
- Airlines: Several airlines experienced system outages, leading to delays and disruptions in flight schedules. The affected systems included those used for ticketing, check-in, and other critical operations.
- Hotels: Major hotel chains reported issues with their reservation and management systems, causing inconveniences for guests and operational challenges for staff.
- Other services: Businesses reliant on Windows systems for their day-to-day operations, including healthcare providers, financial services, and retail operations, also reported significant disruptions.
CrowdStrike's response
Upon recognizing the issue, CrowdStrike quickly acknowledged the problem and began working on a fix. The company issued the following statement:
"We are aware of the issues caused by our recent software update and are working diligently to resolve them. Our priority is to restore normal operations for all affected customers as quickly as possible. We apologize for the inconvenience and appreciate your patience."
George Kurtz, Crowdstrike's CEO and Co-Founder, issued this statement on LinkedIn early this morning after organizations in Australia first began reporting outage issues:
"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they're communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers."
While the exact technical details are still being investigated, preliminary analysis suggests that the update conflicted with specific Windows system files, leading to the BSOD. Such incidents underscore the complexities involved in software updates, particularly those related to cybersecurity tools that interact deeply with operating systems.
For users affected by the update, the best recommendation is to follow protocols issued by CrowdStrike—although there are a host of social media posts with various solutions, some reporting they work, others are utterly frustrated with remediation efforts. Here are some immediate steps to hopefully mitigate the impact:
- Revert to Previous State: Use system restore points to revert to a state before the update was applied. This can help in regaining system stability.
- Safe Mode Boot: Booting into Safe Mode can allow users to uninstall the problematic update without encountering the BSOD.
- Contact Support: Reach out to CrowdStrike's support team for guidance and updates on the resolution process.
- Backup Critical Data: Ensure that all critical data is backed up to prevent any loss during troubleshooting and recovery efforts.
Joseph Lewis, CISO & Director, Cybersecurity Program Office, at the
U.S. Centers for Disease Control and Prevention, wrote this on LinkedIn: "CISO's thought of the morning... don't sleep. If I hadn't slept I wouldn't have woken up to this disaster with CrowdStrike wreaking havoc all over my agency. All jokes aside, this is pretty serious. It's like a ransomware attack without the ransom (or the ware!). Deep breaths, everyone.. it's gonna be a challenging day. Keep an eye out on your people... they're going to need leadership support today."
This incident serves as a critical reminder of the importance of robust testing and contingency planning in software updates. Even well-established companies like CrowdStrike can encounter unforeseen issues that have wide-ranging impacts.
Violet Sullivan, AVP, Cyber Solutions Team Leader, at Crum & Forster, posted this on LinkedIn as her own flight scheduled for today was delayed:
"CrowdStrike update pushed out in the middle of the night has had global IT outage issues. Key Impacts in the U.S.:
✈️Airline Operations: American Airlines and others faced temporary halts and delays but have since resumed operations. I know because I'm heading to my delayed flight right now.
🏥Healthcare: Significant disruptions in GP practices in the UK; similar impacts likely in U.S. healthcare systems due to global IT interconnectivity.
💻IT Services: #CrowdStrike's Falcon platform login issues widely reported.
This isn't just an endpoint security issue— this is causing widespread manual workarounds and delays in everything from 911 services at municipalities to grounded flights.
8:00 a.m. update:
"The fault has been linked to a misconfigured .sys file pushed by CrowdStrike to customer devices running the Falcon endpoint sensor.
While the misconfigured file was automatically propagated to endpoint devices, there is no way to similarly automate the fix for crashed computers. IT staff will have to manually restart each workstation in safe mode and remove the offending software."
🚑🚒Some reports say that CrowdStrike's update impacted some 911 emergency service agencies in the state of New York (EMS, police, fire department), Alaska, and Arizona, as well as 911 services in parts of Canada."
Cybersecurity and IT teams are scrambling, with some able to make the needed fixes to smaller systems, while others are bringing in extra help to plow through the dearth of work ahead of them to get systems at larger organizations back online.
Derek Fisher, Executive Director of Product Security at JPMorgan Chase & Co., said what a lot of cybersecurity professionals are feeling in his LinkedIn post this morning:
"Woke up to a fair amount of chatter this morning regarding the global IT outage stemming from a defect in CrowdStrike's Falcon product and impacting services from #airports to banks across the #globe.
This shows the cascading consequences of defects or vulnerabilities within widely used systems and third-party dependencies. Even those designed to protect those systems.
Notably, the outage affected not just individual companies but critical infrastructure worldwide, including significant disruptions at major U.S. carriers such as Delta, United, and American Airlines, and spread to #banks and #healthcare services across various continents. Emergency government meetings, like the UK's COBR session, underscore the severity of the situation.
Meanwhile, companies and governments are rushing to mitigate impacts, which continue even after Microsoft declared the underlying cause fixed.
The outage emphasizes the web of the modern #digital interconnection and interdependence that can lead to a #global domino effect.
If you're running Falcon, the recommendation is go to the support portal for more information."
This seems to be a routine bulleted list added to most SecureWorld News articles about breaches and third-party software-related incidents, but here are key takeaways/advice for organizations:
- Pre-Deployment Testing: Ensure thorough testing of updates in a controlled environment before rolling them out to production systems.
- Incremental Rollouts: Consider phased rollouts to catch potential issues early and limit widespread impact.
- Backup and Recovery Plans: Maintain up-to-date backups and have recovery plans in place to quickly restore operations in case of disruptions.
- Vendor Communication: Establish clear communication channels with vendors to receive timely updates and support during incidents.
This GitHub post offers this "Automated Workaround in Safe Mode using Group Policy."
Several cybersecurity vendors who partner directly or indirectly with CrowdStrike are messaging customers and providing LinkedIn posts to try to help. Here's a LinkedIn post from ThreatLocker:
"As many are aware, there has been a situation with CrowdStrike whereby a faulty channel file has caused many Windows computers to have a blue screen.
ThreatLocker has not been affected by this issue, as we do not use CrowdStrike internally. However, ThreatLocker and CrowdStrike have numerous mutual customers.
Read here for potential solutions (link was added to the ThreatLocker site)."
Cybersecurity experts are chiming in with commentary, as well.
"It is still too early to judge how such an error occurred, and whether a code fault with the driver or an unanticipated and undocumented change in the Windows Operating System, which CrowdStrike was unable to predict, is responsible," said Rob Reeves, Principal Cyber Security Engineer at Immersive Labs. "It is clear, however, that the heavy reliance on Falcon has become a double-edged sword and is causing untold disruption to business operations worldwide. The severity of this incident serves as a stark wake-up call, highlighting the critical need for rigorous and dependable testing of EDR and ELAM drivers in cybersecurity systems. Now more than ever, it is crucial to reassess and overhaul current testing procedures, swiftly identifying and addressing any issues that arise.
"This prompts reflection on whether security product updates should be automatically applied universally for up-to-date protection or if customers should maintain control over the update process, ensuring thorough testing prior to implementation," Reeves said.
"It's difficult to attribute this outage to one specific issue. Last night there was an issue with Microsoft's cloud service, and this morning through a CrowdStrike update. This caused the perfect storm to deliver a global outage," said Nick France, Chief Technology Officer at Sectigo. "CrowdStrike is a security software, which many companies use and have on their systems, so when an update happens that causes an issue, it can affect companies worldwide. If those companies went offline to respond in case it was due to a cyber attack, they now need to reboot their systems to patch the update issue. Bad updates can happen, and this shows the global impact they can have when coming from an integral piece of software, such as CrowdStrike."
One LinkedIn poster provided a tongue-in-cheek video showing cubicles full of laptops and monitors with the quote, "You've been CrowdStruck," playing off the AC/DC song, "Thunderstruck."
Others on social media are imploring competitors of CrowdStrike to NOT "strike" while the iron is hot by reaching out to CISOs and the like to offer their alternative solution.
George Kamide, Co-Host of the Bare Knuckles & Brass Tacks podcast and Head of Product Marketing at Cinder, offered this bit of advice in a LinkedIn post:
"Security and IT teams are in for a very long weekend. Security teams and leaders already face a lot of friction from biz ops and revenue teams. The idea that security software has bricked many machines is not going to help. I hope we can get past the schadenfreude and fingerpointing, and get to the work of getting systems back online. Because my friends stuck in airports today don't given a damn whose 'fault' it is right now. They just wanna get home.
Reach out to offer assistance. Do they need food delivered? Could they use a comms template you're using? Do you have comms skills they could use? They may not avail themselves of it, but let them know you're thinking about them.
If you're on the vendor side, today is not a day for cold calling or ambulance chasing. It might be a day to re-examine and detail your QA and testing processes."
CrowdStrike was founded in 2011 with the mission to address sophisticated cyberattacks with advanced endpoint protection and expert intelligence. More than half of Fortune 500 companies use CrowdStrike software, according to the company.
CISA is offering an alert, as well.
