Revealed: Details of 'First of Its Kind' Disruptive Power Grid Attack

Firewalls crashing, communications lost, and the realization that this was not a technology failure of the U.S. power grid.

It appears to be a first-of-its-kind cyberattack that reveals the risks of our increasingly connected infrastructure.

U.S. power grid cyberattack detailed in new report

An organization called the North American Electric Reliability Corporation (NERC) revealed details of the new type of attack in a four-page "Lesson Learned" document.

Here is what we know:

• The power grid attack happened during March 2019, in the western United States.
• Operators at a power control center started losing communication with "multiple remote power generation sites" for minutes at a time.
• Operators determined the problem: for some reason, internet-facing firewalls were rebooting and going offline.
• Each reboot severed communications for no more than five minutes between a controller and a generation site, however, this kept happening and the power grid cyberattack continued for nearly 10 hours.

Power grid cyberattack: the cause and blocking the attack

NERC's publication details a rapid investigation and (thankfully) an incident response plan that was quickly put into action.

This helped reveal what caused the attack and eventually stopped it. The power generator also reached out to its security vendor for help:

"After an initial internal investigation, the [power grid] entity decided that, in order to fully characterize the nature of the reboots and the potential causes, the firewall manufacturer should review logs."

What did the logs reveal? Apparently, a failure to patch.

"Subsequent analysis determined that the reboots were initiated by an external entity exploiting a known firewall vulnerability. After receiving this notification, the entity initiated their event reporting procedure as dictated by their cybersecurity incident response plan. Along with identifying the cause of the reboots, the firewall manufacturer offered a firmware update that would address the vulnerability."

This led to a significant judgment call on the part of the power grid entity. Do you install firmware updates in the middle of production? What if that causes a disruption or there is a technical glitch?

Then again, if you don't patch, the cyberattack will continue.

NERC details what happened next:

"The entity assessed the update details and determined it was appropriate to deploy immediately. The entity first deployed the firmware patch on a firewall within a non-critical environment at the entity's control center that would not impact operational assets and monitored the changes for any adverse effects.

After seeing no adverse effects, the entity deployed the firmware patch at an operational generation site that night. After monitoring traffic in the production environment overnight and early the following morning, the entity deployed the update to all remaining BES assets that had common hardware with the firmware vulnerability."

The patches worked, the firewalls remained secure, and communication between power control and power generation sites became reliable again.

Lessons learned from U.S. power grid attack

For one thing, we lucked out with this attack. It occurred against "low impact" sites and during March 2019. But what if it had been a key part of the U.S. power grid in the middle of winter or the peak of summer?

Lost power could have meant lives lost; that's what is at stake here. 

Here are a few cybersecurity best practices the North American Electric Reliability Corporation (NERC) lists in its post-event analysis of this attack:

• Follow good industry practices for vulnerability and patch management.
• Have as few internet-facing devices as possible.
• Use access control lists (ACLs) to filter inbound traffic prior to handling by the firewall.
• Layer defenses. It is harder to penetrate a screening router, a virtual private network terminator, and a firewall in series than just a firewall (assuming the ACLs and other configurations are appropriate).
• Employ redundant solutions to provide resilience and online maintenance capabilities:
  • Of the entity's sites impacted by the firewall reboot, not all experienced communications disruptions. Following the event, it was discovered that the sites running firewalls in high availability/redundant pair configuration maintained communications during the reboots. At sites utilizing this design, the secondary firewall maintained communications while the primary firewall rebooted.

And there is much more in NERC's publication: Lesson Learned: Risks Posed by Firewall Firmware Vulnerabilities.

Will anything else come out of this successful power grid cyberattack?

Does the power operator face any ramifications from the attack? Industry publication E&E News asked about fines related to patching failures in this case:

"Large power utilities are required to check for and apply fixes to sensitive grid software that could offer an entry point for hackers. NERC declined comment on whether the March 5 incident would lead to any enforcement actions, though the nonprofit has levied multimillion-dollar cybersecurity fines against power companies in the recent past.

Late last month, NERC announced it had reached a $2.1 million penalty settlement with an unnamed utility— also based out West—over a spate of cybersecurity violations dating back to 2009."

Regardless, perhaps this attack will make future attacks less likely as more organizations in critical infrastructure adopt cybersecurity best practices.

Was this the first disruptive power grid attack?

We've reached out to NERC for confirmation that this was the first known disruptive cyberattack on the U.S. electrical power grid. However, E&E News, an energy industry publication, believes that it is:

The "cyber event that causes interruptions of electrical system operations," as the attack was categorized in the jargon of DOE electric disturbance forms, made waves in critical infrastructure security circles as a first-of-its-kind case study.

No U.S. electrical utility is known to have experienced any disruptive cyberattack in the past, a surprising fact given that utilities routinely find themselves in the crosshairs of the world's most sophisticated hackers...." 

After reading all of this, are you surprised that a failure to patch is what allowed this cyberattack to work?