(CNN) -- Faced with another frigid winter and desperate to keep the lights on, Ukraine's power grid operator secretly imported bespoke equipment designed to withstand Russian electronic warfare attacks with the help of US officials, CNN has learned.
Engineers at U.S. tech giant Cisco spent weeks building and testing the new equipment in a lab in Austin, Texas, and delivered a prototype to Ukraine in the spring with the help of a U.S. Air Force plane carrying humanitarian aid, according to Cisco.
- Less ammunition, an artist imprisoned and Kyiv's new "foothold" in the south: the latest news from Ukraine
After Ukrenergo, the state-owned operator of the Ukrainian power grid, quietly confirmed that the new equipment was working despite Russian attacks on its GPS systems, Cisco sent Ukraine dozens of pizza box-sized hardware kits worth an estimated $1 million, which were installed across the country. Ukrenergo executives told CNN.
In this undated photo, an employee of Ukrenergo, Ukraine's state-owned power grid operator, works on the power grid. Credit: Courtesy of Ukrenergo
The new equipment, which had not been previously reported, could offer a crucial lifeline to Ukraine's power grid, which remains a key target of Russian attacks as the Kremlin's war enters its second full winter. Over the past two years, Russian missile and drone strikes have destroyed about 40% of the electrical substations and related equipment that Ukrenergo operates across the country, the grid operator told CNN.
In a rare cyberattack that has just become public, hackers connected to Russia's military intelligence agency, the GRU, caused a blackout in Ukraine in October 2022, according to U.S. experts.
"We expect them to continue, especially this winter," Illia Vitiuk, head of cybersecurity at Ukraine's SBU security service, said of Russian hacking attempts at power plants.
However, the problem that Cisco was intended to help fix is due to Russian radio jammers interfering with the GPS systems that Ukrenergo also relies on to manage the flow of energy in Ukraine.
- U.S. Defense Secretary Austin arrives in Kyiv to meet with Ukrainian leaders
Dinner near Stanford
The secret operation, described to CNN by sources inside Cisco, Ukraine and the US government, is the latest example of how the Biden administration has turned to US companies to help defend Ukraine, while trying to keep Washington away from a direct confrontation with Russia.
SpaceX has provided satellite coverage used by the Ukrainian military. Microsoft helped move huge Ukrainian government data centers out of the country before the invasion. The CEO of Denver-based data analytics company Palantir has boasted that the company's software has been used for "most targets" of the Ukrainian military in Ukraine.
- 'We are fully ready': Ukraine prepares for new Russian attacks on energy infrastructure as winter approaches
Officials from multiple U.S. agencies played a discreet role in introducing Cisco equipment into Ukraine, according to the sources. The Pentagon handled the flights, the Department of Energy helped coordinate the delivery of the equipment, and, according to Ukrenergo, the Commerce Department hosted crucial meetings earlier this year between a handful of U.S. and Ukrenergo tech executives and managers, who were eager to find new ways to defend their network from Russian attacks.
In February, during a dinner at an upscale steakhouse near Stanford University, Ukrenergo executives shared war stories with their contacts at Cisco, which has been doing business in Ukraine for years.
Ukrainian grid operators were facing a serious but underreported problem: constant GPS jamming that the Russian and Ukrainian armies use to interfere with guided missiles was also disrupting the visibility of Ukrainian power grid operators, who rely on GPS-based clocks to transmit information about the flow of power from one place to another.
Sitting at the table that night was Joe Marshall, a veteran investigator at Talos, Cisco's cyber intelligence unit, who listened intently to the Ukrainians explaining their problem over steaks and drinks. Marshall has been protecting the power systems of Ukraine and other countries from sabotage for years, but it has never faced a problem like Ukrenergo's.
After dinner, Marshall went back to his hotel and racked his brains looking for a possible solution.
"Time was a factor," he said. "We're talking about people's lives."
Marshall spent hours watching YouTube videos posted by an electronic warfare expert, and also received tips from U.S. officials and industrial cybersecurity experts at Cisco and elsewhere.
Cisco, the world's largest maker of computer networking equipment, had resources to spare. Marshall and his team of more than a dozen engineers set to work adapting a very common piece of equipment, called an industrial Ethernet switch, to the specific needs of the Ukrainian network.
Cisco estimated the cost of building materials and shipping the switches at $1 million, but the company said it donated the equipment to Ukrenergo for free.
- Winter arrives in Ukraine, but Kyiv adapts its tactics
Taras Vasyliv, who oversees the sending of power to Ukrenergo, likened the custom-made switches to a "flashlight" for a surgeon trying to operate in the dark.
The switch allows an electrical substation, which has the crucial task of converting power from high to low voltage, to communicate with other parts of the power grid. Most importantly, these switches were equipped with their own internal clocks that could calculate accurate time measurements, providing an element of redundancy and giving visibility to network operators even when GPS systems are down.
Otherwise, "you're blind," Vasyliv said in a telephone interview from Kyiv.
A building damaged by a missile at a high-voltage electrical substation, operated by Ukrenergo, which supplies more than 6 million consumers in multiple cities as part of the national grid.
Andrew Kravchenko/Bloomberg/Getty Images
Several of his colleagues have been killed during the war, Vasyliv told CNN, as the Russian military has shelled Ukrenergo's infrastructure. But keeping the lights on and avoiding the next airstrike motivates him to keep going.
"Do your job, and do it very well," he tells himself.
Sneaking the switch into Ukraine
A few weeks after the dinner in Silicon Valley, Marshall and his team had developed a prototype. To see if it really worked, Cisco had to figure out how to get them to Ukraine.
Marshall, a former Pentagon IT contractor in Alabama, turned to a U.S. official to find a flight departing in April from an East Coast military base. The flight headed to Germany before arriving in Rzeszów, Poland, a humanitarian and military support center located about 100 kilometers from the Ukrainian border.
From there, the prototypes were loaded onto a train to head to Ukraine, where they were discreetly handed over to Vasyliv and his team of engineers at Ukrenergo.
With its Kyiv offices partially destroyed by shelling, Vasyliv said its engineers tested the switch in a discreet office in western Ukraine.
"They looked like 1970s California startups and not a fancy lab," he explains.
The switches worked, and Cisco ramped up production so dozens more could reach Ukraine.
U.S. officials familiar with Cisco's project were reluctant to talk about specific shipments for fear of putting Russia on notice of its ability to thwart them. After all, the same GRU cyber-sabotage team that cut off electricity in Ukraine had previously damaged the servers of Polish logistics companies that provided services to Ukraine, according to Microsoft.
But over the course of three months last winter, the Department of Energy "identified, procured, and shipped" nearly 20 tons of electrical equipment to Ukraine on U.S. Air Force cargo planes, the department said in February.
- Russia parades Ukrainian prisoners of war as "volunteers" for its army, in apparent violation of international law
Years of Russian attacks on the Ukrainian network
U.S. officials often coordinate the delivery of key technology to Ukraine behind the scenes. The U.S. Department of Defense is paying SpaceX to provide its Starlink satellite service in Ukraine, the department said in July, without disclosing the contract price.
This photo provided by Cisco shows damage to a substation in Ukraine. Credit: Courtesy of Cisco
U.S. officials tasked with protecting their country's power sector have also been studying Russia's digital sabotage of the Ukrainian grid for nearly a decade, not only to help Ukraine, but also to make sure U.S. power companies know how to defend themselves against cyberattack techniques.
When the GRU first used cyberattack tools to cut off power to about 225,000 Ukrainians in the winter of 2015, according to a U.S. indictment and private experts, the Department of Homeland Security sent a team to Ukraine to study the forensic aspects of the attack. Another cyberattack that disrupted Ukraine's power supply in 2016 showed that the Russians were evolving their techniques.
On October 10, 2022, the GRU attacked an unidentified Ukrainian electrical facility, "causing an unplanned power outage" at the same time that the Russian military was launching airstrikes on electrical infrastructure nationwide, according to U.S. cybersecurity firm Mandiant, which responded to the attack. The extent of the power outage caused by the attack was unclear. Ukrainian officials told CNN that it can be difficult to distinguish whether the blackout is the result of airstrikes or a cyberattack.
But the incident raised the possibility that Russia's cyberattack unit was becoming faster in developing new tools to disrupt power supplies in Ukraine, accelerated by the pace and demands of the war.
Last year's cyberattack in Ukraine "demonstrates the evolution of enhanced and faster [operational technology] threat capabilities that could be leveraged in the US," NERC, the US network regulator, said in a statement to CNN, referring to cyber capabilities that target industrial equipment.
At least one of the Energy Department's elite research labs, which invests millions of dollars a year in anticipating new cyberattack threats to the US grid, will closely study the methods used by the GRU in the October 2022 cyberattack on Ukraine, sources familiar with the matter told CNN.