‘Hero work’: how a savvy digital tripwire helped State uncover a Chinese hack

A recent Chinese-linked hack of U.S. government emails detected in June may have gone unnoticed for much longer were it not for an enterprising government IT analyst.

A State Department cybersecurity expert spearheaded an effort to implant a custom warning mechanism into the agency’s network more than two years ago in anticipation of future hacks, the officials said, shedding new light on how they spotted the breach, top State Department officials told POLITICO.

The tripwire-like alert went off almost immediately when Chinese spies targeted the agency’s Microsoft email systems in mid-June, enabling the agency to tip off Microsoft and the rest of the U.S. government to the sophisticated spying campaign. The hack, which Microsoft disclosed in July, still compromised the unclassified emails of top officials at the State and Commerce Departments, including Commerce Secretary Gina Raimondo and Nicholas Burns, the U.S. ambassador to China. 

The disclosure from the State Department underscores both how federal agencies are adapting to beat back increasingly sophisticated cyber threats — and how easily the Chinese hackers might have gotten away with the spying caper.

Christopher Painter, the former cybersecurity coordinator for the State Department under both the Obama and Trump administrations, said that while it was “great” the analyst spotted the potential issue, “these discoveries sometimes come down to luck.”

“In an odd way, despite all the advances we’ve had in cybersecurity … it sometimes comes down to one person seeing something that’s anomalous,” Painter said.

The State Department was the first to report the activity to the U.S. government and to Microsoft. The firm has said the hackers used a powerful digital key they stole via a cascade of internal security mishaps to breach more than two dozen organizations globally, and at least 10 within the U.S. — none of which spotted the intrusion until the State Department did.

The analyst who built this, whom the State Department officials would not name, did “hero work,” said Kelly Fletcher, the agency’s chief information officer and head of the bureau of information resource management.

The State Department’s actions likely prevented Beijing from gaining more extensive access to the private communications of key U.S. officials amid an intense period of diplomacy between the world’s two largest economies.

Since the State Department caught the hack, Raimondo, Secretary of State Antony Blinken, Treasury Secretary Janet Yellen and U.S. climate envoy John Kerry have all traveled to China.

As far as the State Department is concerned, the hack “began in June, and ended in June,” said Fletcher.

The incident also highlights the importance of in-house cybersecurity expertise, even as a growing number of U.S. federal agencies and private companies transition to external cloud computing services, like those provided by Microsoft.

The State Department has recently undertaken a multi-year effort to beef up its cybersecurity work, and agency officials said they would not have been able to catch the hack just a few years ago.

The U.S. government has not officially blamed Beijing for the hack, and Fletcher and other State Department IT officials would not comment on what the hackers were after or who they were. But Microsoft confirmed in July that it was Chinese hackers, and after returning from an official visit to China earlier this month, Raimondo also pointed the finger at Beijing.

“They did hack me, which was unappreciated, to say the least,” Raimondo said on NBC’s “Meet the Press.”

“I brought it up, clearly. Put it right on the table.” Chinese officials have not directly denied the hack, but have accused the U.S. of carrying out similar operations.

State Department personnel first built the alert — known internally as “Big Yellow Taxi” — roughly two years ago out of an abundance of caution, according to Gharun Lacy, deputy assistant secretary and assistant director of the State Department’s Diplomatic Security Service for Cyber and Technology Security. That’s when an agency analyst spotted a potential security gap involving an unidentified application that was connected to cloud email inboxes.

Fearful that the problem could be exploited in the future, the analyst flagged the problem to his advisers, prompting the department to work with Microsoft on building the digital equivalent of a tripwire.

Lacy said the alert went off a handful of times during the two ensuing years, and that each time analysts determined the alerts were false positives. But when it went off again in mid-June, State Department analysts quickly sensed something serious was afoot.

They saw the alert “fire in a cluster, a volume; that’s unusual, not quite what we’d normally see,” Lacy said.

While Fletcher and Donna Bennett, the department’s chief information security officer, flagged the issue to State’s leadership, Lacy said his team worked “24/7” over the Juneteenth holiday weekend with Microsoft to determine what was wrong.

The team eventually determined that it merited a “significant cyber incident response plan,” which once activated involves creating a task force to focus on the disruption. Lacy said in this case, it included sending a diplomatic courier overseas to “gather critical evidence,” bringing in law enforcement personnel, and using artificial intelligence to analyze the data from the incident. Officials from the Cybersecurity and Infrastructure Security Agency also worked to assess the attack’s full scope.

The State Department stood up the task force before the end of June, and Lacy said it remained active for a month afterward.

“All of that work over the course of several days led to us … the interagency, State, CISA, everyone pulling together that information that Microsoft needed to resolve this in their environment,” Lacy said.

Other federal agencies were also closely following the incident. Bennett said the CIO Council, a group of federal agency IT leaders, had “multiple meetings” each day after discovering the breach.

Microsoft issued a software fix for the flaw that had been exploited, but neither the company nor State were confident it would work.

“What you don’t want to do is push a patch and then begin to drink champagne,” said Fletcher. For a period, she said, “I was holding my breath.”

Ultimately, Microsoft determined the software fix had done its job, and it went public with the incident in July. A Microsoft spokesperson declined to comment on the company’s response.

The hack has generated significant criticism of Microsoft from lawmakers, government cybersecurity officials and security industry brass because only customers who had purchased an enhanced security license, known as E5, had access to the type of forensic trail necessary to determine whether a hack had taken place.

Under pressure from CISA, Microsoft later agreed to bundle a slate of basic security features into its core licensing packages, theoretically giving more users the ability to spot a similar campaign in the future. But the fact that State’s alert system was custom-built explains why other victims with E5 licenses did not spot the activity earlier.

When it comes to new types of data some companies now have access to, what matters is “how you use it,” said Fletcher.

Eric Goldstein, the executive assistant director of CISA, said in an emailed statement that cloud security providers have an obligation to provide strong security to their users, and that those users should hold their providers to account.

“At the same time,” he wrote, “it is essential for cloud computing customers to maintain responsibility for their enterprise cybersecurity” and invest in their own defenses, so that they can better analyze the data cloud providers share with them.

That is precisely what the State Department says it’s doing.

At the State Department’s Foreign Affairs Cybersecurity Center in Maryland, dozens of professionals sort through the daily threats reported by the diplomatic community, sent in at a rate of more than 1,500 incidents a month. According to a State Department cybersecurity official, granted anonymity to discuss details of the center, the building houses around 100 servers that pull in between 17 and 20 terabytes of security event information per day — a dramatic rise from the less than 10 terabytes of data pulled in daily five years ago.

Bryan Ware, a former assistant director for Cybersecurity at CISA during the Trump administration, said State’s tripwire was a great example of a new security approach government is pushing — known as “zero trust” — which involves recognizing that hackers are bound to slip through an organization’s cyberdefenses at some point.

“No, don’t trust that your firewall prevented everything,” Ware said. “freaking analyze your data, always be asking questions, and hunt for adversaries. That’s what State did, and it was awesome.”