US agency smashes innocent computers, TVs and even mice to squash itsy-bitsy malware infection

The US Department of Commerce, in a misguided attempt to wipe out what it mistakenly thought was a massive infection, has crushed, killed and destroyed $170,000 worth of printers, TVs, and even humble little computer mice.

It would have darn well kept right on going in its quest to very literally, very physically pulverize a weensy bit of malware, but, oh well, it ran out of money.

That’s actually a bit of a relief, considering that the rest of its terrified IT gear, quivering in terror but ultimately spared the chopping block, was valued at $3 million.

A pull-no-punches report [PDF] (title: “Malware Infections on EDA’s Systems Were Overstated and the Disruption of IT Operations Was Unwarranted”) from the auditor of the Commerce Department’s Office of Inspector General (OIG) that was published earlier this year recounted the hardware slaughter, which was triggered by a Department of Homeland Security (DHS) warning of potential malware infection.

Essentially, the Commerce Department agency in question, the Economic Development Administration (EDA), freaked out after getting tipped off by Department of Homeland Security’s Computer Emergency Response Team (CERT) that there was a malware outbreak in December 2011.

But wait, it gets more thump-your-head-on-the-desk-ier.

That $170,000 tab only accounted for the department’s disposal budget. The actual bill, which included hiring an outside security consultant, was about $2.7 million.

The carnage and recovery took close to a year, with the expenditure of taxpayer money including these expenses:

• $823,000 for the security contractor’s investigation and advice,
• $1,061,000 for temporary equipment and email/Internet access (requisitioned from the Census Bureau),
• $4,300 to destroy the $170,500 worth of equipment,
• $688,000 paid to contractors to help develop a long-term response.

All that for what turned out to be garden-variety infection on a handful of PCs that easily could have been reimaged, it turns out.

Here’s how it went down:

On 6 December 2011, CERT sent the infection warning to the Commerce Department’s Computer Incident Response Team (DOC CIRT), which subsequently determined there was an infection in systems at the Herbert C. Hoover Building.

DOC CIRT informed the EDA, along with the National Oceanic and Atmospheric Administration (NOAA), that their IT systems were potentially infected.

Within a few weeks, NOAA had pinpointed the infected component, cleaned it up and placed it back into operation.

The EDA, on the other hand, came to believe its systems were riddled with infection.

The department, fearing it would infect everybody else, requested that the Commerce Department pull its plug out of the wall by quarantining its systems from the building’s network.

That shut down its operational capabilities, including email and Internet access, access to database apps, and access to any information kept on servers on the building’s network. Hence, the need to requisition the Census’s infrastructure.

According to the auditor’s report, the fault doesn’t lie entirely with the EDA. Mostly, but not all.

Rather, its folly was instigated by bungled communications.

The miscommunication started with the DOC CIRT’s initial notification on 6 December 2011. An inexperienced incident handler goofed, unknowingly requesting the wrong network logging information, which led to a malware notification that listed 146 network components.

That’s a lot. That’s actually more than half of all the EDA’s IT gear.

Two days later, a building network staffer corrected that number. It wasn’t 146 virus-ridden components; rather, it was a mere 2 components that were behaving maliciously.

So DOC CIRT sent a second email to the EDA, correcting that earlier, wildly inflated number.

Unfortunately, the second email was hazy: it lacked a clear explanation that the first notification was way off, the report said:

“Nowhere in the notification or attachment does the DOC CIRT incident handler identify that there was a mistake or change to the previously provided information.”

Five weeks of back-and-forth ensued, with the DOC CIRT thinking they were talking about two funky components and the EDA continuing to think that the sky was falling, and nobody really checking to make sure the initial correction had clearly gotten through to the EDA.

And so it went, on and on, for almost a year, with the EDA hiring an outside security contractor to look for malware and to assure EDA that its systems were clean and impregnable, the contractor finding malware on six systems that were easily repaired by reimaging, and the contractor ultimately declaring the systems pretty clean—not that the contractor wanted to call anything “impregnable,” mind you.

EDA’s CIO figured the agency was under attack from a nation state, and that’s when the shards started to fly. The agency destroyed uninfected desktop computers, printers, cameras, keyboards, and mice.

Finally, the destruction budget ran dry. EDA actually intended to resume the wholly unwarranted destruction of its IT systems, which were infected by, at the very worst, common malware, once funds became available.

There’s plenty of blame to go around. The auditor found that DOC CIRT staff failed to properly document the initial incident response activities, to establish the extent of the malware infection, and to perform a required containment procedure, on top of having inexperienced staff mucking things up.

The EDA takes the lion’s share of the blame, though. The auditor concluded that the agency decided to swap out its entire IT infrastructure based on misinterpretation of recovery recommendations and that its recovery efforts were “unnecessary”.

Furthermore, the report concluded:

“Based on EDA’s erroneous belief that it had a widespread malware infection, and its incorrect interpretation of recovery recommendations, EDA focused its recovery efforts on replacing its IT infrastructure and redesigning its business applications. EDA should have concentrated its resources on quickly and fully recovering its IT systems (e.g., critical business applications) to ensure its operational capabilities.”

None of this, unfortunately, comes as a surprise, given how mismanaged the agency’s security has been.

Problems date back as far as 2006 with tasks as common as patch management (some patches weren’t applied for years), and with malware dating back to 2009, not to mention a lack of monitoring for suspicious activity in 2006 and 2012.

It doesn’t require a nation state to take down an agency so inattentive to security, as the report points out—this house of cards could be toppled with the nudge of common, everyday, run-of-the-mill malware.

Hopefully, both the EDA and the DOC CIRT will take the auditor’s recommendations to heart.

Hopefully, they’ve learned, after this embarrassing fiasco, that chainsaws applied to hardware aren’t a cost-effective or necessary approach to remediating malware.

Found this article interesting? Follow Graham Cluley on Twitter, Mastodon, or Threads to read more of the exclusive content we post.

---