On the third floor of an office building in St. Paul, a conference room has been turned into a makeshift command center, with cubicles and computer monitors looking toward a giant pull-down screen.
On a recent Tuesday, the room is dark except for the glow of computer monitors lighting a half dozen faces, all glued to their screens. One of the employees, Dave, who always wears a bowtie on Tuesdays, is sitting by his blue lava lamp and a unopened Obi Wan Kenobi figurine and trying to find any traces of “Petya.”
Petya is a new version of ransomware that happened to be ravaging systems in the Ukraine and Russia. It’s called ransomware because it infects computer systems and shuts them down until a ransom is paid. Next to two giant computer monitors, Dave has a small laptop not connected to the system that he’s using to browse hacker forums.
There was some dispute as to whether the ransomware was actually Petya, but “it’s walking like that duck and it’s quacking like that duck,” Dave said. “Some financial institutions in the Ukraine and Russia are down. An oil company in Russia is down.”
This makeshift office is Minnesota’s Security Operations Center, where nine people working in staggered shifts form the front line of the state government’s cybersecurity defense, protecting the data of more than five million citizens from hackers around the world. On a given day, the state’s systems are scanned several million times for potential vulnerabilities. These nine staffers are part of a total team of 61 people who work in cybersecurity for the entire state. To put that number into perspective, a corporation like U.S. Bancorp has more than 500 people working on the issue.
To many in the state, cybersecurity may seem like an abstract problem — data sitting “in the cloud” or on servers unseen. Yet a breach of that security could have far-reaching and devastating real-life consequences. State governments hold on to all sorts of personal data, from Social Security numbers, tax records and license information to marriage, birth and death records.
And yet, as states make more government information and services available online, thwarting cyberattacks becomes ever more difficult. This month, a hacker upset that the police officer charged with killing Philando Castile was found not guilty found a weakness in the state’s databases, stealing emails and passwords.
“If there are people who hack into those systems, critical government systems will not work that society depends on,” said Christopher Buse, the Chief Information Security Officer for the state. “If we lose health care data on systems, you can’t just pay somebody money to get the genie back in the bottle. Their health records are out there, or the name of undercover police officers, you can’t make them safe again once that’s out in the wild.”
A bigger issue than staffing, say IT officials, is the state’s decades-old computer systems, which are scattered at more than two dozen locations across Minnesota and are not able to be secured by any modern technology. And while fixing those problems was part of a heated debate at the Capitol during the 2017 legislative session, the issue ultimately wasn’t resolved, even if everyone agrees it’s not going away. “It’s not if we’re going to be attacked,” said Sen. Paul Anderson, R-Plymouth. “It’s how you respond when we do.”
‘Flooded’ with old technology
Buse used to be the one trying to poke holes in Minnesota’s cybersecurity defenses. He spent 19 years in the Office of the Legislative Auditor, most of that time doing audits of state government IT systems. “I made my living throwing darts at IT professionals,” Buse said. “Now I’m on the other side. I’m the dart board.”
Back then, much of state business was still done on paper, with the computerized data it did have stored on large mainframe systems. One of the last audits Buse worked on in 2001 found major weaknesses with the Department of Public Safety’s online vehicle registration renewal system, which left citizen data vulnerable to disclosure and fraud. The audit’s recommendation: shut it down.
The move was controversial. “Citizens liked the fact that they could sit in their living room and do their vehicle tabs,” Buse said. “That was … when our Legislature really started to look at IT.”
In 2011, lawmakers created Minnesota IT, or MNIT, an entire state agency run by the state’s Chief Information Officer. The idea behind it was to consolidate the IT being done by more than 78 state agencies, boards and commissions into one agency, which would make it more efficient — and more secure.
But those efforts are expensive. This session, the department and Gov. Mark Dayton requested $125 million to make the state government’s systems more secure. Part of the money was to go toward updating some of the state’s decades old computer systems, which are so old they have their own special coding, and are incompatible with any modern operating systems or security functions. Another $74 million would have gone into various IT security improvements, including adding more cybersecurity staff, as well as new software that blocks hackers.
“Government is literally flooded with old technology,” Buse said. “When you look at [$125] million, it seems staggering, but when you look at the number of agencies and the age of these systems and what systems cost these days, the numbers add up pretty quickly.”
Lawmakers had a $1.65 billion surplus to spend during the last legislative session. But the funding was competing with tax cuts, transportation funding and plenty of other priorities. Rep. Sarah Anderson, the Republican chair of the House State Government Policy and Finance Committee, said she wasn’t inclined to put funding into cybersecurity until every state agency was on board with consolidating their IT functions and becoming more secure.
“One of the key pieces you need is consolidation, because that limits the entry points for securing our data in government. When these agencies refuse to participate in the consolidation, that’s a problem,” she said.
Then, during the Legislature’s special session, cybersecurity funding got caught up in bargaining between the governor and Republican legislative leaders. Republicans said they offered about $22 million in cybersecurity funding to the governor. But instead of using it for that purpose, he used the money to maintain current staffing levels for state agencies. “They used it to fund agencies across the board,” said Anderson.
Democrats say Republicans forced the administration’s to choose between cuts to state agencies or funding for cybersecurity.
Data centers in ‘closets’
In Minnesota, “the cloud” is in an actual place: a secret location filled with stacks of humming servers.
This so-called “data center” is where state data is physically held. To get to this facility, someone must first give you the address, which isn’t publicly listed. The entrance is not visible from any street, and the building is mostly underground to protect it from natural disasters. Once there, entrance into the building requires several clearance points and fingerprint identification. Inside, backup generators keep the facility running at all times — even if power to the city where the facility is located were completely cutoff.
The servers hold petabytes of data, things like unemployment records, tax information and private health data from MNsure, the state’s health insurance exchange. Monitors look at what data is going in and out of the servers, and a complicated cooling system keeps them from overheating. Particle sensors and can tell if anything changes in the physical space.
For MNIT, this is the future. Since forming in 2011, the agency has consolidated 49 different data centers in the state into 27 centers. But not all data centers are created equally. Older centers were sometimes the size of a closet, without any of the highly-controlled physical and technological features in the secure facility.
“Think of it as anywhere from the closet in your apartment or home to this,” Thomas Schaeffer, the Chief Operating Officer for MNIT, said from the secure facility. “They vary that wildly. The ones we closed early were the closets. Those were low hanging fruit, the 20 or so left are the more complex ones.”
Ultimately, the department wants to have only six data centers, with three of them being highly-secure facilities like this one. That would cost about $14 million, which was included in the governor’s request last session.
“The cloud is just like your computer at home, at scale. This is just a large scale computer processing unit that has all of those controls built into it,” Schaeffer said. “To secure all the state’s data the way it is currently configured is very difficult.”
Threats are everywhere
Paul Anderson is serving his first term in the state Senate, but he used to work on technology and IT issues in former Republican Gov. Tim Pawlenty’s office, and he spent a few years working with a tech startup in the private sector.
He authored a bill, along with Rep. Jim Nash, R-Waconia, that would have created a Legislative Commission on Cybersecurity. The bill passed unanimously in the state Senate, but it never made it to a vote in the House.
The idea was to create a space where legislators can actually dig into the issues, instead of discussing it sporadically or when security breaches occur. At least 13 states have recently created some kind of statewide cybersecurity task force, commission or advisory council.
The state has yet to face a major breach of its data, but the threats are everywhere, Anderson said. “It drives me crazy that we can’t get more of a focus on this issue,” he said, adding that he plans to bring the issue up next session, which convenes in February 2018.
“The amount of work that MNIT has to do with the limited resources they have is not adequate for the enterprise they have to run,” he said. “I believe we have to put more attention to this. It’s making sure we have the resources to manage the attacks that are inevitably going to happen.”
