As calls for accountability from the Minneapolis Police Department continue to mount since the death of George Floyd, city officials are turning to data in hopes of identifying problematic police officers before they use excessive force. 

In his pitch to reform the MPD, Police Chief Medaria Arradondo announced at a press conference two weeks ago that the department plans to partner with Chicago-based Benchmark Analytics to develop an early intervention system (EIS) to monitor officer behavior.

The idea is to track early warning signs of misbehavior; as officers change assignments throughout their careers, there’s not a systematic way to track their record. “I know that there have been questions raised about when an incident occurs, ‘Well, Chief Rondo, how come you didn’t know about this person’s number of complaints and what have you?’” Arradondo said. “We as an organization need to evolve and use technology to our advantage. And [that] doesn’t have to wait for every 30 days for an employee review or every year.” 

The concept of EIS in the field of law enforcement is not a new one. Police departments across the country have for years looked to these systems to oversee the daily operations of their officers. Minneapolis developed an EIS in 2009, but a report by the Department of Justice in 2015 found, among other things, that it was not being used uniformly, that it was perceived as a wellness program as opposed to an accountability tool, and that a “lack of automation prevented electronically tracking and flagging of behaviors of concern in a systematic manner.” The report made recommendations for MPD to overhaul its system; however, the Star Tribune reported that those plans never got off the ground. 

Now, following the police killing of Floyd, discussions about implementing a more effective early warning system as part of the solution to weeding out problematic cops is taking on new life. While the details of MPD’s agreement with Benchmark have yet to be released, here’s what we know about the company and the research behind early intervention systems.

The ‘next chapter’ in police reform

Founded in 2017 by two former Chicago police leaders, Benchmark Analytics’ EIS system, called First Sign, was based on research from the University of Chicago’s Center for Data Science and Public Policy (CDSPP). 

CDSPP’s work emerged from the Obama administration’s Task Force on 21st Century Policing in 2015, and the researchers’ aim was to develop an EIS that would rectify the old model, which they say was reactive, inaccurate and didn’t prioritize at-risk officers for intervention. 

Former CDSPP director Rayid Ghani, who is now a professor at Carnegie Mellon University, said he found through working with the Charlotte-Mecklenburg Police Department in North Carolina that the systems often gave alerts after the officer had engaged in problematic behavior several times.

For example, if an officer had three or more use-of-force incidents within the last three months, the system would raise a flag. The criteria for flagging officers was also fairly random – the EIS would flag over half of the officers for potential problematic conduct but then also miss about half the officers who actually engaged in misconduct.

“The problem with that reactive approach is that you don’t prevent things and you can’t fix problems. You can only react punitively,” Ghani said. “You can’t really do anything to prevent them from doing the first few things that they were going to do.”

After developing the system with the Charlotte-Mecklenburg Police, the center later licensed its system to Benchmark Analytics, which markets itself as the “only evidence-based” early warning system. 

The company’s CEO and co-founder, Ron Huberman, a former assistant deputy superintendent in the Chicago Police Department, said their system moves past this “threshold” model that punishes officers after a certain number of use-of-force incidents. Instead, they analyze existing police department data, using as many as 25 variables, such as use of force or arrest information, to identify patterns in officer behavior. 

What makes the First Sign EIS different is that it will be able to target the few problematic police officers on the force with more accuracy than before, according to Huberman. 

“We think of ourselves as a fair broker, in the sense that we’re just using data and science to identify this problematic conduct,” Huberman said. “Police reform has been around 200 years;  this is not a new thing. Evidence-based data driven tools [are] a super new thing to the world of police reform. I’m going to argue to you, it’s the next chapter of police reform. And we’re on the front end of that next chapter.”

Former Minneapolis Mayor R.T. Rybak said while he was in office he saw Benchmark Analytics as a “breakthrough” that could equip police departments with tools they didn’t have before to oversee police. He later joined Benchmark Analytics as a founding board member, though he is no longer on the board. 

“Too often I would be in a settlement conference after a horrible incident and ask when we first knew something was going wrong with this [officer] and find out that we had some information in two or three or four different databases that together told us something more important. The research showed that we didn’t have to speculate what makes the cop go wrong and what helps them get back on track,” said Rybak, who is also the CEO of the Minneapolis Foundation.

The foundation was initially set to fund the project through its first year, but Rybak has since said they have dropped their involvement after receiving pushback from community activists, according to the Associated Press. Mayor Jacob Frey said that the city will search for alternative sources of funding or use existing city money to pay for the system instead. While he declined to specify how much the Benchmark Analytics system would cost, Frey said at a press conference Friday that these systems typically cost $100,000 to $300,000.

Still, it’s not entirely clear how effective the system will be at identifying problem police officers. To Ghani, using 25 variables to analyze police behavior seems like a small number.

“The more proactive, preventative you want to be the more subtle signs that you’re looking for. And so, for different types of people, for different departments in different beats in different jurisdictions, you’re gonna find that those things change,” Ghani said. For example, there are several variables with just use of force data: whether any weapons were used, if there were any injuries, whether it was investigated and what the results of the investigation were. 

“If we knew exactly what those 25 variables were, you know, life would be really easy, because we just sort of measured those and then you stopped people,” Ghani said. 

A spokesperson for Benchmark Analytics did not provide a requested list of the variables used by the company’s EIS in time for publication.

More questions

It’s up to the police department to decide what action they want to take once Benchmark notifies them. But what police departments should ideally do next after an officer has been flagged is up for debate.

According to Ghani, researchers are still studying whether police departments are capable of choosing the right interventions to prevent officer misconduct.

“What we want to evaluate is not just our ability to predict, but much more importantly our ability to prevent – because we know we can predict, to some extent, which officers are high-risk. Now the question is, ‘Does the department have the right programs, interventions in place so that if given that list of high-risk officers they can prevent [misconduct] from happening?’” Ghani said. 

Which interventions are most effective is also an open question.

“Some of the interventions are not you press a button and problem solved. And so, how much time do you need in order for the intervention to work is a really important piece, like how often do you do training? How often do you do counseling? What does the cool-down period look like, is it a day, a week, a month? … At what point do you really have to get rid of [the officer]?” Ghani asked. 

Huberman said Benchmark, in partnership with the University of Chicago and the 70 police departments that use First Sign, is currently studying which interventions work best. Supervisors have a range of options, such as training, coaching, counseling or simply re-assigning the officer to work with a “higher-performing” colleague. 

“Sometimes it’s just thinking about who works with whom, which is a very simple intervention. It’s how you structure a relationship between a front-line supervisor and an officer. Meaning how they can serve as a mentor, and debrief with officers after use-of-force events, do supervisor ride alongs on 911 calls to help provide that officer with coaching and feedback about how they’re interacting,” Huberman said. 

The DOJ report in 2015 found that MPD primarily used coaching and mediation, and recommended that the department broaden its range of interventions. The DOJ additionally reported that the Minneapolis’ system was too reactive, included little input from personnel to define “problematic behavior” and lacked the ongoing training needed to change those behaviors. 

An early intervention system won’t fix racism

As Arradondo continues to introduce his plans to reform the department, activists are increasingly demanding more drastic measures, including abolishing and defunding MPD altogether. This sentiment contrasts with the philosophy underpinning an EIS – that police departments just need a better, more advanced way to monitor the few problematic cops. 

Nekima Levy Armstrong, a Minneapolis civil rights attorney and long-time activist, was there when the DOJ sought community input during its evaluation of MPD about five years ago. While she thinks an EIS could be a great way to monitor rookie cops, what the department really needs is a culture change, Levy-Armstrong said. 

“They have so many officers on the force who have engaged in excessive force or engaged in domestic abuse at home. Many who have killed people on the job and there’s been no real discipline or accountability. Many for whom the city has paid out large sums of money in settling excessive force cases. So they really now need to clean house, based on who is currently on the force.”

How MPD’s internal affairs department has dealt with complaints in the past could determine how well it flags officers under the new system. 

“If you have a corrupt IA process and it never finds anything to be unjustified and it never sustains a complaint, then the system is going to basically say, I’m not going to flag anyone because nobody’s ever done anything bad before,” Ghani said. 

‘Not a cure’

The city’s Office of Police Conduct Review received 2,013 complaints against police officers within its jurisdiction between 2013 and 2019. About 1.5% of those complaints resulted in suspensions, demotions or terminations, according to a CNN analysis.

After the EIS is implemented, the department needs to be transparent and tell the public how these systems are used – how many officers were flagged, why were they flagged, how many incidents were prevented and how many were missed, and so on, Ghani said. 

“I think an EIS is a small piece of what every police department needs to have, but it’s not a cure for bad policing and for racism.”