Skip to Content

Support MinnPost

Introducing MinnPost's Minneapolis crime app

Click to view the crime app
Click to view the crime app

Crime doesn’t happen in a vacuum, but spot news coverage rarely tells the whole story. To even begin to understand crime trends and changes in our cities, we have to look at the data.

At MinnPost, we hope to do crime reporting in a different way — a way that encourages readers to dig into the data and ask questions, enables our reporters to track down stories, and encourages public officials to make this data more transparent. That’s why we are launching www.minnpost.com/crime, an interactive application that displays monthly Minneapolis crime data.

If you take a look at the application, you may notice that autotheft in Minneapolis has been declining steadily since 2005 (it's down about 50 percent). And in nearly every category, 2005-06 was the crime peak citywide in the last 10 years. You can also see that crime in Downtown West is up 17 percent this month, but down compared to the same month last year. The high incident rate — 35.66 compared to Minneapolis’s 6.51 — can be explained, in part, by downtown’s low number of residents and high number of visitors (people who work downtown or visit for entertainment purposes).

These are just a few of trends you can find by exploring the monthly aggregate data. We’ve also made it easy to locate your current neighborhood while you’re out and about (the app is built responsively, so it will work on your mobile device). Most importantly, this is just the beginning of how we want our crime data reporting to develop. Read more about our methodology and plans below; we welcome your questions and ideas at data@minnpost.com.

Where’s St. Paul (or city of your choice)?

Both the Minneapolis Police Department and the St. Paul Police Department release some level of crime data in public places online. The SPPD releases incident level data on a regular interval, but the data is not geocoded, and St. Paul district councils are fairly large geographical areas. On the flip side, the MPD releases aggregate numbers of incidents grouped by category and by Minneapolis neighborhood.

We decided to start with Minneapolis because it was easier to get the data into a structured format where we could do analysis and use it an application, and because the neighborhood level data lent itself to simpler geographical representation.

Our goal is to provide data for St. Paul, as well as other cities in Minnesota, in future iterations of the app. This will be an ongoing project, as many municipalities do not consistently report this data — at least not in ways that are transparent and accessible to the public.

All about the data

Parsing crime stats is a complex business, from the sheer size of the data to the number of variables that can affect changing rates over time. We’re presenting the data in as clear and unsensationalized way as possible, but that doesn’t replace human analysis. By being transparent about the limitations of this data, we hope to encourage reporters and citizens alike to ask questions and work with public officials to get better crime information.

In the application we use the standard crime rate measure, which is the number of crimes per 1,000 residents. This measure helps normalize crime in areas with varying population. To determine the crime rate, we used 2000 and 2010 census populations to create a linear projection for current neighborhood populations. This is a standard way to measure crime across multiple locations, but it is not without its flaws.

One of the biggest issues with the crime rate measure is that there are neighborhoods in Minneapolis that have a very low population (according to the latest census numbers). This means that places like the Humboldt Industrial Area in northern Minneapolis, population zero, can have an extremely high crime rate even though there were very few crimes in that area.

Another flaw in the standard crime rate is that it doesn’t take commercial activity into account — neighborhoods where a significant number of people commute during the day or night. Downtown West, as mentioned above, is an example of an area that draws many non-resident visitors and shows a higher crime rate because of it.

The data powering the initial version of the crime application are from structured reports released online by the Minneapolis Police Department. These are monthly reports that use the FBI Structured Crime Report (SCR) categories for each neighborhood in Minneapolis.

Note that the MPD reports are counted a bit differently than the FBI reports. The SCR reports count the highest offense in a crime event, while the MPD reports count each offense in a single crime event. This means that if a robbery ended up with a homicide, the SCR report would count only the homicide, while the MPD report counts both a robbery and homicide. Additionally, the MPD reports the date of the crime while the SCR uses the date of the report.

The MPD reports are not perfect: There are some missing months, the reports only go back about 10 years, the names of neighborhoods are not consistent, and the formatting of each report varies. We have done our best to ensure that these issues are taken into account and believe that the application is as accurate as this data allows.

From the start of our initial development of the crime application, we have been in contact with both the SPPD and MPD about getting more granular and structured crime data for each city. Unfortunately, due to both high costs and unresponsiveness from the agencies, we have yet to receive complete datasets for either city.

Having granular, complete, and historical data would allow us to do more in-depth analysis.

  • Granular data would allow us to look at smaller areas (e.g., along the Midtown Greenway); neighborhoods are a somewhat arbitrary boundary when it comes to crime. Often high crime is isolated to a couple of blocks, but aggregated data does not allow us to explore this.  

  • More complete data would allow us to look at factors like exact locations, time of day, demographic issues, and detailed categories (like bike theft).

  • Having more historical data would help us see long term trends. Crime may change significantly from month to month, but it often takes years for crime trends to change overall. Even with 10 years of data, we do not have a complete picture of how crime is changing in our city.

How we built it (the nerd stuff)

As with all our interactive pieces, you can see the most of our code and techniques on GitHub.

The first part of this project involved turning dozens of Excel reports into a single data source, which was not trivial given how much the formats of the reports vary. We decided to use ScraperWiki to create this comprehensive data source. ScraperWiki is an excellent tool for turning unstructured data on websites into a structured database with an API. While we were building the app, though, the service started to experience some downtime and we realized we could not fully depend on it.

Fortunately, the data in the app is not updated too frequently (once a month). To work around the ScraperWiki issues, we built a small proxy service that sits in front of the API and caches the results and will only invalidate the cache if the original request returns a good HTTP status code. All Good Proxy is easy to deploy on Heroku if you need a similar solution.

While we got our dataset and API ready to go, the interactive team also worked on sketching out the user interface and design. We approach our apps from a mobile first perspective, and wanted to give users a clear and simple navigation experience. Crime data can be overwhelming, so we identified the top questions we wanted to answer:

  1. How many incidents have happened 
  2. What are the crime rates in my neighborhood and how do they compare
  3. How is crime changing over time (historical context)

From there we sketched, reviewed design direction, mocked up some prototypes, and continued to iterate as we received feedback and tested the app.

Alan built the app using some of our usual tools: jQuery, Moment, Underscore, Backbone, Leaflet, and Highcharts. We tried out a few new things with this application, including Backbone.stickit, developed by the New York Times, which allows Backbone to create a two-way binding with display and data elements. This drives features like animating the number changes and switching out values. We used Chroma to color the map, which allowed us to scale arbitrary color values for the map (necessary because the crime rate breakdown will change each month).

As with all of our interactive projects, the crime app is responsive and can be viewed on different screen sizes (e.g., your mobile phone). For this project, we tried out a CSS framework called Unsemantic to help with a fluid grid layout. The features are the same on mobile as they are in a desktop browser, but for mobile sizes we highlight the option to locate crime stats for your current location, which makes it easy for people to get data about whatever neighborhood they may be visiting.

We welcome your feedback and suggestions. Check out the app and let us know what you think.

Get MinnPost's top stories in your inbox

Related Tags:

Comments (6)

St Paul crime data

St. Paul also releases monthly crime data for police grids that are relatively small compared to district councils. I have successfully requested the police grid shapefiles and crime data (in Excel) in the past.

Crime

Auto theft has probably been declining because cars now have more antitheft devices (chipped keys).

UCR and CODEFOR data are useful, but the categories are funky. For example Theft/Larceny is considered a Part I crime, but shoplifting probably isn't "serious." State statute changes over the last few years have effected the categorization also. Domestic can now be a felony, as well as DWI. Drugs are listed as a Part II crime (less serious) but 20ish percent of people in prison are there for drug related offenses. Clearance rates (the amount arrested for each reported crime) might be 40%. Police also generate the majority of the reports of crime in some categories, such as drugs, DWI or prostitution. Enforcement can drive the rate of reported crime.

The major flaw I see is that this analysis only looks at new crime. It does not look at the churn of probation violations, warrants for not appearing in court, holds for other jurisdictions, traffic or other reasons for police stops. But nobody really tracks that well. If you look at the MPD weekly crime reporting online, note that total arrests are double the reported arrests for UCR (FBI Uniform Crime Reporting) crimes.

Also, this does not track tickets. They are processed through a separate court system in Hennepin and Ramsey counties and amount to several hundred thousand a year.

At the end of the day, if a community feels safe is the true measure of progress. I wish we could do a regular public safety survey but those were done away with in Minnesota over a decade or two ago. But cudos to MPD for being as transparent as they are for as long as they have.

Additional data point?

Have you concidered including weather data in your analysis?

More crime data caveats

You are doing a good job of including and explaining the caveats involved in using "known and reported" crimes to describe both the underlying crime rate and the law enforcement measurement system. This is especially problematic making comparisons over time and across jurisdictions where law enforcement reporting policies can differ.

One of the biggest problems of equating reported crime with the underlying crime rates is the majority of property crimes may go unreported and therefore uncounted. The media coverage of violent crime may also encourage over-estimates of the likelihood of being a victim. This in turn may increase fear unnecessarily.

Finally, this dynamic works through the political process and campaigns and can lead to criminal justice policies that forclose better options for scarce resources. I believe the key question from the surveys below was about fear of walking near where you live; which is now a key question with growing health, health policy and future budget implications.

See:
Troubling perceptions: 1993 Minnesota crime survey: Statewide study of crime victimization, January 6, 1994. (24p. 874K, PDF) | Report details

Changing perceptions: 1996 Minnesota crime survey: Examines the public perception of the crime rate, December 1, 1996. (25 p., 675K, PDF) | Report details

Keeping watch: 1999 Minnesota crime survey: Third annual crime victim survey, March 1, 2000. (20 p., 135K, PDF 3.0) | Report details

Paying the price: the rising cost of prison : Examines policy options to a rising prison population, March 1, 1996. (27 p., 1046K, PDF 2.0) | Report details

http://www.mnplan.state.mn.us/cj/

mn planning

It's a little funny to me that we quote mn planning crime survey stats. This was a govt unit organization that was abolished during the first Pawlenty administration. And the website has not been taken down. And we expect the state to somehow solve the crime and cost problem?

MN Planning Criminal Justice Statistical Analysis Center reports

Craig,
I cited the MN Planning reports because I was involved in creating them, not because they are the most up-to-date trends in describing the frequency, severity and distribution of crime. My synthesis was based on knowing the strengths and weaknesses of different criminal justice related data sets, how they relate to each other, and how the themes of crime, public safety and justice are weighed to reducing the scope of the problem.

The context in describing crime and the impact on individuals, families, neighborhoods, communities, states and our nation should be as clear as possible. Ultimately crime must also be compared with other social issues (i.e. jobs, education, health, transportation) in the "Big Picture" of how we address problems and create a sustainable environment that improves Minnesota.

I'm hoping that the underlying values, effectiveness of methods, and common consensus on direction might be displayed through the political and resulting governmental process.