Crime doesn’t happen in a vacuum, but spot news coverage rarely tells the whole story. To even begin to understand crime trends and changes in our cities, we have to look at the data.
At MinnPost, we hope to do crime reporting in a different way — a way that encourages readers to dig into the data and ask questions, enables our reporters to track down stories, and encourages public officials to make this data more transparent. That’s why we are launching www.minnpost.com/crime, an interactive application that displays monthly Minneapolis crime data.
If you take a look at the application, you may notice that autotheft in Minneapolis has been declining steadily since 2005 (it’s down about 50 percent). And in nearly every category, 2005-06 was the crime peak citywide in the last 10 years. You can also see that crime in Downtown West is up 17 percent this month, but down compared to the same month last year. The high incident rate — 35.66 compared to Minneapolis’s 6.51 — can be explained, in part, by downtown’s low number of residents and high number of visitors (people who work downtown or visit for entertainment purposes).
These are just a few of trends you can find by exploring the monthly aggregate data. We’ve also made it easy to locate your current neighborhood while you’re out and about (the app is built responsively, so it will work on your mobile device). Most importantly, this is just the beginning of how we want our crime data reporting to develop. Read more about our methodology and plans below; we welcome your questions and ideas at data@minnpost.com.
Where’s St. Paul (or city of your choice)?
Both the Minneapolis Police Department and the St. Paul Police Department release some level of crime data in public places online. The SPPD releases incident level data on a regular interval, but the data is not geocoded, and St. Paul district councils are fairly large geographical areas. On the flip side, the MPD releases aggregate numbers of incidents grouped by category and by Minneapolis neighborhood.
We decided to start with Minneapolis because it was easier to get the data into a structured format where we could do analysis and use it an application, and because the neighborhood level data lent itself to simpler geographical representation.
Our goal is to provide data for St. Paul, as well as other cities in Minnesota, in future iterations of the app. This will be an ongoing project, as many municipalities do not consistently report this data — at least not in ways that are transparent and accessible to the public.
All about the data
Parsing crime stats is a complex business, from the sheer size of the data to the number of variables that can affect changing rates over time. We’re presenting the data in as clear and unsensationalized way as possible, but that doesn’t replace human analysis. By being transparent about the limitations of this data, we hope to encourage reporters and citizens alike to ask questions and work with public officials to get better crime information.
In the application we use the standard crime rate measure, which is the number of crimes per 1,000 residents. This measure helps normalize crime in areas with varying population. To determine the crime rate, we used 2000 and 2010 census populations to create a linear projection for current neighborhood populations. This is a standard way to measure crime across multiple locations, but it is not without its flaws.
One of the biggest issues with the crime rate measure is that there are neighborhoods in Minneapolis that have a very low population (according to the latest census numbers). This means that places like the Humboldt Industrial Area in northern Minneapolis, population zero, can have an extremely high crime rate even though there were very few crimes in that area.
Another flaw in the standard crime rate is that it doesn’t take commercial activity into account — neighborhoods where a significant number of people commute during the day or night. Downtown West, as mentioned above, is an example of an area that draws many non-resident visitors and shows a higher crime rate because of it.
The data powering the initial version of the crime application are from structured reports released online by the Minneapolis Police Department. These are monthly reports that use the FBI Structured Crime Report (SCR) categories for each neighborhood in Minneapolis.
Note that the MPD reports are counted a bit differently than the FBI reports. The SCR reports count the highest offense in a crime event, while the MPD reports count each offense in a single crime event. This means that if a robbery ended up with a homicide, the SCR report would count only the homicide, while the MPD report counts both a robbery and homicide. Additionally, the MPD reports the date of the crime while the SCR uses the date of the report.
The MPD reports are not perfect: There are some missing months, the reports only go back about 10 years, the names of neighborhoods are not consistent, and the formatting of each report varies. We have done our best to ensure that these issues are taken into account and believe that the application is as accurate as this data allows.
From the start of our initial development of the crime application, we have been in contact with both the SPPD and MPD about getting more granular and structured crime data for each city. Unfortunately, due to both high costs and unresponsiveness from the agencies, we have yet to receive complete datasets for either city.
Having granular, complete, and historical data would allow us to do more in-depth analysis.
Granular data would allow us to look at smaller areas (e.g., along the Midtown Greenway); neighborhoods are a somewhat arbitrary boundary when it comes to crime. Often high crime is isolated to a couple of blocks, but aggregated data does not allow us to explore this.
More complete data would allow us to look at factors like exact locations, time of day, demographic issues, and detailed categories (like bike theft).
Having more historical data would help us see long term trends. Crime may change significantly from month to month, but it often takes years for crime trends to change overall. Even with 10 years of data, we do not have a complete picture of how crime is changing in our city.
How we built it (the nerd stuff)
As with all our interactive pieces, you can see the most of our code and techniques on GitHub.
The first part of this project involved turning dozens of Excel reports into a single data source, which was not trivial given how much the formats of the reports vary. We decided to use ScraperWiki to create this comprehensive data source. ScraperWiki is an excellent tool for turning unstructured data on websites into a structured database with an API. While we were building the app, though, the service started to experience some downtime and we realized we could not fully depend on it.
Fortunately, the data in the app is not updated too frequently (once a month). To work around the ScraperWiki issues, we built a small proxy service that sits in front of the API and caches the results and will only invalidate the cache if the original request returns a good HTTP status code. All Good Proxy is easy to deploy on Heroku if you need a similar solution.
While we got our dataset and API ready to go, the interactive team also worked on sketching out the user interface and design. We approach our apps from a mobile first perspective, and wanted to give users a clear and simple navigation experience. Crime data can be overwhelming, so we identified the top questions we wanted to answer:
- How many incidents have happened
- What are the crime rates in my neighborhood and how do they compare
- How is crime changing over time (historical context)
From there we sketched, reviewed design direction, mocked up some prototypes, and continued to iterate as we received feedback and tested the app.
Alan built the app using some of our usual tools: jQuery, Moment, Underscore, Backbone, Leaflet, and Highcharts. We tried out a few new things with this application, including Backbone.stickit, developed by the New York Times, which allows Backbone to create a two-way binding with display and data elements. This drives features like animating the number changes and switching out values. We used Chroma to color the map, which allowed us to scale arbitrary color values for the map (necessary because the crime rate breakdown will change each month).
As with all of our interactive projects, the crime app is responsive and can be viewed on different screen sizes (e.g., your mobile phone). For this project, we tried out a CSS framework called Unsemantic to help with a fluid grid layout. The features are the same on mobile as they are in a desktop browser, but for mobile sizes we highlight the option to locate crime stats for your current location, which makes it easy for people to get data about whatever neighborhood they may be visiting.
We welcome your feedback and suggestions. Check out the app and let us know what you think.
