Skip to Content

How we built the same-day registration map

Earlier this week, MinnPost published an interactive map showing same-day registrations in precincts throughout Minnesota. Here, we walk through how we made it happen — both to document our work and, we hope, to help others who might be interested in building something similar.

Getting ready

Want to create maps like MinnPost does? Here's what you'll need to start:

What's the first step when working with data? Back it up. Save a copy of the original somewhere safe (we use a private GitHub account to keep much of our data and shape files organized, and we'll likely change to a more elaborate system in the future).

Cleaning the data

We like working with csv files. Convert the data into a csv file with in2csv:

in2csv 2008_general_results_calculated.csv

If you haven't used csvkit, Chris Groskopf has great slides to get you introduced. We stripped out some columns we know we wouldn't be using to make the file more manageable:

csvcut -c 2,11,12,13,15,16,17,21,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,49, 50,51,52,53 2008_general_results.csv > 2008_general_results_cut.csv

We recommend creating a file "audit" to document which command you used to change your data. It's good documentation, attribution, and it can be extremely helpful when something goes wrong.

Generally, data needs meddling to display correctly on a map. For our same-day registrations map, we needed to calculate percentages of registrations and election results for each precinct. We wrote a quick python script to do the number crunching for us.

This could have been done in Excel with formulas, but writing a script leaves exact documentation of how we came up with our new numbers. This way, when we realize that some percentages are over 100, we can double check our math and know that it was an error in the numbers we received from the state.

The script also creates a new field — a combination of the MCD and precinct codes. A shapefile is actually a bunch of files zipped together, and the .dbf contains the hard data. Our shapefile does not include a unique identifier shared with the election data, so we created the field "CTUPRE" with this script to modify the .dbf file of the shapefile.

Join the data with the shapefile

Finally we are ready to join the data with our shapefile. Fire up QGIS.

Add a vector layer, and locate the shapefile including our custom unique identifier. You should see a nice map of Minnesota.

To import the csv file, QGIS needs to know some information about what type of data each column in our csv holds. It looks for a file with the same name as the csv, but with file extension ".csvt". Be sure to create this file before importing the csv into QGIS. Then, add the csv file as another vector layer. Right click the csv later and look at the properties — the fields should have the correct data types. If not, double check that .csvt file.

Look at the properties of the shapefile layer, and go to the joins tab. We want to add a join with the csv layer (2008_general_results_calculated), and use the CTUPRE for both the join and target field.

To check that this worked, let's add a simple styling rule to the map. Go to the properties of the shapefile and click on the style tab. Created a graduated style over EDRPercent, change the mode to Quantile, hit Classify, and then hit OK. If everything worked, you should see a more colorful version of the state. Right click the shapefile layer, and save it as an ESRI Shapefile.

Style the map with TileMill

We now have a new shapefile with fun data embedded in it. Open up TileMill and create a new project — pick a smart filename, as we'll need this later to render the map. We chose "same_day_registration_interaction." Don't include the default data; we'll be creating our own styles. Click add to create the project.

In the lower left corner, add a layer. Click browse under data source, and locate the new shapefile you saved from QGIS. If you've ever done CSS, TileMill styling will make you feel right at home. Choose an ID for the file, and click Save & Style to get a good starting point. TileMill has virtually endless possibilities for styling maps — the sample projects included with TileMill are a good place to look for examples of what can be done with map styling. Play around and make your map look nice — and don't forget to hit save to refresh the map.

To display our data, we wanted to color our map based on the percentage of voters who registered on election day. This is easily done by using targeted styles like this:

#sameday_registration [EDRPercent >= 0] { polygon-fill: #d9d2e9; }
#sameday_registration [EDRPercent > 5] { polygon-fill: #b4a7d6; }
#sameday_registration [EDRPercent > 10] { polygon-fill: #8e7cc3; }
#sameday_registration [EDRPercent > 15] { polygon-fill: #674ea7; }
#sameday_registration [EDRPercent > 25] { polygon-fill: #351c75; }
#sameday_registration [EDRPercent > 40] { polygon-fill: #20124d; }

For our map, we decided to add some context layers. We used MapBox's tiles for states and roads (when adding a layer, change the source to MapBox and browse what they have), and we added county lines from our shapefile database.

Adding interaction

When you're happy with the map's style, it's time to work on the interaction. At the lower left corner of TileMill, click on the hand icon to bring up the interaction window. We decided to use a teaser, which is displayed on hover. Choose the layer you want to add interaction to (sameday_registration), and TileMill will show you the available data fields (a better way to view this data is by clicking the magnifying glass on the layers tab).

Whatever you put in this teaser field will show up as the tooltip, but anything in {{{ mustache tags }}} will be replaced by the actual data values. Here's what we used:

{{{MCDName}}}, Precinct {{{PRECINCT_N}}}
Same day registration
{{{EDRPercent}}}%
2008 U.S. presidential results
{{{PRPercent}}}% Republican
{{{PDPercent}}}% Democrat
2008 Minnesota legislative results
{{{MNRPercent}}}% Republican
{{{MNDPercent}}}% Democrat

Render and deploy the map tiles

Finally, we're ready to show this map to the rest of the world. In the upper right corner, export the project as MBTiles. Set a pin at the center of the map, and shift-drag a boundary box. Decide which zoom levels to render, but beware: each additional level adds many, many more tiles. We rendered our same-day registration map at levels 6-12, and it ended up being more than 25,000 tiles. This process will take a few minutes, so sit tight.

When the tiles have been rendered, we'll want to extract the files and organize them in a logical way. Our team uses Fabric to make render and deployment as painless as possible (highly recommended), but here are the basic steps:

1. Copy the .mbtiles file we just exported to our project directory.

2. Run mb-util to extract the tiles and grids from the .mbtiles file.

3. Change a few file locations to set everything up for deployment to Amazon S3.

4. Use invar to send the files up concurrently, drastically improving deployment speed.

You can skip steps 3 and 4 if you're only working locally.

With our fabfile, each time we want to deploy a change to the map, we type:

fab deploy_all

And Fabric takes care of the rest. Our fabfile should be fairly universal; just make a few changes near the top of the file to get it looking for the right directories.

Once you've extracted the tiles, you should see a directory tree with .png images of your map. These are the tiles, and they are strategically named and placed into folders that many javascript mapping libraries can handle. The .grid.json files contain a text-based image of the .png files with data embedded in them. Read more about it here; it's pretty amazing.

We use Leaflet for our client-side mapping software, and Wax to connect the interaction data with the map. Wax requires a special .json file that describes where to look for the tiles and grids, and what the tooltip should display. This javascript will use that json file to create a map, complete with interaction.

wax.tilejson('../same_day_registration_interaction.json',
    function(tilejson) {
        var map = new L.Map('map-div')
            .addLayer(new wax.leaf.connector(tilejson))
            .setView(new L.LatLng(46.3, -94.2), 7);
        var interaction = wax.leaf.interaction(map, tilejson);
    });

Wax will create a div with css class .wax-tooltip that contains the tooltip data when a user hovers over the map. The tooltip was originally hidden behind the map; we had to change its z-index and position to get it to display correctly.

Finally, the map should be ready to put into your CMS. We used the legend data from TileMill to create a key, which we styled and placed near the text.

Check out our finished map and let us know what you think.

Get MinnPost's top stories in your inbox

Related Tags:

About the Author:

Comments (1)

wow...

I'm a science dork--give me protein or DNA sequences and I'm happy to run them through various programs to identify key features and characteristics. I'm also interested in exercising my artistic side on paint programs with layers and textures. But this...is intimidating. Nice job--I'll leave these things up to you.