… why I am never going to be a programmer. Last week I saw several reporters tweeting about checking the Idaho Secretary of State’s Election’s page throughout the day so as not to miss any new candidate filings during the current Idaho elections filing period.
During candidate filing season, I check Ben Ysursa's website more often than I check Facebook. I'm not sure that's a good thing.
— Kevin Richert (@KevinRichert) February 28, 2012
I feel for Kevin and the rest of the Idaho Capitol Press Corps. While the Secretary of State does make a good effort to provide public information, and Secretary of State Ben Ysursa has been a champion of public records, there has got to be a better way.
I’ve been reading a lot about data journalism and thought I could follow some of the tutorials out there and create a nice little web scraper to tease out the scoops on candidate registration. I am really inspired by sites like ProPublica, which puts together awesome databases like this one on what influenced support on SOPA/PIPA or one on unequal course offerings in schools across the nation that incorporates maps. I like maps. The Guardian in the UK is also doing amazing things with data journalism and making information public and available and accessible. These guys make it look easy.
As with anything, it’s harder than it looks, and Brett ended up doing all of the heavy lifting on this.
Here is what (I think) we did:
- Wrote a ruby script (a little computer program) that automatically downloads the .pdf from the Secretary of State’s website as often as we like, converts it to text using Xpf, which preserves the spacing, and then uses Regular Expressions to parse the text into fields: District, Office, Party, Name, Address. The program runs remotely, on a server and activates itself. Yeah, it’s a bot.
- Use our nicely formated text file to build a Google Fusion table. Again, this is automated, using Google Maps/Fusion API commands. Google Fusion Tables allows us to easily place candidates on a Google Map, though it’s a bit more difficult to actually build the map on a web page, and requires a working knowledge of Javascript, another programming language.
- I told Brett I’d take it from there, but again, it was not that simple. First of all, I could not figure out how to embed all the javascript I’d need in a blog post on this blog, so I had to just make a new web page, that looks like it’s on this blog. Also, I really don’t get the details of javascript anyway. So once again, Brett bailed me out.
- I found a cool use of fusion tables that I wanted to emulate, allowing the user to select Senate, House A or House B races to display on the map so that they could easily see which districts had contested races. I spent hours trying to customize the Chicago homicide map linked above to my own needs, but gave up in desperation. Brett figured out the right calls we’d need to redraw the map (it’s not really that complicated, but it’s like learning a foreign language).
- Finally, after I’d say 25 hours between us, though my hours were much less valuable than his, we had a workable map. I’m still bummed it’s not as pretty and full of functionality as the Propublica projects are, but I think it’s quite useful.
So what can you do with this web app? First of all, I think it’s the first Google map of the newly drawn legislative districts. I converted the .shp file (L93) that the Commission published to a .kml file (which Google Maps reads) using Qgis on my Mac. So now people cal fly around the new districts, find their house, make sure that candidates actually live in their districts, figure out which legislators have the best nearby hunting spots, etc. UPDATE: The locations plotted for candidates are their filing addresses, not necessarily their home addresses, and thus may be located in a different district.
Second of all, you can easily see which races are contested for the May 15 Primary Elections by selecting one of the three Legislative races and looking for red or blue markers on the map. This map is constantly updated with new filing information. Candidate filing closes in a week, on March 9, so we will have to figure out how to make this information useful on an ongoing basis. I’d like to add web links and Twitter feed info for candidates and links to news articles about the races (if anyone wants to help, please speak up)!
Please let us know what you think and feel free to use any of the info you glean from our web app!