Introduction
City Scrapers mission
Public meetings are important spaces for democracy where any resident can participate in civic life and hold public officials accountable. But how does the public know when meetings are happening? It isn’t easy! These events are spread across dozens of websites, rarely in useful data formats. That’s why City Bureau worked with a team of civic coders to develop and coordinate the open-source City Scrapers project, which standardizes, scrapes and shares these meetings in a central database, in collaboration with City Bureau’s Documenters program.
The mission of the City Scrapers project is to increase access and transparency around public meetings across the U.S. by making it easier for everyone to know when and where public meetings are held. We’re sharing our code and process because we believe this model can work in other cities.
City Bureau, City Scrapers and the Documenters Program
The City Scrapers project grew out of the work of City Bureau, a nonprofit civic journalism lab based on the South and West sides of Chicago that brings journalists and communities together in a collaborative spirit to produce media that is impactful, equitable and responsive to the public. We operate three year-round programs: our Civic Reporting Fellowship, the Public Newsroom and the Documenters program.
Documenters was launched in spring 2016 to fill information gaps and optimize civic engagement, focusing on public meetings. Anyone can sign up to be a Documenter and, after attending at least one of our trainings, you can get paid to document public meetings via note-taking, live-tweeting and audio-recording. Documenters are making public meetings more accessible and giving us insight on political processes and decision-making as they happen.
As interest in the Documenters program grew, City Bureau quickly ran up against a problem: across the dozens of city and county agencies, there was no centralized calendar of meeting dates. Sifting through poorly formatted websites for meeting information wasn’t the best time of our lives, so in August 2017, we launched the open-source City Scrapers project. Over the better part of the year, our dedicated team of experienced and novice coders built scrapers (programs that pull details from web pages) that collected the meeting information from government websites.
In building an aggregator for public meetings, we had two key goals: identifying assignments for our Documenters, as well as making all local public meeting information truly accessible for everybody. After City Scrapers was finished, we launched Documenters.org, a web application that manages assignments and allows anyone to see and search for public meetings according to location, topic, date, etc.
To review the differences between the City Scrapers code and the Documenters program, see our Next Steps section in this toolkit.
Define your goals
The reasons you’ve ended up here may more or less resemble our own. Either way, before you get too in the weeds, it’s a good idea to consider the following big-picture questions:
- What issues am I most interested in?
- How closely will I follow the City Scrapers model? How can I tailor it to suit my goals?
- What specific pieces of information will be most useful to me?
- What do I see this project accomplishing?
- What assets do I bring to this project? What knowledge, tools, or connections do I need to cultivate?
- What is my timeframe for completion?
- Who is this project serving? How well does my end product deliver?