Team

The City Scrapers project was a diverse, inclusive open-source project. We’re proud of the community that grew out of the project, but it was a more intense and long-drawn-out process because of it. We’ve outlined that process below but if you have an on-staff developer or a budget to contract a developer, they should be able to knock out all the scrapers within a couple months, depending on the scope of the project.

Staff developer

Follow your organization’s guidelines.

Contract developer(s)

Contracting

Follow your organization’s guidelines.

Bounty system

A bounty is a cash reward offered for development. Bounties are usually offered as an incentive for fixing software bugs or implementing minor features but, paired with a hands-on approach to project management, they can also be used for short-term, well-defined projects such as the City Scrapers project. To implement a bounty system, select coders who you know and trust, work out a rate per spider built by each coder and bundle similar scapers in groups on your Agency Table for more efficient development. Bundling scrapers has the added benefit of allowing coders of varying skill levels to work independently while helping each other along the way. In June 2018, City Bureau commissioned a set of public meeting scrapers for Detroit that operated on a bounty system.

Web tools like Bountysource and Bountify can simplify this process.

Volunteer Coding Community

Forking the template repo

If you plan on maintaining a City Scrapers project in your area, fork our city-scrapers-template project on GitHub and follow the instructions there to get started.

Convening a community

Plan ahead so your volunteers feel secure within your project scope. How many staffers can you commit to the project? How many coders can you handle at any given time? How long are the meetings, on which day and at what time? Will you be providing food for each meeting? (The answer is yes). What mode of communication will you use across the life of the project—email might work for the first week but may quickly become inefficient as your community grows.

Answering these questions and others before work begins will have a profound effect on your project. Take the time to plan and keep scale in mind. The first City Scrapers community was maintained over the course of a year in Chicago, but that doesn’t mean it takes a year is the minimum length of time. Your project plan should be in tune with, or adapt to, your timeline.

Outreach

Begin with a list of independent coders, developers who operate networks of coders (such as your local Code for America) and adjacent technologists who can serve as ambassadors for your project within your local tech community.

If starting from scratch, be sure to create an easy to use sign-up form (we recommend a Google Form or Airtable) and add a link in your message. Go the extra mile and draft a informational document that all interest organizations and individuals can refer to in deciding whether they’d like to participate. Explanatory language that can be emailed widely to folks on you list will give responders a central place to view content (Hint: using Google Docs can reduce the number of emails sent as a result of any changes in the project plan). Click here to see our informational document. And here’s an email example from the Chicago City Scrapers project:

Hi all, City Bureau is looking for experienced and emerging coders for the first phase of an open source project. We’ll be collecting public meetings in Chicago from a variety of disparate sources and creating scrapers, chrome extensions, and other tech tools to funnel information on the dates, times and locations of those public meetings to a central database.

This will be a volunteer-based project but a great opportunity for coders looking to expand their experience with scrapers and learn new skills. In addition to City Bureau staff, Pro Publica Illinois News Applications Developer David Eads will be on hand to host workshops and tutorials once a week at City Bureau’s South Side newsroom during our Tuesday working sessions. Folks will need to have/share laptops, be willing to work with the command line and be comfortable writing code. And possess a generally high tolerance for some initial frustration—which we will work through together.

Volunteers who contribute to the open source public meetings aggregator will be credited as collaborators. All contributors are encouraged to attend all 5 working sessions.

Who: City Bureau: a nonprofit, South Side civic journalism lab that brings journalists and communities together in a collaborative spirit to produce responsible media coverage and encourage civic participation.

When: Every Tuesday in August, 4-8p (August 1, August 8, August 15, August 22, August 29)

Where: Build Coffee, 6100 S. Blackstone Ave.

To sign up: link

For more information: https://docs.google.com/document/d/1sXS2VaCRpzr6ENd7loCNBqTeNniJ3nQUF-c0nwfhnfg/edit?usp=sharing

Refining the goal

We found that the biggest draw for City Scrapers volunteers was the practical value of the project. When onboarding new volunteers, we thoroughly explained the problem we were trying to solve and how their collective efforts could produce accountability and transparency for our city. Coders often mentioned that this motivation kept them engaged even when they struggled with a particular piece of code. Having a goal in mind, in our case the publication of Chicago’s public meetings in a single location, helped keep our community united and focused on the tasks at hand.

Providing educational opportunities

Pair-sharing: Pair programming is the practice of pairing up coders to work on programming tasks. This can be done by pairing up two developers seated at the same computer, sharing a keyboard and monitor but with the increasing popularity of remote coding platforms, it’s now possible to pair-share from anywhere in the world. Pair-sharing typically involved role-swapping, where one coder (typically the more advanced of the pair, demonstrates a skill while the other watches then demonstrates the same skill. Lessons learned during pair-sharing can then be repeated and expanded upon during independent work sessions. We found that pair-sharing worked especially well in the City Scarpers project because of our emphasis on coders with varying skill levels working together toward a common goal. This involved assessing the skill level of incoming coders and matching them with less experienced developers on a recurring basis, typically at each in-person meeting. Pair-sharing can last for any length of time as coders learn new skills and pass on their skills to others.

Programs like Donut can be integrated with Slack.

Maintaining your community

Over the course of a year our community of experienced and novice coders worked together to build code to scrape meeting information from city websites in Chicago. The City Scrapers community met in-person regularly, collaborated on GitHub and communicated via Slack. As our community grew in the first quarter of the project we created a committee of volunteers who demonstrated skill, leadership and interest in being more involved with the project. We soon increased our in-person meetings from bi-monthly to weekly. In collaboration with City Bureau staff, our committee outlined long-term goals, coordinated with partners and helped orient newcomers to the project. This last piece was crucial for integrating newcomers without coding experience as our community grew to more than 40 contributors by the end of the Chicago project in September 2018.

That same month, the first City Scrapers community outside of Chicago was started in Pittsburgh by a member of the Chicago project who relocated for work.

Scheduling coding meet-ups and check-ins

Define your availability internally. We recommend weekly or bi-monthly meetings but it’s your call.

Define a process for lead staff/volunteers at each meeting. Our Coding Session Staging document was used to informally plan roles for each of our in-person coding sessions.

One-time events

Our Design-a-thon was a one-day challenge to design a tool that uses the data collected by the scrapers. It helped us collect ideas about how to best share the information with the public, boosted our visibility and was a fun time to generally make connections in the Chicago tech scene.

Online communication

Slack is a free(mium), easy way to digitally convene your community so that coders can stay engaged and informed even if they can’t make it to every meeting.

Developing a Code of Conduct

This step is often overlooked but often critical to success. Define a Code of Conduct that all coding community agree to abide by throughout their time on the project. The Contributor Covenant inspired an early mission for the open-membership, open-source City Scrapers project, outlined in the project README file located on GitHub. In addition to the Covenant, this document states that “The City Bureau Labs community welcomes contributions from everyone. We prioritize learning and leadership opportunities for under-represented individuals in tech and journalism.

We hope that working with us will fill experience gaps (like using git/github, working with data, or having your ideas taken seriously), so that more under-represented people will become decision-makers in both our community and Chicago’s tech and media scenes at large.”

Here’s the Chicago City Scrapers Code of Conduct as an example (feel free to adapt as needed).

Reward your team

Pizza party? BBQ? Field Day? Reward hard work with a collective gathering that reflects your team’s mission—a simple survey or Doodle poll to solicit good dates and times is helpful to make sure as many people as possible can attend.

Additional Resources

Our lead coding committee volunteer Rebecca Wei produced a series of video tutorials that can you and your staff/coding community navigate and troubleshoot common issues.