Rules Reform Outlines Next Steps in Open Data

We are taking dramatic steps towards more transparency so members of the press and public can see how our legislative process works … – East Harlem Council Speaker Melissa Mark-Viverito

In a great article today by the Gotham Gazzette  ow on May 14 the New York City Council unanimously passed a rules reform package that will change the internal operation of the 51-member legislative body. While member item reform has received much of the press, the Council is also committing to making its actions much more transparent by altering how some data will be presented on its website.

The City Council website is a wealth of information. Visitors can find all currently proposed bills, member voting on particular bills, legislation sponsorship information, and proposed and/or passed legislation dating back to 1998. You can even watch hearings online if you couldn’t make it to City Hall (see it on Harlem World Magazine home page on the right hand column).

Unfortunately, even for frequent users of the site, finding some of that information can be a cumbersome task. If you know your way around you can, for example, find how each council member voted on a bill. But, you can’t find a complete voting record for each council member. You can find out if a member was present at a hearing, but you can’t find their total attendance record for the session (you can for kids in schools but not for city council members).

Some information, like what bills a council member has sponsored and whether those bills have passed, are available through the searchable database and can even be exported into an Excel spreadsheet. Having data in a spreadsheet is better than in a PDF to search and manipulate the information, but it is by no means easy to compare multiple members this way. It is also not effective for compiling all laws already on the books or proposed on a particular topic (all bike laws for example).

The new rules will not fix the site’s problems entirely, but are a huge first step. The legislative tracking rule (rule 5.110) will require the Council to publish all proposed laws and resolutions and “any information associated with each proposed local law or resolution” in a machine-readable format “as soon as practical.” This includes committee reports, agendas, calendars, hearing testimony, transcripts, committee assignments, and voting records, according to the language in the bill.

We are taking dramatic steps towards more transparency so members of the press and public can see how our legislative process works …

“We are taking dramatic steps towards more transparency so members of the press and public can see how our legislative process works,” Council Speaker Melissa Mark-Viverito said at City Hall on May 14.

Most of the data on the council site is currently available in Word document or PDF form. The new rule will require the Council to publish in a format, machine readable, that third party developers can utilize. Developers will be able to use the raw data to create endless user-friendly applications such as a council member vote tracker application, an attendance tracker application, or an app that shows all legislation being proposed related to a specific topic.

The MTA is a great example of this system already at work. It began releasing bulk data on everything from train schedules to lost and found items and for several years now, developers have been creating mobile apps using the data. Several of these apps are used by New Yorkers every day to find what transportation lines are delayed or how long a trip will take. The MTA created a yearly contest to create official MTA apps like MTA Subway Time, but there are popular third party-hosted apps like iTransNYC.

“In so many areas of our lives, access to data has helped transform how people understand what is going on in the world,” Council Member Brad Lander said at City Hall on May 14. “This will help people understand better what the Council is doing and evaluate what the Council is doing.”

Timeline for Change
How and when these changes to the city council website will take place is still up in the air. Lander said he hoped the Council could get the details sorted out by the end of the year, but several things will need to change to make it happen.

The council website runs the Legistar system which is used by municipalities around the country. The system can export some data into Excel spreadsheets, but currently the site does not have a feature to produce and display all of the data required by the new rule in an open data format.

Council staff indicated the Council is in ongoing discussions with Granicus, the vendor that runs the Legistar system, about what features can be available to accommodate its needs. Due to an increase in requests for open data around the country, there are a number of other cities pushing for similar features, so changes could be made in the coming months, according to the staff member.

People anxious to obtain council data may not have to wait for Granicus. The Sunlight Foundation, a non-partisan non-profit that uses technology to make government accountable, is eager to step in in the meantime.

Sunlight is currently scraping data on the council website to put on their new city-focused website, opencivicdata.org. The site aims to be a legislative resource similar to their previous project, openstates.org.

Data scraping, building code to read and pull data from a non-machine readable format like a PDF or Word document, is not fool-proof: extremely tech-savvy individuals can, at times, manipulate the data; scrapers can miss pieces of information.

“Scraping is not 100 percent perfect,” Rebecca Williams, a policy analyst for Sunlight Foundation said by phone May 8, adding that this is why they want the Council to release the data as open data. “If they change the site, our scrapers can break.”

Williams said she hopes to see the Council eventually use an open application program interface (API). This format, used by the MTA for much of its data and subsequent apps, allows developers to pull proprietary data directly from a site. It is currently the most effective way for developers to build applications.

“By opening up the legislative process with an open API we can empower civic hackers and entrepreneurs to create applications that make our government accessible and accountable to all New Yorkers,” Council Member Ben Kallos said in an emailed statement. Kallos, a long-time open data advocate, said he applauds the rules reform, adding it is a technological leap forward.

Public Technology Plan
The value of opening up data as a way to increase transparency is unquestionable, but simply having data accessible in machine readable format cannot be the end point.

“The whole concept behind open data is that the government is going to tell people what they are up to,” Michael Flowers, urban science fellow at the Center for Urban Science and Progress, and the city’s first chief analytics officer, said by phone May 9. “Simply releasing data sets on an open data portal doesn’t achieve that. It is a necessary start, but it doesn’t achieve that.”

After historic open data laws were passed in 2012, the City of New York created an online portal with over 1100 sets of machine readable data. While applications have been created by both the City and third party developers, there are still mountains of data that lay untouched.

“Some of the metrics that get applied for the success of an open data program are kind of ridiculous,” Flowers said. “I would judge the program by what people have done with it.”

As it sits in its current format, the data is unreadable to people who do not have the skill to code – which is the vast majority of the people in the city.

“It is not a cool thing for the government to get a clubby relationship with a very narrow segment of the 8.5 million New Yorkers to the exclusion of the rest of the 8.5 million,” Flowers said.

He suggested that keeping developer needs in mind when releasing the data is critical, but there must be a balance, as well as some accountability.

“If [developers] want people to rely on them, they will have to come up with tools that allow the public to access that information in a meaningful way as well,” Flowers said.

The Council, which, ironically, passed the open data law for the city but lags behind on its own site, will be able to learn from the City in dealing with open data. As part of its rules package the Council will create a Public Technology Plan. The plan is only a commitment at this point, but will be a way for the Council to roll out changes related to open data without codifying it in its official rules. Putting technology details in the rules, such as requiring a certain type of code, could actually be counterproductive because technology changes so rapidly and the rules can’t be modified again until the next session in 2018.

“We can’t fully evaluate the council’s rule changes as they pertain to embracing technology until the technology plan is created and implemented, but their intent appears to be to expand the use of technology to make the Council more accessible to the public,” Alex Camarda, director of public policy at Citizens Union*, said by email May 14.

There is no formal committee to which the technology plan is assigned, although the technology committee is a likely landing spot, according to a source in the Council. The source said key players who use open data would likely be consulted to make recommendations on the plan, but there was no determination as to whether these meetings would be open to the public or if proposals would be made and then presented for consideration at a public hearing (source).

 Photo Council members before their May 14 Stated Meeting (William Alatriste).