New York City’s Data is Open—But Not Always Accessible

It appeared the fight for government data was won on Feb. 29, 2012 when then-Mayor Michael Bloomberg signed the historic Open Data Policy into effect. But nearly two years later, the battle still rages on.

The Open Data law was groundbreaking, forcing a flood of data that was once locked in city databases on to a government-sponsored portal that is free for anyone to access online. But as advocacy groups and city officials begin to use the data, it is proving to be messy, incomplete, and in some cases useless in the format in which it is presented.

“I am pretty tech savvy and I can’t understand one damn thing in that portal,” Manhattan Borough President Gale Brewer said at her office’s first official meeting on Jan. 27.

Brewer, a former Council member who pushed heavily for the law despite opposition from the Bloomberg administration, is not alone in her sentiment.

“I don’t think posting the data is enough,” said City Councilman Jimmy Vacca, chairman of the Technology Committee. “The data has to be accessible and easy to access. It has to be understandable to people. Then it should help policy makers, like the Council and the administration, use it to achieve an end.”

Data from the New York Police Department, which includes statistics on crime as well as crash data, has proven to be the most complicated. The NYPD data is presented in a PDF and an Excel spreadsheet. While technically in compliance with the law, advocates and members of the tech community argue the data is not in a computer-readable format.

“Despite the fact we have these open data laws, a lot of information is still being released in an inaccessible format,” said Councilman Ben Kallos. “[The NYPD] should no longer distribute information as PDFs. That would go a long way towards making statistics related to crime and vehicular accidents more accessible.”

Releasing the data in a readable format would allow members of the tech community who have the skills and are passionate about traffic safety to create applications at no cost to the city. These could include crash maps that highlight hot spots, monthly reports, even daily updates.

“Things like that are not pie in the sky. They could happen in a few months if the data was published in a format that worked,” said Juan Martinez, general counsel and legislative director at Transportation Alternatives.

Some projects are already being done. In July of 2013 John Krauss released Crashmapper, which plots the intersections of crashes on a map. The project took over a year to create, in part because he had to convert the data from the PDFs to machine readable. Through the process he discovered some of the data was inaccurate due to human error or the fact several streets cross each other more than once.

Martinez argued that opening up data, and ensuring its accuracy, would arm the Community Board and Council members with the necessary information to make educated decisions to improve pedestrian safety in their neighborhoods.

“They will be proactively trying to solve the worst intersections in their neighborhood,” Martinez said.

Mayor Bill de Blasio had his eye on traffic safety prior to taking office. He signed the Vision Zero pledge as a candidate, vowing to bring the number of traffic deaths in the city to zero. During his first two weeks in office seven pedestrians were killed. On Jan. 15, de Blasio called on Police Commissioner Bill Bratton to make immediate changes, including increased police personnel and having speed cameras issue tickets, not warnings.

De Blasio also commissioned a working group to improve the 50 most dangerous corridors and intersections, expand the number of slow zones, and increase resources to deter speeding and failure to yield. The working group is scheduled to release their report Feb. 15.

The mayor did not mention if he would direct the police department to change how the NYPD releases its data to help achieve Vision Zero. City & State reached out to the mayor’s press office to find out if this were a part of de Blasio’s Vision Zero plan, but received no response.

Susan Petito, assistant commissioner of intergovernmental affairs for the NYPD, attended Brewer’s Jan. 27 meeting. During the meeting members of the tech community expressed their concern over the NYPD data.

“That is something I can take back, in terms of unlocking, not only the traffic data in a more useable form, but also the CompStat pages,” Pedito said. The sentiment was a reversal of the Bloomberg administration’s stance, which had been particularly resistant to releasing data.

Without guidance from the de Blasio administration, the onus falls on the City Council to act legislatively. Legislation worked to make the data accessible, but the tech community and advocates realized through the process that laws create boundaries.

“The production of the crime map is testament to the fact that legislating data availability in a very specific way will produce very specific results,” said Noel Hidalgo, co-founder and executive director of BetaNYC, New York City’s Code for America Brigade. He noted the crime map follows the law that only required the map to be laid out by precincts.

Hidalgo said if the data to make the map were made available, something not required in the bill, civic hackers like himself could pull the data to other applications that use Council or Community Board districts.

There was little dialogue with the Bloomberg administration, which made legislation a necessity. But with a new administration—especially one that has similar pedestrian safety goals—Hidalgo believes legislation won’t need to be the vehicle to enact change.

“We want to have this as dialogue first,” Hidalgo said. “Once we get that, then we can pass some legislation. But I don’t want to use legislation right now just to get there.”