Skip to main content

This site is not affiliated with NYC government.

News & Events

Databook Update 8: Get Normalized Data from our New API

Databook has always been two projects in one.

The first project is our data pipeline that collects, normalizes and republishes normalized NYC Open Data so that other people can use it.

The second is to build a general purpose app that makes this information understandable and useful for the public.

In the beginning of the project, we used CartoDB’s API platform to publish the data that comes out of our pipeline. It was easy to deploy, fast and efficient, but it didn’t allow us to control how we share API access with the public. So, we switched to a custom solution that uses the open source FastAPI framework and PostgreSQL.

You can find our new API, along with Swagger docs, at databook-api.wegov.nyc.

The code for our API is posted on Github.

We want to help people use our normalized data! If you’re interested in using another of the data used in Databook or want to use our data directly from our API, please let us know.

Databook Update 7: The City Record Online Remix

New York City’s “City Record” is the official “journal” where city agencies publish news and event information. This info includes public hearings, procurement announcements, contract awards, rule changes, auctions, personnel changes and more.

We’ve used the city’s open data version, which is updated daily, to integrate The City Record’s contents into the WeGov data system, enabling us to integrate it with the 25+ data sets we already display on Agency Profiles.

Now you can view all of an agencies CIty Record news and events on that agency’s profile. You can also subscribe to that information using the RSS and ICS/iCal feeds we provide for each agency.

We also built a front end for the City Record with a few new features. You can now see how many new items are in each of the City Record’s sections, get RSS and iCAL feeds of all items, and even browse detailed information from auctions of city property.

We hope you enjoy this update and please let us know if you have any ideas for how we can improve any of our products.

Databook Update 6: New “Civil Service Titles” Section Added

New York City government has a workforce over 300,000 people, and that workforce is structured by a system of “Civil Service Titles”. These titles define the job roles and salary of every city employee as well as the skills people need to earn that title.

This month we launched a new section called Titles where you can search and browse Civil Service Titles, view their profiles with information about their salaries and unions, and see how that title connects to other important NYC Open Datasets that describe Positions, Job Opening and Civil Lists.

  • Positions shows the number of people an agency is expected to have with a certain title (ex. the NYPD has positions for 25 Accountants.)
  • Job Openings: these are open positions associate with a civil service title (ex. the city has 13 job openings for Accountants.)
  • Civil List: these are the individuals who work for city agencies with a specific civil service title (ex. see the first initial and last name, and salaries of the 26 accountants employed by the NYPD in 2019.)

We also integrated these datasets into the People section of Agency Profiles and added a two more datasets about agency headcounts and demographics.

We’ve researched how to get more details each civil service titles such as their requirements and exam schedules but found that information hard to extract as it is in the links from this PDF of civil service exam schedules.

If you know of more or better data sources, or have ideas or proposals for what we should do next, please let us know via our community participation platform or simply send us a message.

We look forward to collaborating with you!

Databook Update 5: Interactive Citywide Agency Organization Chart & More!

We’re constantly adding and refining Databook features.

For this one, we focused on improving the organization browsing experience and adding functionality to our districts section.

Organizations

  • NYC Citywide Organization Chart: the interactive, hierarchical organizational chart shows the relationship between city officials and agencies. Click and drag to move the chart. Click on an entity to see its Databook profile.
  • All Organizations: a table view of all organizations in Databook (over 1000) with a column showing the number of datasets to which that organization is linked.

Districts

  • Address Search: type in an address to see all its districts with links out to internal and external pages.
  • Links to Community Board: After submitting your address, click the link to your Community Board to go to its website. There’s no easier way to find your CB and go to it’s website!
  • District Links: We added direct links from Community District to their Population Factfinder profiles.

Expect more updates next month!

Databook named a “Top Websites for Urban Planning – 2021” by Planetizen

We were proud, and quite surprised, to see that the Databook was named one of 2021’s “Top Websites for Urban Planning” by Planetizen, a popular publication in the field of urban planning that’s been around for over 20 years.

Choice quotes below.

WeGovNYC, a civic technology initiative led by the nonprofit Sarapis, is hosting a pair of projects that harness the power of the Internet to provide oversight of government planning and spending. Together, the Capital Project Directory and City Agency Database provide transparency on the activities of the government in ways the city is either unwilling or incapable of doing.

The Capital Project Directory in particular taps into the issue of runaway infrastructure construction costs—an especially prevalent challenge for New York but also around the rest of the country. WeGovNYC is part of a growing effort to identify and diagnose the causes of the highest construction costs for infrastructure projects in the world (the Eno Center for Transportation and NYU’s Marron Institute are among the first to research the issue).  

https://www.planetizen.com/features/115081-top-websites-urban-planning-2021

Read the full article here.

WeGovNYC’s Databook Featured in Local News Story

Gotham Gazette, a news site focused on New York politics, published an analysis of New York City’s capital projects that centers WeGovNYC’s Databook Capital Project Directory.

The story New Analysis Details Just How Late and Over-Budget City Infrastructure Projects Run by Samar Khurshid published on September 14, 2021 references the project throughout the article along with many quotes from Sarapis founder and WeGovNYC project leader Devin Balkind.

Front page news at gothamgazette.com

WeGov, a civic technology initiative, examined available capital project data dating back to 2019 but including projects as far back as 2001, to present an easier picture of the city’s infrastructure landscape, and the results are damning. It examined 5,263 projects with an original estimated total cost of nearly $77 billion, but that has grown to $148.5 billion. Of the total projects, 2,624 are running past their projected timelines and 2,990 are over budget; 997 are starting late and 2,946 are ending late.

It explains the project clearly.

The portal maps out the location of specific capital projects, outlines each project’s budget, the source of their funding, and how spending on projects has changed with different iterations of the budget over the years. It is part of a larger data-driven initiative that also analyzes agency-specific data including everything from headcount to expense budgets to City Council discretionary funding.

And quotes Devin directly about the project’s goals.

“The point of this is to help mobilize the public to take a bigger interest in capital projects, to document capital projects, put pressure on the government to release better data about capital projects, to make it evident to the people who actually produce these projects and to OMB, who kind of oversees this, that the public’s watching,” he added.

The story’s author got quotes from the the powerful Mayor’s Office for Management and Budget, a spokesperson said the agency and city are constantly improving the process and releasing more data.

It ends with a quote from Devin.

Balkind hopes that his data project will help the public understand where their taxes are directed and to push the administration to improve its delivery of projects and services. “We’re off the charts in terms of inefficiency and we pay for that all the time,” Balkind said, “and capital projects are a very obvious way we pay for that.”

Databook Update 3 – September 2021

Add Districts Section where users can select district type to see district boundaries, and then click a district to see it’s Databook data.

  • Added District Types Community Districts (CBs), Council Districts (CDs) and Neighborhood Tabulation Areas (NTAs)
  • Associate mapped Capital Projects with their corresponding districts so they can be browsed by district.
  • Add rollups to see Capital Project stats on a District by District Basis”

Databook Update 1 – July 2021

We added Capital Projects section to the Databook. This involved adding new views:

  • Browse all Capital Projects, sort them by budget size and borough, filter by Borough and Category, search with text.
  • Browse all Capital Projects of a specific agency, sort, filter and search them.
  • View an individual Capital Project Profile, its profile info, changes over time and timeline.