4 ms·

The US Census now has an API

173 points by ams1 14y ago

DrJokepu 14y ago
Wouldn't this be better served by a downloadable SQLite file rather than a web service?
- mparlane 14y ago
  How big do you imagine this file would be ? And I am sure someone could eventually create one..
- systematical 14y ago
  It's database agnostic this way and every language supports JSON.
  - jlarocco 14y ago
    Partly to play devil's advocate and partly out of curiosity, do you know of a language with a JSON library but no SQLite library?
    - alecperkins 14y ago
      Browser-based JavaScript. Their CORS and JSONP support suggests they have this use case in mind.
- albertsun 14y ago
  The downloadable files for this dataset are much larger than anyone would want to download at one go.
- smacktoward 14y ago
  The Census Bureau has made their data available for download for a looong time, both over the Web (including a bunch of different interfaces for slicing out just the data you want -- see http://www.census.gov/main/www/access.html http://www.census.gov/main/www/access.html) and FTP (http://www2.census.gov/ http://www2.census.gov/) All the stuff that's available via the Web service is already available via download, along with a lot more. They don't provide it in SQLite format AFAIK, but they do offer a range of other formats that should be easy to get into SQLite if you wish. If there's any problem with their downloadable data offerings, it's that they have so many of them that it can be difficult to figure out exactly which one has the data you're looking for.
- Yen 14y ago
  for you, and your repliers: this data is already available as a downloadable file. Check out ftp://ftp.census.gov/ It's CSV, and it's kinda ugly-looking, but it's available. The 'full' data set is ...rather huge. The full data set for any particular report is rather huge. The advantage of this API, I expect, is it allows people to make quick, machine readable queries for useful subsets of the data. If you want a JavaScript mashup that tells you the number of people who live on your block, use this. If you want to know the population of every block everywhere for data-crunching purposes, just download the CSV.
  - awaythrow 14y ago
    Excellent. Much easier than requesting API keys and parsing JSON. But that's just my opinion. CSV is beautiful. Simple and easy to manipulate into some other format if desired.
    - fennecfoxen 14y ago
      CSV is beautiful - for numbers. When you invoke strings, and quoting, and whatnot, it goes downhill fast. :)
- josephcooney 14y ago
  As other have already commented, this is already available as a download. You can even fire up an EC2 instance with the data pre-downloaded http://aws.amazon.com/datasets/Economics?_encoding=UTF8&jiveRedirect=1 http://aws.amazon.com/datasets/Economics?_encoding=UTF8&...
jack-r-abbit 14y ago
> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that example into a JSON decoder you get an extremely more difficult to use object. Am I missing something? or is this just typical "the government doesn't do tech properly" stuff? EDIT: sorry, I try to be less negative but sometimes it is hard. I do applaud them for at least making the info available. I don't have a use for it but if some else does then dealing with a stupid format is better than not having data at all. EDIT2: (to clarify) I was not saying the data should just be CSV... I'm saying it is and that defeats the purpose of using JSON. They should still use JSON but with their data formatted differently. Using their example, properly closed but truncated to just to first 2 records, the PHP function json_decode() turns it into this array: array ( 0 => array ( 0 => 'P0010001', 1 => 'NAME', 2 => 'state', ), 1 => array ( 0 => '710231', 1 => 'Alaska', 2 => '02', ), 2 => array ( 0 => '4779736', 1 => 'Alabama', 2 => '01', ), ) I don't find that format to be very pleasant.
- tantalor 14y ago
  I also found this odd, but I can think of two reasons, 1. CSV is not a single well-defined format. Some people use rfc4180, but not everybody does. 2. JSON is easier to parse than CSV for JavaScript clients, i.e., browsers. On the other hand, JSON is not a subset of JavaScript.
  - deleted 14y ago
    [deleted]
- pnathan 14y ago
  It's very simple to transform that kind of data into $your_preferred_structure. It's also a very agnostic format.
  - deleted 14y ago
deleted 14y ago
[deleted]
bryanjclark 14y ago
Cool, especially since the census is one of the main reasons why we have computers at all. I went to the Computer History Museum a few months back, and when we were looking through the origins of the modern computer, a lot of it traces back to census needs: we needed to automate the counting of large quantities of uniformly formatted data, so we used punchcards to tally up the data. Fast forward to 2012, and the census is now getting an API. Weird how that works.
adelevie 14y ago
If anyone is interested, the FCC has a fairly comprehensive list of Developers pages for other federal agencies: http://www.fcc.gov/developers http://www.fcc.gov/developers (on the right column). Also, for further reading on the topic of .gov APIs, http://ben.balter.com/2012/06/02/publishing-government-data-that-developers-will-actually-use/ http://ben.balter.com/2012/06/02/publishing-government-data-... is a great start.
jcampbell1 14y ago
Wow, this is good government. I can nitpick the API details, but this is one giant leap in the right direction. Under US law, the federal government can't claim copyright on works produced via tax dollars (makes sense). Since the feds can't require us to provide attribution for all this helpful data, how do we as a community advocate for more open data like this?
acoyfellow 14y ago
This opens my mind up to a whole new slew of ideas using their information so easily..
mkramlich 14y ago
better if was URL of a tarball of structured human-friendly data
alister 14y ago
You need to request a "key" (essentially a user ID) before you can make queries. I can imagine only 3 reasons for this: (1) they want to make you agree to the Terms & Conditions, (2) it gives a way to choke off a DDOS attack that makes repeated complex queries, or (3) the census bureau wants to track how you are using its service. The key is indeed easy to get, but I observe that Google would have exactly the same concerns as 1,2,&3 above, yet they somehow manage to stay in business without making their users sign up to do a web search. The federal government always makes things a little more complicated.
- chernevik 14y ago
  (3B) Census data on some topics can move markets. If someone hacks to get a five minute lead time, the key policy gives them a start on figuring out who and maybe how.
  - fennecfoxen 14y ago
    Right. Because if I hack a website I'm totally going to log in with a totally real set of credentials traceable back to me. Besides, this is the 2010 American Community Survey. Pretty sure it's moved all the markets it's going to move by now.
- yahelc 14y ago
  Google and the Census both don't require individual users to signup to access their data. However, both require that their users signup for an API key to do automated querying. (In fact, Google charges for its Web Search API now.) It's pretty standard for any API to want to be able to identify and potentially meter usage.
10dpd 14y ago
Has anyone considered possible use-cases for this dataset? It's great to get excited about the data format, but actual use-cases currently seem a little limited to me. Of course this will hopefully change as more and more datasets become available.
- mayel 14y ago
  Here is a quick example I put together this weekend (using data from their Excel files, not the API - didn't know about it!) http://babelverse.com/linguistics/funfacts/ http://babelverse.com/linguistics/funfacts/ Looking forward to many more visualizations and analysis thanks to the API :)
blt 14y ago
It's pretty cool that they are putting up the machine time to run queries on the data. I wonder how long it will last if someone makes a popular web app using the census API.
taylorbuley 14y ago
When you sign up for an API key you get the message "Happy querying!" Quite the model of transparency.
andyjohnson0 14y ago
Slightly OT, but UK census data can also be accessed by a web API. Some details are at http://www.programmableweb.com/api/office-for-national-statistics http://www.programmableweb.com/api/office-for-national-stati.... The bad news is that it appears to be SOAP-based.

173 points by ams1 14y ago

DrJokepu 14y ago

Wouldn't this be better served by a downloadable SQLite file rather than a web service?

mparlane 14y ago
How big do you imagine this file would be ? And I am sure someone could eventually create one..
systematical 14y ago
It's database agnostic this way and every language supports JSON.
- jlarocco 14y ago
  Partly to play devil's advocate and partly out of curiosity, do you know of a language with a JSON library but no SQLite library?
  - alecperkins 14y ago
    Browser-based JavaScript. Their CORS and JSONP support suggests they have this use case in mind.
albertsun 14y ago
The downloadable files for this dataset are much larger than anyone would want to download at one go.
smacktoward 14y ago
The Census Bureau has made their data available for download for a looong time, both over the Web (including a bunch of different interfaces for slicing out just the data you want -- see http://www.census.gov/main/www/access.html http://www.census.gov/main/www/access.html) and FTP (http://www2.census.gov/ http://www2.census.gov/) All the stuff that's available via the Web service is already available via download, along with a lot more. They don't provide it in SQLite format AFAIK, but they do offer a range of other formats that should be easy to get into SQLite if you wish. If there's any problem with their downloadable data offerings, it's that they have so many of them that it can be difficult to figure out exactly which one has the data you're looking for.
Yen 14y ago
for you, and your repliers: this data is already available as a downloadable file. Check out ftp://ftp.census.gov/ It's CSV, and it's kinda ugly-looking, but it's available. The 'full' data set is ...rather huge. The full data set for any particular report is rather huge. The advantage of this API, I expect, is it allows people to make quick, machine readable queries for useful subsets of the data. If you want a JavaScript mashup that tells you the number of people who live on your block, use this. If you want to know the population of every block everywhere for data-crunching purposes, just download the CSV.
- awaythrow 14y ago
  Excellent. Much easier than requesting API keys and parsing JSON. But that's just my opinion. CSV is beautiful. Simple and easy to manipulate into some other format if desired.
  - fennecfoxen 14y ago
    CSV is beautiful - for numbers. When you invoke strings, and quoting, and whatnot, it goes downhill fast. :)
josephcooney 14y ago
As other have already commented, this is already available as a download. You can even fire up an EC2 instance with the data pre-downloaded http://aws.amazon.com/datasets/Economics?_encoding=UTF8&jiveRedirect=1 http://aws.amazon.com/datasets/Economics?_encoding=UTF8&...

jack-r-abbit 14y ago

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that example into a JSON decoder you get an extremely more difficult to use object. Am I missing something? or is this just typical "the government doesn't do tech properly" stuff? EDIT: sorry, I try to be less negative but sometimes it is hard. I do applaud them for at least making the info available. I don't have a use for it but if some else does then dealing with a stupid format is better than not having data at all. EDIT2: (to clarify) I was not saying the data should just be CSV... I'm saying it is and that defeats the purpose of using JSON. They should still use JSON but with their data formatted differently. Using their example, properly closed but truncated to just to first 2 records, the PHP function json_decode() turns it into this array: array ( 0 => array ( 0 => 'P0010001', 1 => 'NAME', 2 => 'state', ), 1 => array ( 0 => '710231', 1 => 'Alaska', 2 => '02', ), 2 => array ( 0 => '4779736', 1 => 'Alabama', 2 => '01', ), ) I don't find that format to be very pleasant.

tantalor 14y ago
I also found this odd, but I can think of two reasons, 1. CSV is not a single well-defined format. Some people use rfc4180, but not everybody does. 2. JSON is easier to parse than CSV for JavaScript clients, i.e., browsers. On the other hand, JSON is not a subset of JavaScript.
- deleted 14y ago
  [deleted]
pnathan 14y ago
It's very simple to transform that kind of data into $your_preferred_structure. It's also a very agnostic format.
- deleted 14y ago

deleted 14y ago

[deleted]

bryanjclark 14y ago

Cool, especially since the census is one of the main reasons why we have computers at all. I went to the Computer History Museum a few months back, and when we were looking through the origins of the modern computer, a lot of it traces back to census needs: we needed to automate the counting of large quantities of uniformly formatted data, so we used punchcards to tally up the data. Fast forward to 2012, and the census is now getting an API. Weird how that works.

adelevie 14y ago

If anyone is interested, the FCC has a fairly comprehensive list of Developers pages for other federal agencies: http://www.fcc.gov/developers http://www.fcc.gov/developers (on the right column). Also, for further reading on the topic of .gov APIs, http://ben.balter.com/2012/06/02/publishing-government-data-that-developers-will-actually-use/ http://ben.balter.com/2012/06/02/publishing-government-data-... is a great start.

jcampbell1 14y ago

Wow, this is good government. I can nitpick the API details, but this is one giant leap in the right direction. Under US law, the federal government can't claim copyright on works produced via tax dollars (makes sense). Since the feds can't require us to provide attribution for all this helpful data, how do we as a community advocate for more open data like this?

acoyfellow 14y ago

This opens my mind up to a whole new slew of ideas using their information so easily..

mkramlich 14y ago

better if was URL of a tarball of structured human-friendly data

alister 14y ago

You need to request a "key" (essentially a user ID) before you can make queries. I can imagine only 3 reasons for this: (1) they want to make you agree to the Terms & Conditions, (2) it gives a way to choke off a DDOS attack that makes repeated complex queries, or (3) the census bureau wants to track how you are using its service. The key is indeed easy to get, but I observe that Google would have exactly the same concerns as 1,2,&3 above, yet they somehow manage to stay in business without making their users sign up to do a web search. The federal government always makes things a little more complicated.

chernevik 14y ago
(3B) Census data on some topics can move markets. If someone hacks to get a five minute lead time, the key policy gives them a start on figuring out who and maybe how.
- fennecfoxen 14y ago
  Right. Because if I hack a website I'm totally going to log in with a totally real set of credentials traceable back to me. Besides, this is the 2010 American Community Survey. Pretty sure it's moved all the markets it's going to move by now.
yahelc 14y ago
Google and the Census both don't require individual users to signup to access their data. However, both require that their users signup for an API key to do automated querying. (In fact, Google charges for its Web Search API now.) It's pretty standard for any API to want to be able to identify and potentially meter usage.

10dpd 14y ago

Has anyone considered possible use-cases for this dataset? It's great to get excited about the data format, but actual use-cases currently seem a little limited to me. Of course this will hopefully change as more and more datasets become available.

mayel 14y ago
Here is a quick example I put together this weekend (using data from their Excel files, not the API - didn't know about it!) http://babelverse.com/linguistics/funfacts/ http://babelverse.com/linguistics/funfacts/ Looking forward to many more visualizations and analysis thanks to the API :)

blt 14y ago

It's pretty cool that they are putting up the machine time to run queries on the data. I wonder how long it will last if someone makes a popular web app using the census API.

taylorbuley 14y ago

When you sign up for an API key you get the message "Happy querying!" Quite the model of transparency.

andyjohnson0 14y ago

Slightly OT, but UK census data can also be accessed by a web API. Some details are at http://www.programmableweb.com/api/office-for-national-statistics http://www.programmableweb.com/api/office-for-national-stati.... The bad news is that it appears to be SOAP-based.