25 ms·

Featured dataset cannot be downloaded

60 points by yid 13y ago

yid 13y ago
(Submitter here) This is so frustrating. There are so many links to important datasets that are just broken, such as the NHTSA crash ratings: http://catalog.data.gov/dataset/new-car-assessment-program-ncap-5-star-safety-ratings http://catalog.data.gov/dataset/new-car-assessment-program-n... (try clicking "download" for an empty XML file) The others link to agencies' download pages. It feels like they forgot that the purpose of data.gov is data first.
- sureshv 13y ago
  I've been attempting to extract various bits of government data and found the process to be quite tedious. The fastest way has been to use google vs data.gov to find particular data sets. Edit: Just add '-site:data.gov' to your google search to help find the actual data set ;-)
- 8ig8 13y ago
  I googled that dataset. I was happily going to post this link from my search and be your hero: https://explore.data.gov/Science-and-Technology/DOE-Green-Energy-Patents-Data-Service/4dnd-s6im https://explore.data.gov/Science-and-Technology/DOE-Green-En... But, I figured I'd try to download the CSV to make sure it works. No dice. I end up with this: http://www.osti.gov/home/RETIRED/greenenergy?format=csv http://www.osti.gov/home/RETIRED/greenenergy?format=csv "The DOE Green Energy product has been discontinued. All of the information in DOE Green Energy can be found in SciTech Connect." OK, so I follow the link to SciTech Connect: http://www.osti.gov/scitech/ http://www.osti.gov/scitech/ Of course it is to a general landing page. So I search on that page for: "DOE Green Energy" (using quotes) And I'm presented with one result: "Ellens Test2". Here's a screenshot: http://i.imgur.com/08rHfcZ.png http://i.imgur.com/08rHfcZ.png If that's what you're looking for, then I found it. If not, I give up.
  - yid 13y ago
    > If that's what you're looking for, then I found it. If not, I give up. Thank you for the effort :) Unfortunately, I have no idea what that is, but someone could probably tell Ellen that it works.
jowiar 13y ago
They're offering a pretty decent chunk of change if someone wants to fix it: https://www.usajobs.gov/GetJob/ViewDetails/363324500 https://www.usajobs.gov/GetJob/ViewDetails/363324500
- eli 13y ago
  It's tough to just walk into a GS-14 job if you don't already work for the feds.
- yid 13y ago
  Something makes me feel like that might be an uphill battle. The focus of the entire site appears to be a slick superficial UI. There is no attempt to consolidate data from various agencies into a uniform access format (say a JSON API or periodic dumps to a user-pays S3 bucket). The data is in XML or CSV or even XLS. The whole implementation seems a bit misguided, to be honest.
  - eplanit 13y ago
    I agree. I wonder if CGI (who also developed healthcare.gov) was hired to implement it.
- Glyptodon 13y ago
  Just reading about the requirements and hoops for applying makes me not want to apply.
stevejohnson 13y ago
This headline is very editorialized. Submitter should write a short blog post and submit that.
- yid 13y ago
  Sorry, I thought this was worth sharing and didn't have the inclination to write a blog post.
  - alialkhatib 13y ago
    Those clauses seem contradictory. You have a very clear (and in my opinion on point) critique of what you're presenting us, but you don't have the inclination to write a blog post? You even wrote your own top-level comment providing some context. That would have made a sufficient blog post right there. (A great blog post might have compared bad vs good cases, or provided some suggestions on improving, or even looked more broadly at whether this was an outlier of badly structured/provided data or just the norm) The whole "Try (downloading|accessing) this..." strikes me as the same kind of title as Upworthy's "You'll never believe this..." in that it seems vaguely like link-baiting. I'm a little too curious not to click just to see what the dataset is about.
    - yid 13y ago
      > You have a very clear (and in my opinion on point) critique of what you're presenting us, but you don't have the inclination to write a blog post? Correct. I don't have a blog, and don't care to set one up. And I'd imagine a .gov link would have a lower chance of being linkbait.
      - alialkhatib 13y ago
        There are mediums (pun not intended) where you can make the equivalent of a blog post without much effort (Medium, Quora, I'm sure there are others). And link-bait is more about the marketing used to get people to click on the link (e.g. editorializing), isn't it? I don't want to get into semantics, so let's consider this a minor detail, because the more important matter is that you're editorializing. HN has a pretty clear (and helpful) set of guidelines on posting: http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html >If you want to add initial commentary on the link, write a blog post about it and submit that instead. I'm not trying to harangue you or anything; you make a good point but it's hamstrung by a length-limited title section.
bubbafat 13y ago
The data set disappearing is explained a bit more here: http://www.osti.gov/home/RETIRED/greenenergy?format=csv http://www.osti.gov/home/RETIRED/greenenergy?format=csv Which leads to here: http://www.osti.gov/home/ostiblog/osti-re-focusing-and-re-balancing-its-operations-%E2%80%93-and-refreshing-its-home-page-%E2%80%93-advance-p http://www.osti.gov/home/ostiblog/osti-re-focusing-and-re-ba... Which explains the data is here: http://www.osti.gov/scitech/ http://www.osti.gov/scitech/ Which I appreciate is not what you wanted - but it's what you got.
pbbakkum 13y ago
This is a problem close to my heart, thanks for pointing it out. I have a project up at http://commonwealth.io http://commonwealth.io attempting the reduce the barrier to using data like this. Even when the data is accessible, it could be in any format and you have to load it into a DB to get any value. The idea is to enable real SQL queries directly on data sets, so you don't need to worry about accessibility or how you'll query it. Not much data in it now, and it doesn't solve your immediate problem, but perhaps a better model for storing and representing this data.
ChuckMcM 13y ago
Interesting, I left a note saying the dataset was unavailable. It looks like it was created 22-Apr-2010 so it is probably fairly stale at this point.
bartleeanderson 13y ago
I found this link after playing around at their site Then I just entered green energy as a search term Hope this helps http://www.osti.gov/doepatents/index.jsp http://www.osti.gov/doepatents/index.jsp
ewams 13y ago
This is a great example of a use case for using Bittorrent Sync. They just need to create a read-only secret for each dataset they want to share and then put it on the website. Then the data is P2P'd for cheaper and much easier to maintain. Plus if the Government shuts down or has an issue the data can still be transferred.
dlapiduz 13y ago
The project is open source, you can submit an issue (or a PR) here: https://github.com/gsa/data.gov https://github.com/gsa/data.gov
philipashlock 13y ago
Thanks for pointing this out. We've updated the highlighted dataset to something that is better maintained, but if you find other problems please do ticket them on github (https://github.com/GSA/data.gov/issues https://github.com/GSA/data.gov/issues) These kinds of issues will improve as a new approach to listing datasets is implemented. This is currently underway and is described on Project Open Data - http://project-open-data.github.io/implementation-guide/ http://project-open-data.github.io/implementation-guide/ We're also working on some basic automated tests to ensure listings have the correct MIME type (which is incorrect far too often) and that URLs aren't 404s or redirects or HTML pages rather than raw data. For those interested, please do consider applying for the open position - https://www.usajobs.gov/GetJob/ViewDetails/363324500 https://www.usajobs.gov/GetJob/ViewDetails/363324500 I'll do my best to answer any questions about the position and the process of applying.
murphysbooks 13y ago
Each gov agency is now posting their dataset metadata on their own sites at agency.gov/data.json which should contain the url for the dataset. See http://project-open-data.github.io/schema/ http://project-open-data.github.io/schema/ for details

60 points by yid 13y ago

yid 13y ago

(Submitter here) This is so frustrating. There are so many links to important datasets that are just broken, such as the NHTSA crash ratings: http://catalog.data.gov/dataset/new-car-assessment-program-ncap-5-star-safety-ratings http://catalog.data.gov/dataset/new-car-assessment-program-n... (try clicking "download" for an empty XML file) The others link to agencies' download pages. It feels like they forgot that the purpose of data.gov is data first.

sureshv 13y ago
I've been attempting to extract various bits of government data and found the process to be quite tedious. The fastest way has been to use google vs data.gov to find particular data sets. Edit: Just add '-site:data.gov' to your google search to help find the actual data set ;-)
8ig8 13y ago
I googled that dataset. I was happily going to post this link from my search and be your hero: https://explore.data.gov/Science-and-Technology/DOE-Green-Energy-Patents-Data-Service/4dnd-s6im https://explore.data.gov/Science-and-Technology/DOE-Green-En... But, I figured I'd try to download the CSV to make sure it works. No dice. I end up with this: http://www.osti.gov/home/RETIRED/greenenergy?format=csv http://www.osti.gov/home/RETIRED/greenenergy?format=csv "The DOE Green Energy product has been discontinued. All of the information in DOE Green Energy can be found in SciTech Connect." OK, so I follow the link to SciTech Connect: http://www.osti.gov/scitech/ http://www.osti.gov/scitech/ Of course it is to a general landing page. So I search on that page for: "DOE Green Energy" (using quotes) And I'm presented with one result: "Ellens Test2". Here's a screenshot: http://i.imgur.com/08rHfcZ.png http://i.imgur.com/08rHfcZ.png If that's what you're looking for, then I found it. If not, I give up.
- yid 13y ago
  > If that's what you're looking for, then I found it. If not, I give up. Thank you for the effort :) Unfortunately, I have no idea what that is, but someone could probably tell Ellen that it works.

jowiar 13y ago

They're offering a pretty decent chunk of change if someone wants to fix it: https://www.usajobs.gov/GetJob/ViewDetails/363324500 https://www.usajobs.gov/GetJob/ViewDetails/363324500

eli 13y ago
It's tough to just walk into a GS-14 job if you don't already work for the feds.
yid 13y ago
Something makes me feel like that might be an uphill battle. The focus of the entire site appears to be a slick superficial UI. There is no attempt to consolidate data from various agencies into a uniform access format (say a JSON API or periodic dumps to a user-pays S3 bucket). The data is in XML or CSV or even XLS. The whole implementation seems a bit misguided, to be honest.
- eplanit 13y ago
  I agree. I wonder if CGI (who also developed healthcare.gov) was hired to implement it.
Glyptodon 13y ago
Just reading about the requirements and hoops for applying makes me not want to apply.

stevejohnson 13y ago

This headline is very editorialized. Submitter should write a short blog post and submit that.

yid 13y ago
Sorry, I thought this was worth sharing and didn't have the inclination to write a blog post.
- alialkhatib 13y ago
  Those clauses seem contradictory. You have a very clear (and in my opinion on point) critique of what you're presenting us, but you don't have the inclination to write a blog post? You even wrote your own top-level comment providing some context. That would have made a sufficient blog post right there. (A great blog post might have compared bad vs good cases, or provided some suggestions on improving, or even looked more broadly at whether this was an outlier of badly structured/provided data or just the norm) The whole "Try (downloading|accessing) this..." strikes me as the same kind of title as Upworthy's "You'll never believe this..." in that it seems vaguely like link-baiting. I'm a little too curious not to click just to see what the dataset is about.
  - yid 13y ago
    > You have a very clear (and in my opinion on point) critique of what you're presenting us, but you don't have the inclination to write a blog post? Correct. I don't have a blog, and don't care to set one up. And I'd imagine a .gov link would have a lower chance of being linkbait.
    - alialkhatib 13y ago
      There are mediums (pun not intended) where you can make the equivalent of a blog post without much effort (Medium, Quora, I'm sure there are others). And link-bait is more about the marketing used to get people to click on the link (e.g. editorializing), isn't it? I don't want to get into semantics, so let's consider this a minor detail, because the more important matter is that you're editorializing. HN has a pretty clear (and helpful) set of guidelines on posting: http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html >If you want to add initial commentary on the link, write a blog post about it and submit that instead. I'm not trying to harangue you or anything; you make a good point but it's hamstrung by a length-limited title section.

bubbafat 13y ago

The data set disappearing is explained a bit more here: http://www.osti.gov/home/RETIRED/greenenergy?format=csv http://www.osti.gov/home/RETIRED/greenenergy?format=csv Which leads to here: http://www.osti.gov/home/ostiblog/osti-re-focusing-and-re-balancing-its-operations-%E2%80%93-and-refreshing-its-home-page-%E2%80%93-advance-p http://www.osti.gov/home/ostiblog/osti-re-focusing-and-re-ba... Which explains the data is here: http://www.osti.gov/scitech/ http://www.osti.gov/scitech/ Which I appreciate is not what you wanted - but it's what you got.

pbbakkum 13y ago

This is a problem close to my heart, thanks for pointing it out. I have a project up at http://commonwealth.io http://commonwealth.io attempting the reduce the barrier to using data like this. Even when the data is accessible, it could be in any format and you have to load it into a DB to get any value. The idea is to enable real SQL queries directly on data sets, so you don't need to worry about accessibility or how you'll query it. Not much data in it now, and it doesn't solve your immediate problem, but perhaps a better model for storing and representing this data.

ChuckMcM 13y ago

Interesting, I left a note saying the dataset was unavailable. It looks like it was created 22-Apr-2010 so it is probably fairly stale at this point.

bartleeanderson 13y ago

I found this link after playing around at their site Then I just entered green energy as a search term Hope this helps http://www.osti.gov/doepatents/index.jsp http://www.osti.gov/doepatents/index.jsp

ewams 13y ago

This is a great example of a use case for using Bittorrent Sync. They just need to create a read-only secret for each dataset they want to share and then put it on the website. Then the data is P2P'd for cheaper and much easier to maintain. Plus if the Government shuts down or has an issue the data can still be transferred.

dlapiduz 13y ago

The project is open source, you can submit an issue (or a PR) here: https://github.com/gsa/data.gov https://github.com/gsa/data.gov

philipashlock 13y ago

Thanks for pointing this out. We've updated the highlighted dataset to something that is better maintained, but if you find other problems please do ticket them on github (https://github.com/GSA/data.gov/issues https://github.com/GSA/data.gov/issues) These kinds of issues will improve as a new approach to listing datasets is implemented. This is currently underway and is described on Project Open Data - http://project-open-data.github.io/implementation-guide/ http://project-open-data.github.io/implementation-guide/ We're also working on some basic automated tests to ensure listings have the correct MIME type (which is incorrect far too often) and that URLs aren't 404s or redirects or HTML pages rather than raw data. For those interested, please do consider applying for the open position - https://www.usajobs.gov/GetJob/ViewDetails/363324500 https://www.usajobs.gov/GetJob/ViewDetails/363324500 I'll do my best to answer any questions about the position and the process of applying.

murphysbooks 13y ago

Each gov agency is now posting their dataset metadata on their own sites at agency.gov/data.json which should contain the url for the dataset. See http://project-open-data.github.io/schema/ http://project-open-data.github.io/schema/ for details