4 ms·

Show HN: Pandas, Numpy, and SciPy users can now access our over 4M datasets

93 points by seanrwcrawford 14y ago

gustavopr 14y ago
Wow, thank you very much for this! One criticism, though: there is no clear mention of the service being free (as I think it is, from the login process). Maybe it might be interesting to put this info in the FAQ, and the API limitations for non authenticated / authenticated calls? Just one more thing: the data from the Brazilian Central Bank is not showing up. When you click the raw data link, it goes to a page that displays an error about an expired session.
- seanrwcrawford 14y ago
  Really great points, thanks gustavopr. I'll look into BCB data right now.
blaabjerg 14y ago
This looks like an _awesome_ service! If you don't mind me asking, do you have any grand plans for it business-wise? Will it become a paid service in the future? edit: ah, I see it already is for higher volumes :) That's fair! > This token will allow you to make 100 calls per day. If you wish more API calls please contact us, and we will be more than happy to help you.
- seanrwcrawford 14y ago
  Hey blaabjerg, thanks for the kind words. Our plan for the service is to be able to provide 3rd party premium data for sale at some point; but all data that is currently on the site, and all publicly available data will always remain free and open. RE: the API calls just shoot us a message and we can easily adjust the limit to what you need.
  - blaabjerg 14y ago
    Very cool. Thanks!
steven2012 14y ago
This looks awesome! Where does the data come from? Is this a screenscraped source, or was the data purchased from a reliable data source? If it was screenscraped, are there possibly licensing issues with the NYSE, CME, etc?
- tammerkamel88 14y ago
  Actually, every data page has "look-thru" to the source of the data. bottom of the page, under "Dataset Information", "Raw Data"
dave_sullivan 14y ago
I've been using this service for a while now (it was wikiposit before they came up with a snazzy interface, better API, and new name). It's really useful and has a lot of datasets that are very hard to find for free elsewhere. As far as I can tell, they are mostly scraped (and I've noticed some breakage in the past when sites they're scraping change, which is annoying, but hey, can't argue with free) Nice to see them keep expanding the offering.
isharabash 14y ago
A few thoughts for the online interface: 1) When dragging the time selection slider (below the stage chart), it doesn't move if your cursor moves out of the vertical bounds. 2) It wasn't apparent to me at first that you could toggle variables on the legend -- in fact I didn't even notice the legend right away (maybe make it a bit bigger, and a bit more obvious that it's toggle-able?) 3) The legend changes between vertical stack and inline based on how wide the window is, I'd prefer if it was vertical stacked all the time. I dig it!
- seanrwcrawford 14y ago
  These are great points, thank you. We've actually been working on a very big redesign that will cover off some of what you're asking about; it's a couple months away, but it is on the horizon.
xradionut 14y ago
Interesting site. I'm using pandas, Python and a few other tools to do analysis of public domain data for training and analysis, mainly that from the FDIC, NCUA and FCC. Understanding the "domain" and doing the ETL is 90% of the work, so having "clean" data for others is a nice service.
cachemeifyoucan 14y ago
Good stuff! Started playing with it and I love the interface. Can't wait to see what kind of studies will emerge thanks to this tool.
cliftonk 14y ago
Just stumbled on this a few days ago. Awesome work!

Wow, thank you very much for this! One criticism, though: there is no clear mention of the service being free (as I think it is, from the login process). Maybe it might be interesting to put this info in the FAQ, and the API limitations for non authenticated / authenticated calls? Just one more thing: the data from the Brazilian Central Bank is not showing up. When you click the raw data link, it goes to a page that displays an error about an expired session.

seanrwcrawford 14y ago
Really great points, thanks gustavopr. I'll look into BCB data right now.

blaabjerg 14y ago

This looks like an _awesome_ service! If you don't mind me asking, do you have any grand plans for it business-wise? Will it become a paid service in the future? edit: ah, I see it already is for higher volumes :) That's fair! > This token will allow you to make 100 calls per day. If you wish more API calls please contact us, and we will be more than happy to help you.

seanrwcrawford 14y ago
Hey blaabjerg, thanks for the kind words. Our plan for the service is to be able to provide 3rd party premium data for sale at some point; but all data that is currently on the site, and all publicly available data will always remain free and open. RE: the API calls just shoot us a message and we can easily adjust the limit to what you need.
- blaabjerg 14y ago
  Very cool. Thanks!

steven2012 14y ago

This looks awesome! Where does the data come from? Is this a screenscraped source, or was the data purchased from a reliable data source? If it was screenscraped, are there possibly licensing issues with the NYSE, CME, etc?

tammerkamel88 14y ago
Actually, every data page has "look-thru" to the source of the data. bottom of the page, under "Dataset Information", "Raw Data"

dave_sullivan 14y ago

I've been using this service for a while now (it was wikiposit before they came up with a snazzy interface, better API, and new name). It's really useful and has a lot of datasets that are very hard to find for free elsewhere. As far as I can tell, they are mostly scraped (and I've noticed some breakage in the past when sites they're scraping change, which is annoying, but hey, can't argue with free) Nice to see them keep expanding the offering.

isharabash 14y ago

A few thoughts for the online interface: 1) When dragging the time selection slider (below the stage chart), it doesn't move if your cursor moves out of the vertical bounds. 2) It wasn't apparent to me at first that you could toggle variables on the legend -- in fact I didn't even notice the legend right away (maybe make it a bit bigger, and a bit more obvious that it's toggle-able?) 3) The legend changes between vertical stack and inline based on how wide the window is, I'd prefer if it was vertical stacked all the time. I dig it!

seanrwcrawford 14y ago
These are great points, thank you. We've actually been working on a very big redesign that will cover off some of what you're asking about; it's a couple months away, but it is on the horizon.

xradionut 14y ago

Interesting site. I'm using pandas, Python and a few other tools to do analysis of public domain data for training and analysis, mainly that from the FDIC, NCUA and FCC. Understanding the "domain" and doing the ETL is 90% of the work, so having "clean" data for others is a nice service.

cachemeifyoucan 14y ago

Good stuff! Started playing with it and I love the interface. Can't wait to see what kind of studies will emerge thanks to this tool.

cliftonk 14y ago

Just stumbled on this a few days ago. Awesome work!