3 ms·

1. open financial tick data -- opentick used to have some, but that's gone as far as i can tell. also, one of the biggest problems with financial data is clean

1. open financial tick data -- opentick used to have some, but that's gone as far as i can tell. also, one of the biggest problems with financial data is cleaning it. with an open dataset, people might be encouraged to clean the open dataset like people contribute patches to open source code. 2. open restaurant data -- comprehensive listings of restaurants with information like hours, address, possibly menus.

transatlantic 16y ago
I've spent a non-trivial amount of effort trying to figure out how to do number two at scale, with a high level of accuracy, in a manner that could eventually be profitable. So far I'm not close.
- arthurdent 16y ago
  I would be interested in finding out more about your approach to this or chatting about ways to make this happen. I found this old thread: http://news.ycombinator.com/item?id=911853 http://news.ycombinator.com/item?id=911853 but couldn't find your contact information.
  - buro9 16y ago
    Over at Yell we have such data and I'm trying to persuade them to open it up through a public facing API. Restaurants, locations, contacts details, opening times, photos of the facade and sometimes of the inside, some menus. We also have a lot of other data for things like bars (what beers they have, do they have a garden, how many TV screens) and other business types. The obstacles we've encountered to opening the data are: 1) Fear. Parts of Yell fear that the data they spend time and money creating and collating will be scraped and the crown jewels given away for free. 2) Politics. Different parts of Yell view the opening of data differently and will fight over any move even if they are in general agreement... nothing gets done. I am trying to argue that on #1 that if they really need to fear something it should be not moving with the times. And also through the use of things like API keys that we could rate limit to some extent to prevent major scraping occurring if they really think it's a big problem. And on #2 it's currently part of their business to argue a lot it seems. I believe that if they could see the value in opening data that both would be a non-issue and it would just be done. So I'm also working on trying to document examples of how people could use such an API that is beneficial to Yell. And on that note... if anyone has such examples I'd be glad to hear them.
    - steveitis 16y ago
      It's pretty simple. I've never heard of Yell before now. If they'd opened up this data, than I would have. Especially if they only allowed usage under an attribution required license.
aw3c2 16y ago
Open restaurant data is slowly added into OpenStreetMap. I too think it is a great resource (or will be), same for shops, doctors etc.
revorad 16y ago
1. The R Empirical Finance Taks View lists a number of packages with data - http://cran.r-project.org/web/views/Finance.html http://cran.r-project.org/web/views/Finance.html. Especially check out quantmod- http://www.quantmod.com http://www.quantmod.com.
ig1 16y ago
Cleaning financial tick data can be very tricky, because it depends very much on how you want to use the data for. If you're not careful any models you build on the cleaned data can fall apart in real world usage because they fail to take into account the abnormalities in market data.
retube 16y ago
Your number 2 is interesting. My company actually basically does this. We crawl the UK web for business sites, incl. retail and hospitality/leisure and suck out contact details, addresses, products, services, food & drink items on menus, opening times as well as a few other things. We're looking at making available an API.

transatlantic 16y ago

I've spent a non-trivial amount of effort trying to figure out how to do number two at scale, with a high level of accuracy, in a manner that could eventually be profitable. So far I'm not close.

arthurdent 16y ago
I would be interested in finding out more about your approach to this or chatting about ways to make this happen. I found this old thread: http://news.ycombinator.com/item?id=911853 http://news.ycombinator.com/item?id=911853 but couldn't find your contact information.
- buro9 16y ago
  Over at Yell we have such data and I'm trying to persuade them to open it up through a public facing API. Restaurants, locations, contacts details, opening times, photos of the facade and sometimes of the inside, some menus. We also have a lot of other data for things like bars (what beers they have, do they have a garden, how many TV screens) and other business types. The obstacles we've encountered to opening the data are: 1) Fear. Parts of Yell fear that the data they spend time and money creating and collating will be scraped and the crown jewels given away for free. 2) Politics. Different parts of Yell view the opening of data differently and will fight over any move even if they are in general agreement... nothing gets done. I am trying to argue that on #1 that if they really need to fear something it should be not moving with the times. And also through the use of things like API keys that we could rate limit to some extent to prevent major scraping occurring if they really think it's a big problem. And on #2 it's currently part of their business to argue a lot it seems. I believe that if they could see the value in opening data that both would be a non-issue and it would just be done. So I'm also working on trying to document examples of how people could use such an API that is beneficial to Yell. And on that note... if anyone has such examples I'd be glad to hear them.
  - steveitis 16y ago
    It's pretty simple. I've never heard of Yell before now. If they'd opened up this data, than I would have. Especially if they only allowed usage under an attribution required license.

aw3c2 16y ago

Open restaurant data is slowly added into OpenStreetMap. I too think it is a great resource (or will be), same for shops, doctors etc.

revorad 16y ago

1. The R Empirical Finance Taks View lists a number of packages with data - http://cran.r-project.org/web/views/Finance.html http://cran.r-project.org/web/views/Finance.html. Especially check out quantmod- http://www.quantmod.com http://www.quantmod.com.

ig1 16y ago

Cleaning financial tick data can be very tricky, because it depends very much on how you want to use the data for. If you're not careful any models you build on the cleaned data can fall apart in real world usage because they fail to take into account the abnormalities in market data.

retube 16y ago

Your number 2 is interesting. My company actually basically does this. We crawl the UK web for business sites, incl. retail and hospitality/leisure and suck out contact details, addresses, products, services, food & drink items on menus, opening times as well as a few other things. We're looking at making available an API.