4 ms·

Building an API on open data

65 points by freyfogle 9y ago

AznHisoka 9y ago
Geocoding might be a corner case in terms of the usefulness of open data. The most valuable data is usually owned by huge corporations with lots of users (ie Amazon, Yelp, Visa, Google). Data usually is valuable because it's compiled from the usage of many people. So the only way to really build an API from that is to scrape them.
- mtmail 9y ago
  I'd add government data, their open data portals are often subpar. For example in Germany the national weather data was made (licensed) open data this week (in German https://www.golem.de/news/deutscher-wetterdienst-wetterdaten-sind-jetzt-open-data-1708-129258.html https://www.golem.de/news/deutscher-wetterdienst-wetterdaten...)
  - thesehands 9y ago
    It's one thing that the uk seems to do pretty well at https://data.gov.uk https://data.gov.uk lots of data available. Not all in uniform format but it is at least available
    - wongarsu 9y ago
      Eurostat (the EUs statistics institiution) also has a lot of statistics freely available http://ec.europa.eu/eurostat/data/statistics-a-z/abc http://ec.europa.eu/eurostat/data/statistics-a-z/abc with permissive license, data download in CSV and XML as well as a REST service. Their online data viewer isn't pretty, but one of the more powerful examples as well.
    - tenkabuto 9y ago
      https://www.data.gov https://www.data.gov is pretty great for the U.S. Government’s open data
    - gregmac 9y ago
      Canada has a bunch available as well: http://open.canada.ca/en/open-data http://open.canada.ca/en/open-data
  - sudshekhar 9y ago
    What are some good websites/apis built using government data? AFAIK, most countries have started publishing open datasets about things such as weather, transportation, politician backgrounds, even law! But I've yet to see some useful websites/apis built on top of them. Most are related to politics, visualization and/or research (which is a laudable enough but doesn't make them worth checking regularly).
- freyfogle 9y ago
  Or you could get many people and organizations to contribute data under an open license. That's exactly what OpenStreetMap is: http://www.openstreetmap.org http://www.openstreetmap.org
  - mtmail 9y ago
    https://opencorporates.com/info/working_for_the_public_good https://opencorporates.com/info/working_for_the_public_good in the UK might be another example. They don't seem to run to make profit and one can download their huge compiled data sets.
- garysieling 9y ago
  OpenLibrary is pretty useful too, in part because Amazon's product API has aggressive license restrictions
- yellowbkpk 9y ago
  Check out OpenAddresses: we're trying to solve the address/location data aggregation problem so you don't have to. We do all the data scraping and collating for you so you can work on making a geocoder. https://openaddresses.io/ https://openaddresses.io/
lsiebert 9y ago
It's always interesting to see a web service written in perl.

65 points by freyfogle 9y ago

Geocoding might be a corner case in terms of the usefulness of open data. The most valuable data is usually owned by huge corporations with lots of users (ie Amazon, Yelp, Visa, Google). Data usually is valuable because it's compiled from the usage of many people. So the only way to really build an API from that is to scrape them.

mtmail 9y ago
I'd add government data, their open data portals are often subpar. For example in Germany the national weather data was made (licensed) open data this week (in German https://www.golem.de/news/deutscher-wetterdienst-wetterdaten-sind-jetzt-open-data-1708-129258.html https://www.golem.de/news/deutscher-wetterdienst-wetterdaten...)
- thesehands 9y ago
  It's one thing that the uk seems to do pretty well at https://data.gov.uk https://data.gov.uk lots of data available. Not all in uniform format but it is at least available
  - wongarsu 9y ago
    Eurostat (the EUs statistics institiution) also has a lot of statistics freely available http://ec.europa.eu/eurostat/data/statistics-a-z/abc http://ec.europa.eu/eurostat/data/statistics-a-z/abc with permissive license, data download in CSV and XML as well as a REST service. Their online data viewer isn't pretty, but one of the more powerful examples as well.
  - tenkabuto 9y ago
    https://www.data.gov https://www.data.gov is pretty great for the U.S. Government’s open data
  - gregmac 9y ago
    Canada has a bunch available as well: http://open.canada.ca/en/open-data http://open.canada.ca/en/open-data
- sudshekhar 9y ago
  What are some good websites/apis built using government data? AFAIK, most countries have started publishing open datasets about things such as weather, transportation, politician backgrounds, even law! But I've yet to see some useful websites/apis built on top of them. Most are related to politics, visualization and/or research (which is a laudable enough but doesn't make them worth checking regularly).
freyfogle 9y ago
Or you could get many people and organizations to contribute data under an open license. That's exactly what OpenStreetMap is: http://www.openstreetmap.org http://www.openstreetmap.org
- mtmail 9y ago
  https://opencorporates.com/info/working_for_the_public_good https://opencorporates.com/info/working_for_the_public_good in the UK might be another example. They don't seem to run to make profit and one can download their huge compiled data sets.
garysieling 9y ago
OpenLibrary is pretty useful too, in part because Amazon's product API has aggressive license restrictions
yellowbkpk 9y ago
Check out OpenAddresses: we're trying to solve the address/location data aggregation problem so you don't have to. We do all the data scraping and collating for you so you can work on making a geocoder. https://openaddresses.io/ https://openaddresses.io/

lsiebert 9y ago

It's always interesting to see a web service written in perl.