3 ms·

Yes, you're right. We think it's important that this data is openly available

by pranade 13y ago

twerquie 13y ago
You're probably about to discover that the data is also not free.
- jydarche 13y ago
  The facts are not copyrightable. http://www.lib.umich.edu/copyright/facts-and-data http://www.lib.umich.edu/copyright/facts-and-data But there's probably a violation of the terms of use, so maybe it's not a good idea to say to everybody that you take the data on the official website.
  - pranade 13y ago
    Yeah, it seems the facts are the most important part. I mean the data should be openly available, right? and that lowering the barrier to use by programmers and data scientists should improve the quality of how people interact with and experience the olympics. We're hoping it will help devs build super interesting visualizations and/or apps (that otherwise weren't possible).
    - loceng 13y ago
      Is the fact that facts aren't copyrightable the basis for how you're planning to allow APIs to be created from most sites?
  - nandhp 13y ago
    The sochi2014.ru terms of use does not prohibit the use of robots/crawlers/scrapers. http://www.sochi2014.com/en/terms-of-use http://www.sochi2014.com/en/terms-of-use
    - jydarche 13y ago
      But, they prohibit the copy of the content from the website, no matter the way you did it. Point 5.2: "Except as expressly stated on the Website, you are not entitled to copy, publish, download, modify and redistribute the information provided on the Website without the prior written consent of the Sochi 2014 Organizing Committee."
      - monkeyspaw 13y ago
        IANAL, but that sounds either like you can't do all of those things together (the "and" conjunction), or you can't even read the website. After all, if you can't download it, you can't view it.
untog 13y ago
Well, good luck to you. I think you'll find out that the IOC disagree with you, and that the Olympics is far more of a money-grab than an international coming together of the world.
mbesto 13y ago
Serious question - have you look very deep into the legality of what you're doing? I have no doubt in the world there is huge value in your service but generally speaking your website violates the most basic terms of use of pretty much any site that exists. That being said - you (and pretty much every other web scraper in the world) enters into this grey area of "binding contract" in the terms of use. Personally I'd be very scared about your service being able to scale for two fundamental reasons (1) the more attention you attract as you scale, the more legislation will be targeted at you and (2) people who want to protect their data will create services to block your efforts. note - IANAL and I say this not to scare you, but hopefully to help.
- meritt 13y ago
  There are numerous web-scraping tools out there, many are easy enough to use for non-programmers (Connotate and Mozenda for instance) so, while you have a point, precedent shows that web scraping is prominent and not going away anytime soon. I applaud innovation in this area. Your #2 is simply not plausible unless Kimono carefully follows robots.txt and consistent user-agent, which is rarely the case for a competent web-scraping platform. Legality is also a strong word. Web scraping is rarely illegal in the sense that you're committing a crime. A company could definitely issue a C&D and potentially go after you in a civil lawsuit but it's very infrequent (sadly not so much anymore with all the bullshit CFAA suits against "hackers" lately) that a government entity would pursue someone for scraping data. The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping.
  - mbesto 13y ago
    Precedent is also a strong word. There are numerous people on the road who go over the speed limit. It's prominent and not going away anytime soon. This is not a cause for precedence. Precedence implies that a legal case take place in order to dictate the law. AFAIK, between your unsubstantiated comment and my general knowledge of this area, the reason no one has provided a scraper service at scale isn't because the technology or execution wasn't right, but rather that it's simply illegal to do so. > The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping. That's simply not true. Here's an example: http://www.yelp.com/static?country_=US&p=tos http://www.yelp.com/static?country_=US&p=tos "By accessing or using the Site, you are agreeing to these Terms and concluding a legally binding contract with Yelp Inc., a Delaware corporation headquartered in San Francisco, California ("Yelp"). Do not access or use the Site if you are unwilling or unable to be bound by the Terms." "Use any robot, spider, site search/retrieval application, or other automated device, process or means to access, retrieve, scrape, or index any portion of the Site or any Site Content;" My overall point? Kimonolabs is slowly painting a target on their back and it's only a matter of time before they'll have to "lawyer up".

Yes, you're right. We think it's important that this data is openly available

by pranade 13y ago

Yes, you're right. We think it's important that this data is openly available

twerquie 13y ago

You're probably about to discover that the data is also not free.

jydarche 13y ago
The facts are not copyrightable. http://www.lib.umich.edu/copyright/facts-and-data http://www.lib.umich.edu/copyright/facts-and-data But there's probably a violation of the terms of use, so maybe it's not a good idea to say to everybody that you take the data on the official website.
- pranade 13y ago
  Yeah, it seems the facts are the most important part. I mean the data should be openly available, right? and that lowering the barrier to use by programmers and data scientists should improve the quality of how people interact with and experience the olympics. We're hoping it will help devs build super interesting visualizations and/or apps (that otherwise weren't possible).
  - loceng 13y ago
    Is the fact that facts aren't copyrightable the basis for how you're planning to allow APIs to be created from most sites?
- nandhp 13y ago
  The sochi2014.ru terms of use does not prohibit the use of robots/crawlers/scrapers. http://www.sochi2014.com/en/terms-of-use http://www.sochi2014.com/en/terms-of-use
  - jydarche 13y ago
    But, they prohibit the copy of the content from the website, no matter the way you did it. Point 5.2: "Except as expressly stated on the Website, you are not entitled to copy, publish, download, modify and redistribute the information provided on the Website without the prior written consent of the Sochi 2014 Organizing Committee."
    - monkeyspaw 13y ago
      IANAL, but that sounds either like you can't do all of those things together (the "and" conjunction), or you can't even read the website. After all, if you can't download it, you can't view it.

untog 13y ago

Well, good luck to you. I think you'll find out that the IOC disagree with you, and that the Olympics is far more of a money-grab than an international coming together of the world.

mbesto 13y ago

Serious question - have you look very deep into the legality of what you're doing? I have no doubt in the world there is huge value in your service but generally speaking your website violates the most basic terms of use of pretty much any site that exists. That being said - you (and pretty much every other web scraper in the world) enters into this grey area of "binding contract" in the terms of use. Personally I'd be very scared about your service being able to scale for two fundamental reasons (1) the more attention you attract as you scale, the more legislation will be targeted at you and (2) people who want to protect their data will create services to block your efforts. note - IANAL and I say this not to scare you, but hopefully to help.

meritt 13y ago
There are numerous web-scraping tools out there, many are easy enough to use for non-programmers (Connotate and Mozenda for instance) so, while you have a point, precedent shows that web scraping is prominent and not going away anytime soon. I applaud innovation in this area. Your #2 is simply not plausible unless Kimono carefully follows robots.txt and consistent user-agent, which is rarely the case for a competent web-scraping platform. Legality is also a strong word. Web scraping is rarely illegal in the sense that you're committing a crime. A company could definitely issue a C&D and potentially go after you in a civil lawsuit but it's very infrequent (sadly not so much anymore with all the bullshit CFAA suits against "hackers" lately) that a government entity would pursue someone for scraping data. The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping.
- mbesto 13y ago
  Precedent is also a strong word. There are numerous people on the road who go over the speed limit. It's prominent and not going away anytime soon. This is not a cause for precedence. Precedence implies that a legal case take place in order to dictate the law. AFAIK, between your unsubstantiated comment and my general knowledge of this area, the reason no one has provided a scraper service at scale isn't because the technology or execution wasn't right, but rather that it's simply illegal to do so. > The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping. That's simply not true. Here's an example: http://www.yelp.com/static?country_=US&p=tos http://www.yelp.com/static?country_=US&p=tos "By accessing or using the Site, you are agreeing to these Terms and concluding a legally binding contract with Yelp Inc., a Delaware corporation headquartered in San Francisco, California ("Yelp"). Do not access or use the Site if you are unwilling or unable to be bound by the Terms." "Use any robot, spider, site search/retrieval application, or other automated device, process or means to access, retrieve, scrape, or index any portion of the Site or any Site Content;" My overall point? Kimonolabs is slowly painting a target on their back and it's only a matter of time before they'll have to "lawyer up".