4 ms·

It's similar in the sense that it's a listing of APIs. But we're providing non-official APIs that work by automating / scraping websites. Our platform runs Head

by paps 8y ago

It's similar in the sense that it's a listing of APIs. But we're providing non-official APIs that work by automating / scraping websites. Our platform runs Headless Chrome instances behind the scenes. As a matter of fact, the automation code is available here: https://github.com/phantombuster/api-store https://github.com/phantombuster/api-store

chrismeller 8y ago
So eventually you’ll have reliability issues because your data source is a violation of someone’s TOS?
- paps 8y ago
  Well, it's up for debate. We automate websites on behalf of our users (that is, logged in as them). Which means the site knows at all time who's doing what and can take action in case of abuse. Also, we see more and more ruling indicating that scraping is in fact legal. Websites can block users according to their ToS but they can't take legal action against them or us. Maybe. In any case, our platform also provides the tools for anyone to automate any website (make them into an API). That part is just a developer tool.
  - madamelic 8y ago
    >We automate websites on behalf of our users (that is, logged in as them). Which means the site knows at all time who's doing what and can take action in case of abuse. So you break ToS on your user's account, thereby risking their own and not yours... Even better. >Also, we see more and more ruling indicating that scraping is in fact legal. It doesn't matter if it is legal. What matters more is if the service considers it a violation of an implicit agreement not to abuse servers with rapid API requests (Big props if you are already throttling) --- Like your service is a great idea, but breaking ToS on your user's accounts is super no-bueno in my opinion. I scrape too but I am always under the complete understanding the service can ban my account or IP at any time.
    - dosycorp 8y ago
      How is it breaking TOS? If you are projeting the users intent its just like they are accessing the site through a prosthesis. Its doing things they could do by hand, if they spent the time. Even a browser "automates" a http request instead of you having to type it by telnet.
      - drusepth 8y ago
        >How is it breaking TOS? Obviously, it's up to each site's specific TOS. Tons of sites explicitly call out scrapers and non-human/automated means of accessing the site. You might debate over definitions and intent, but ultimately it's up to the site owners when they say, "you know what? X _is_ against the ToS and we're just gonna ban anyone doing it"; users won't/don't have any recourse to argue their point. For example, here's a few relevant parts for the top sites on Phantom Buster: Facebook: https://www.facebook.com/apps/site_scraping_tos_terms.php https://www.facebook.com/apps/site_scraping_tos_terms.php >These terms govern your collection of data from Facebook through automated means, such as through harvesting bots, robots, spiders, or scrapers ("Automated Data Collection"), as well as your use of that data. You will not engage in Automated Data Collection without Facebook's express written permission. Instagram: https://help.instagram.com/478745558852511 https://help.instagram.com/478745558852511 >We prohibit crawling, scraping, caching or otherwise accessing any content on the Service via automated means, including but not limited to, user profiles and photos (except as may be the result of standard search engine protocols or technologies used by a search engine with Instagram's express consent). Linkedin: https://www.linkedin.com/help/linkedin/answer/56347/prohibited-software-and-extensions?lang=en https://www.linkedin.com/help/linkedin/answer/56347/prohibit... > In order to protect our members’ data and our website, we don't permit the use of any third party software, including "crawlers", bots, browser plug-ins, or browser extensions (also called "add-ons"), that scrapes, modifies the appearance of, or automates activity on LinkedIn’s website. I'd give a read through each of the APIs offered and make sure that users know 1) your service has the potential to get accounts banned for use, and 2) since the service is on behalf of the user's accounts, it'll be their accounts getting banned if the websites ban anyone. FWIW I wrote this comment while watching a bot (that I wrote) play a game on my behalf on a second monitor. :)
  - naiv 8y ago
    Wow, you are located in San Francisco. I would have guessed in India or somewhere else where you might not fear being sued. >> "is in fact legal" There is a big difference between "legal" and "court decision" . If a court will rule in hiq labs favor vs. linkedin it doesn't make scraping linkedin automagically legal for you.
  - nunez 8y ago
    Yikes.
deleted 8y ago
[deleted]
amelius 8y ago
Wouldn't it be easy for e.g. Facebook to train a classifier on the behavior of your scrapers, and from there block you?

by paps 8y ago

chrismeller 8y ago

So eventually you’ll have reliability issues because your data source is a violation of someone’s TOS?

paps 8y ago
Well, it's up for debate. We automate websites on behalf of our users (that is, logged in as them). Which means the site knows at all time who's doing what and can take action in case of abuse. Also, we see more and more ruling indicating that scraping is in fact legal. Websites can block users according to their ToS but they can't take legal action against them or us. Maybe. In any case, our platform also provides the tools for anyone to automate any website (make them into an API). That part is just a developer tool.
- madamelic 8y ago
  >We automate websites on behalf of our users (that is, logged in as them). Which means the site knows at all time who's doing what and can take action in case of abuse. So you break ToS on your user's account, thereby risking their own and not yours... Even better. >Also, we see more and more ruling indicating that scraping is in fact legal. It doesn't matter if it is legal. What matters more is if the service considers it a violation of an implicit agreement not to abuse servers with rapid API requests (Big props if you are already throttling) --- Like your service is a great idea, but breaking ToS on your user's accounts is super no-bueno in my opinion. I scrape too but I am always under the complete understanding the service can ban my account or IP at any time.
  - dosycorp 8y ago
    How is it breaking TOS? If you are projeting the users intent its just like they are accessing the site through a prosthesis. Its doing things they could do by hand, if they spent the time. Even a browser "automates" a http request instead of you having to type it by telnet.
    - drusepth 8y ago
      >How is it breaking TOS? Obviously, it's up to each site's specific TOS. Tons of sites explicitly call out scrapers and non-human/automated means of accessing the site. You might debate over definitions and intent, but ultimately it's up to the site owners when they say, "you know what? X _is_ against the ToS and we're just gonna ban anyone doing it"; users won't/don't have any recourse to argue their point. For example, here's a few relevant parts for the top sites on Phantom Buster: Facebook: https://www.facebook.com/apps/site_scraping_tos_terms.php https://www.facebook.com/apps/site_scraping_tos_terms.php >These terms govern your collection of data from Facebook through automated means, such as through harvesting bots, robots, spiders, or scrapers ("Automated Data Collection"), as well as your use of that data. You will not engage in Automated Data Collection without Facebook's express written permission. Instagram: https://help.instagram.com/478745558852511 https://help.instagram.com/478745558852511 >We prohibit crawling, scraping, caching or otherwise accessing any content on the Service via automated means, including but not limited to, user profiles and photos (except as may be the result of standard search engine protocols or technologies used by a search engine with Instagram's express consent). Linkedin: https://www.linkedin.com/help/linkedin/answer/56347/prohibited-software-and-extensions?lang=en https://www.linkedin.com/help/linkedin/answer/56347/prohibit... > In order to protect our members’ data and our website, we don't permit the use of any third party software, including "crawlers", bots, browser plug-ins, or browser extensions (also called "add-ons"), that scrapes, modifies the appearance of, or automates activity on LinkedIn’s website. I'd give a read through each of the APIs offered and make sure that users know 1) your service has the potential to get accounts banned for use, and 2) since the service is on behalf of the user's accounts, it'll be their accounts getting banned if the websites ban anyone. FWIW I wrote this comment while watching a bot (that I wrote) play a game on my behalf on a second monitor. :)
- naiv 8y ago
  Wow, you are located in San Francisco. I would have guessed in India or somewhere else where you might not fear being sued. >> "is in fact legal" There is a big difference between "legal" and "court decision" . If a court will rule in hiq labs favor vs. linkedin it doesn't make scraping linkedin automagically legal for you.
- nunez 8y ago
  Yikes.

deleted 8y ago

[deleted]

amelius 8y ago

Wouldn't it be easy for e.g. Facebook to train a classifier on the behavior of your scrapers, and from there block you?