7 ms·

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only sho

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

pranade 13y ago
Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.
- chii 13y ago
  how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data? of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.
  - the_french 13y ago
    Would there any way to fake the beginning of an OAuth session with Facebook, Google or any other OAuth authenticated site? Kind of like replaying cookies to hijack sessions?
    - garyjob 13y ago
      The route of proxying the web page presents much difficulty in doing actual authentication on Facebook or Google's website via the proxied webpage without first rewriting most of the javascript and hijacking their Ajax calls on the fly. The approach I took was to hijack the Cookies from the browser once the user has signed in after on e.g. Facebook via the browser extension. The route of proxying the website does in fact do away with the need to install any external 3rd libraries. This browser extension I built coupled with the web service its integrated to does allow for scraping of pages from Facebook, Google and LinkedIn logged in pages as well. https://chrome.google.com/webstore/detail/krakeio/ofncgcgajhgnbkbmkdhbgkoopfbemhfj https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...
      - davedx 13y ago
        Hah, I've been working on this recently with Facebook, on a TV set-top-box. It was painful and I ended up giving up. xd_arbiter.php is the key, I think.
- easy_rider 13y ago
  I'm wondering how you will be able to with the numerous ways of CSRF protection implementations.
  - deleted 13y ago
    [deleted]
- billmalarky 13y ago
  I hope that the lite plan will feature auth handling, I can't imagine the service being useful in most cases without it.
  - pranade 13y ago
    We're working on auth... it's the most requested feature at the moment. And we're still beta at the moent, so all usage is free
    - billmalarky 13y ago
      Glad to hear it. I was just saying that I think auth should be included as a basic feature in all paid for plans.
automately 13y ago
Creator of Automately here, our service could definitely be something you might be interested in. While we aren't directly in the business of web scraping, we do have a powerful automation service that can accomplish those needs using simple javascript and our powerful scalable automation API. We are accepting early access requests right now. Check us out! http://automate.ly/ http://automate.ly/

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

pranade 13y ago

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

chii 13y ago
how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data? of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.
- the_french 13y ago
  Would there any way to fake the beginning of an OAuth session with Facebook, Google or any other OAuth authenticated site? Kind of like replaying cookies to hijack sessions?
  - garyjob 13y ago
    The route of proxying the web page presents much difficulty in doing actual authentication on Facebook or Google's website via the proxied webpage without first rewriting most of the javascript and hijacking their Ajax calls on the fly. The approach I took was to hijack the Cookies from the browser once the user has signed in after on e.g. Facebook via the browser extension. The route of proxying the website does in fact do away with the need to install any external 3rd libraries. This browser extension I built coupled with the web service its integrated to does allow for scraping of pages from Facebook, Google and LinkedIn logged in pages as well. https://chrome.google.com/webstore/detail/krakeio/ofncgcgajhgnbkbmkdhbgkoopfbemhfj https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...
    - davedx 13y ago
      Hah, I've been working on this recently with Facebook, on a TV set-top-box. It was painful and I ended up giving up. xd_arbiter.php is the key, I think.
easy_rider 13y ago
I'm wondering how you will be able to with the numerous ways of CSRF protection implementations.
- deleted 13y ago
  [deleted]
billmalarky 13y ago
I hope that the lite plan will feature auth handling, I can't imagine the service being useful in most cases without it.
- pranade 13y ago
  We're working on auth... it's the most requested feature at the moment. And we're still beta at the moent, so all usage is free
  - billmalarky 13y ago
    Glad to hear it. I was just saying that I think auth should be included as a basic feature in all paid for plans.

automately 13y ago

Creator of Automately here, our service could definitely be something you might be interested in. While we aren't directly in the business of web scraping, we do have a powerful automation service that can accomplish those needs using simple javascript and our powerful scalable automation API. We are accepting early access requests right now. Check us out! http://automate.ly/ http://automate.ly/