4 ms·

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to wha

by jardah 9y ago

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

GSGSGS 9y ago
Are you Jaroslov ? :)
- jardah 9y ago
  Jaroslav, yes, I'm the author. Did you notice any problems or ways how I can improve it?
  - GSGSGS 9y ago
    Not that i can see from a surface view, i think documentation can be improved :). Personally like the idea of APIFY, saw it a few months ago. Are you guys hiring ? :D
    - jancurn 9y ago
      Yes we are! Please see https://www.apify.com/jobs https://www.apify.com/jobs
      - johnnyfived 9y ago
        It's great that you're communicating openly on HN. I just sent an application for the Junior Web Developer position. Looking forward to hearing back!
    - jardah 9y ago
      :D yep the documentation needs a lot of work. It started as a test of an idea, then slowly became a usable tool and the code was getting incrementaly more complex without me event noticing. I only added the readme on github yesterday and there are basicaly no tests... :(
rmateus 9y ago
Is it able to deal with digital certificates?
- jancurn 9y ago
  You can use https://www.htbridge.com/ssl/ https://www.htbridge.com/ssl/ for that
_Chief 9y ago
how about caching the default entry (static url instead) + attribs, to ease demoing. at the moment it's been analyzing for more than >5mins
- jardah 9y ago
  Good idea and I would implement that if I used an API from server to get the response. But currently I'm at the same time testing stability of Apify "Actor" solution and proxies, so for my case it's good that there are real requests with real responses, even if it's just from demo. Btw the fact that it's running for 5 minutes is a bug, that I will look at, since there is a timeout of 2 minutes and there are no hanging runs or runs that ended with timeout.
  - ComputerGuru 9y ago
    You also don’t want to get your server blocked by yelp if they do rate limiting.
    - bpicolo 9y ago
      They have fairly aggressive scraper detection (and this is also against their ToS)
    - jardah 9y ago
      It's why I'm using proxies, every request is routed through different proxy address and the application as whole is rate limited. So hopefully I'm not making too much traffic on yelp. They are just a perfect example because they are using all types of data I'm looking for. When I find more good examples I will add them and rotate them for every page load. Btw when it comes to ToS and scraping, this is not much different from accessing their website through normal browser only instead of rendered content we should you analyzed data. The page is only loaded once same as in browser.
JustARandomGuy 9y ago
Suppose I wanted to extract an image that gets loaded async via Javascript (For example, a Pinterest page). How would that work? Looking at your documentation, it looks like I could parse the XHR array you supply. Could you suggest any other ways? I'm calling out Pinterest as an example here because they try to block their images from being easily downloaded, but if you have any other examples I'd like to hear them. It would be great if the page analyzer could supply a list of all the assets loaded with the web page; for example, any asset with a media type of image/* is listed in an images array, and so forth.
- jardah 9y ago
  Actually the list of assets shouldn't be that hard. Looking at pinterest the xhr requests for images are loaded immediately when page is open, so potentialy it then it's catched in onRequest function (only now I'm aborting the requests to save network trafic). I will try it our tomorrow and let you know in comment. Also, looking at pinterest, it's server rendered through ReactJS, so there is #initial-state script tag with first few images preloaded as urls, so if you cared only about the images on top without scrolling then this is the safest bet.

by jardah 9y ago

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

GSGSGS 9y ago

Are you Jaroslov ? :)

jardah 9y ago
Jaroslav, yes, I'm the author. Did you notice any problems or ways how I can improve it?
- GSGSGS 9y ago
  Not that i can see from a surface view, i think documentation can be improved :). Personally like the idea of APIFY, saw it a few months ago. Are you guys hiring ? :D
  - jancurn 9y ago
    Yes we are! Please see https://www.apify.com/jobs https://www.apify.com/jobs
    - johnnyfived 9y ago
      It's great that you're communicating openly on HN. I just sent an application for the Junior Web Developer position. Looking forward to hearing back!
  - jardah 9y ago
    :D yep the documentation needs a lot of work. It started as a test of an idea, then slowly became a usable tool and the code was getting incrementaly more complex without me event noticing. I only added the readme on github yesterday and there are basicaly no tests... :(

rmateus 9y ago

Is it able to deal with digital certificates?

jancurn 9y ago
You can use https://www.htbridge.com/ssl/ https://www.htbridge.com/ssl/ for that

_Chief 9y ago

how about caching the default entry (static url instead) + attribs, to ease demoing. at the moment it's been analyzing for more than >5mins

jardah 9y ago
Good idea and I would implement that if I used an API from server to get the response. But currently I'm at the same time testing stability of Apify "Actor" solution and proxies, so for my case it's good that there are real requests with real responses, even if it's just from demo. Btw the fact that it's running for 5 minutes is a bug, that I will look at, since there is a timeout of 2 minutes and there are no hanging runs or runs that ended with timeout.
- ComputerGuru 9y ago
  You also don’t want to get your server blocked by yelp if they do rate limiting.
  - bpicolo 9y ago
    They have fairly aggressive scraper detection (and this is also against their ToS)
  - jardah 9y ago
    It's why I'm using proxies, every request is routed through different proxy address and the application as whole is rate limited. So hopefully I'm not making too much traffic on yelp. They are just a perfect example because they are using all types of data I'm looking for. When I find more good examples I will add them and rotate them for every page load. Btw when it comes to ToS and scraping, this is not much different from accessing their website through normal browser only instead of rendered content we should you analyzed data. The page is only loaded once same as in browser.

JustARandomGuy 9y ago

Suppose I wanted to extract an image that gets loaded async via Javascript (For example, a Pinterest page). How would that work? Looking at your documentation, it looks like I could parse the XHR array you supply. Could you suggest any other ways? I'm calling out Pinterest as an example here because they try to block their images from being easily downloaded, but if you have any other examples I'd like to hear them. It would be great if the page analyzer could supply a list of all the assets loaded with the web page; for example, any asset with a media type of image/* is listed in an images array, and so forth.

jardah 9y ago
Actually the list of assets shouldn't be that hard. Looking at pinterest the xhr requests for images are loaded immediately when page is open, so potentialy it then it's catched in onRequest function (only now I'm aborting the requests to save network trafic). I will try it our tomorrow and let you know in comment. Also, looking at pinterest, it's server rendered through ReactJS, so there is #initial-state script tag with first few images preloaded as urls, so if you cared only about the images on top without scrolling then this is the safest bet.