4 ms·

I was looking at several scraping solutions (e.g. imacros, selenium) that can handle DHTML for a project and they all have significant performance issues since

I was looking at several scraping solutions (e.g. imacros, selenium) that can handle DHTML for a project and they all have significant performance issues since they need to render the actual pages before processing them. A couple of thousands or rows isn't a problem but try anything more and you got a real performance bottleneck.

odonnell 16y ago
DHTML is server-side. You mean AJAX. Also, think of the page as an interface to a more lightweight web service. You should probably be parsing that directly.
- thirdusername 16y ago
  He's referring to this: http://en.wikipedia.org/wiki/Dhtml http://en.wikipedia.org/wiki/Dhtml I'm not sure what DHTML you are thinking of that would be server-side.
  - odonnell 16y ago
    Fuck, thinking of SHTML for some reason.
chsonnu 16y ago
Have you tried Watir? I'm not sure if it'll solve your performance issue, but it's been at least twice as fast as Selenium for me.