3 ms·
Write a JS snippet that will press the "Load more" button every 2s on the web interface. Let it run for a lot of time, on Chrome DevTools, then scrape the page
by fiatjaf 10y ago
Write a JS snippet that will press the "Load more" button every 2s on the web interface. Let it run for a lot of time, on Chrome DevTools, then scrape the page with Javascript. Seems relatively easy.
- xiconfjs 10y agoI did something like this with firefox, but after a while firefox was using so much RAM, it took him 10-20 sec. for a single request - I gathered ~40000 history entries with this technique. Perhaps Chrome DevTools will do a better job - I'll try.
- fiatjaf 10y agoOk, you have too many history entries. That will be a problem everywhere. Perhaps removing CSS, media and images from the DOM will reduce memory consumption and get your farther?