3 ms·
Doesn't mention the hardest part I found when developing a crawler - dealing with pages whose content is mostly dynamic and generated client side (SPA's). Even
by wizofaus 4y ago
Doesn't mention the hardest part I found when developing a crawler - dealing with pages whose content is mostly dynamic and generated client side (SPA's). Even using V8 it's hard to do reliably and performantly at scale.
- sanjayts 4y ago> Doesn't mention the hardest part ... dealing with pages whose content is mostly dynamic and generated client side (SPA's) Given this is from 2004 I'm not surprised.
- wizofaus 4y agoThat was about when I was writing my crawler (not for search but for rules-based analysis). Even in 2004 a lot of key DOM elements were created/modified client side.
- wizofaus 4y agoThough I do remember now that we solved it by having a separate mechanism for accessing pages that required logging in or had significant client-side rendering by allowing the user to record a macro that was played back in a headless browser. Within a few years though it was obvious a crawler would need to be able to automatically handle client scripts.
- deleted 4y ago[deleted]
- marginalia_nu 4y agoIs this an actual problem, though? It seems like not even Google deals well with these types of pages. Often those types of pages are difficult to link to as well, as they're often highly stateful, applications rather than documents, and what you're seeing may not be what you get when you click the link. Not really what you want in a search engine.
- wizofaus 4y agoIt was certainly a requirement for the tool we were developing. Google's crawler most definitely does support client-side rendering: https://www.botify.com/blog/martin-splitt-javascript-rendering https://www.botify.com/blog/martin-splitt-javascript-renderi....