4 ms·

> Doesn't mention the hardest part ... dealing with pages whose content is mostly dynamic and generated client side (SPA's) Given this is from 2004 I'm not sur

• wizofaus 4y ago
  That was about when I was writing my crawler (not for search but for rules-based analysis). Even in 2004 a lot of key DOM elements were created/modified client side.
  • wizofaus 4y ago
    Though I do remember now that we solved it by having a separate mechanism for accessing pages that required logging in or had significant client-side rendering by allowing the user to record a macro that was played back in a headless browser. Within a few years though it was obvious a crawler would need to be able to automatically handle client scripts.
  • deleted 4y ago
    [deleted]