3 ms·
With the way websites work today surely the only possible way to build a search engine is to make something like a headless browser (similar to PhantomJS) that
by onion2k 12y ago
With the way websites work today surely the only possible way to build a search engine is to make something like a headless browser (similar to PhantomJS) that crawls the web like a user, seeing what the user sees, ignoring everything that's hidden from the user, and interpreting the importance of pages like a user would. Just parsing the HTML source of the page won't even get close to seeing the key features of a page any more.
Impressive work by Google to do that at scale, of course, but they'd be dead in the water if they didn't.