4 ms·
crawling the web is only easy if all you ever do is following hrefs... we're in the world of single-page turing-complete web apps and you're underestimating the
by baq 9y ago
crawling the web is only easy if all you ever do is following hrefs... we're in the world of single-page turing-complete web apps and you're underestimating the amount of pure compute, ingress and egress bandwidth necessary to keep the index up to date. there's a reason google has designed and built their data centers, their server chassis and even power supply.
- kinkrtyavimoodh 9y agoMere difficulty of a problem does not make those who do solve it a natural monopoly. Of course these player may still be a practical (in effect) monopoly due to their being way better than the competition. Making a kickass image editing application like Adobe Photoshop is difficult too (as anyone who has had to suffer GIMP would attest), but that doesn't mean Photoshop forms a natural monopoly.