3 ms·
The moat is worse than we think. New search engines are not only hobbled by the bandwidth and processing power and storage required to spider the web, but by th
by falcolas 5y ago
The moat is worse than we think. New search engines are not only hobbled by the bandwidth and processing power and storage required to spider the web, but by the websites who will preemptively disallow them because they're not Google.
I can't imagine trying to build a new search engine when the landscape is intentionally (if justifiably) hostile to new search engines.
- ALittleLight 5y agoCan't you just set your user agent to Google bot?
- pixl97 5y agoYou cant set your IP range to a Google IP
- nr2x 5y agorobots.txt is a joke tho.
- mrkramer 5y agoElaborate? It is the best thing web has for crawling rules. Yea ofc crawling rules can be broken and violated but all serious internet search engines respect robots.txt.