5 ms·
Originally I intended to make the crawler code available under an open source license at GitHub. However, as I better understood the cost that crawlers impose o
by mindcrime 8y ago
Originally I intended to make the crawler code available under an open source license at GitHub. However, as I better understood the cost that crawlers impose on websites, I began to have reservations. My crawler is designed to be polite and impose relatively little burden on any single website, but could (like many crawlers) easily be modified by thoughtless or malicious people to impose a heavy burden on sites. Because of this I’ve decided to postpone (possibly indefinitely) releasing the code.
Given that there are plenty of existing, open-source crawling engines out there, I don't see how this decision is really accomplishing anything. Concretely, Apache Nutch[1] can crawl at "web scale" and is apparently the crawler used by Common Crawl.
There’s a more general issue here, which is this: who gets to crawl the web?
This, to me, is the most interesting issue raised by this article. In principle, there's no particular reason that, say, Google, has to dominate search. If somebody clever comes up with a better ranking algorithm, or some other cool innovation, they should be able to knock Google off their perch the same way Google displaced Altavista. BUT... that's only true if anybody can crawl the web in the first place... OR something like Common Crawl reaches parity with the Google's of the world, in both volume and frequency of crawled data.
The first scenario is definitely questionable. Sure, you can plug the Googlebot user agent string into your crawler, but plenty of sites are smart enough to look at other factors and will reject your requests anyway. (I know, I used to work for a company that specialized in blocking bots, crawlers, etc.)
It really is a bit of a catch-22. Site owners legitimately want to keep bad crawlers/bots from A. consuming excessive resources, and B. stealing content, from their sites. But too much of this will lock us into a search oligopoly that isn't good for anybody (except maybe Google shareholders).
[1]: https://en.wikipedia.org/wiki/Apache_Nutch https://en.wikipedia.org/wiki/Apache_Nutch
- danielrhodes 8y agoThis seems like an oversimplification of Google’s value prop. Crawling the web is a somewhat trivial problem these days - ranking pages, removing spam, personalizing it to the viewer, and doing so in a matter of milliseconds from wherever you are in the world is a far greater problem and competitive advantage.
- mindcrime 8y agoPretty sure this is an oversimplification of Google’s value prop. I didn't say anything about Google's value prop. My point is, whatever their value prop is, it's built on top of their ability to crawl the web at mass scale, and very quickly. So anybody who wants to compete with Google, by being better at "ranking pages, removing spam, personalizing it to the viewer," or whatever, will need to be able to do crawl in a similar manner. IOW, crawling is part of the "price of admission".
- petters 8y ago> IOW, crawling is part of the "price of admission". Yes, but I think the comment you responded to were saying that it is an insignificant part these days.
- mindcrime 8y agoYes, but I think the comment you responded to were saying that it is an insignificant part these days. Maybe so. In which case, I'd say I disagree. Great crawling ability is definitely a necessary, but not sufficient, condition for building a competitive search engine. And while the technical aspects of building a large scale search engine have been at least partly trivialized by OSS crawling software, elastic computing resources in the cloud, etc., what is at issue is the possibility of site owners blocking anybody who isn't (Google|Bing|Baidu|etc). In this context, that's my concern: being blocked from crawling, if you're not already on the "allowed" list. Hence my reference to the question quoted above, from TFA.
- kbenson 8y agoI think the terminology you are looking for (or at least that you could use that might trigger people to accurately infer what you are trying to express), is that search and crawling are the foundation of everything google has built, in multiple aspects. In one aspect, it was literally their beginning, from which they were able to build a business and expand. In another, much more on point aspect, it underlies the majority of their services, either directly of a few steps removed. Like the foundation of a house, it may not always be the most visible aspect, and it may be taken for granted, but its contribution to the integrity of the whole can't be underestimated.
- cyberpunk0 8y agoEveryone gets to crawl. The net should have no restrictions, period
- colechristensen 8y agoDoes that lack of restrictions include allowing me to decide who has access to things I make available?
- anc84 8y agoNothing stops you from not serving those who you don't want to serve.
- dredmorbius 8y agoCrawling can be quite debilitating to sites.
- TomK32 8y agoI don't see any issue at all if the bot is respecting the robots.txt file. Any malicious user will figure some other evil way anyways, be it Nutch or a network of intelligent lightbulbs.
- merb 8y agoso you will actually respect: User-agent: google Allow: / User-agent: * Disallow: / well, yes it's good behavior to actually respect it, but well I've seen such robots.txt already which makes it really painful to create a competing search engine.
- taeric 8y agoThis has me ridiculously curious now. Is that common? Other than a random sampling of sites I go to, there a good way to get numbers on how often this is used? Edit: In my scanning, I have to confess that wikipedia's robot file is the best. Fairly heavily commented on why the rules are there. https://en.wikipedia.org/robots.txt https://en.wikipedia.org/robots.txt
- _trampeltier 8y agoI did run a Yacy web crawler (P2P websearch https://yacy.net https://yacy.net) a while ago. As far I remember I just saw Yandex for a few times disallowed in the robots.txt when I had trouble with crawling a site. Mostly I just got an empty website for my Yacy crawler instead the "real" Website.
- benfrederickson 8y agoI analyzed the top 1 million robots.txt files looking for sites that allow google and block everyone else here: https://www.benfrederickson.com/robots-txt-analysis/ https://www.benfrederickson.com/robots-txt-analysis/ - it's a relatively common pattern for major websites
- gcb0 8y agojust do like browsers did with user agent strings. call your bot "botx (google crawler compatible)" and crawl everything that allows Google bot without any weight on your conscience.
- dredmorbius 8y agoCrawl, but share. Another possible approach is for sites to be self-indexing, with sharable indices, and validation (and penalties) for deceptive practices (either false term inclusion or exclusion). The simple existence of a websearch and index protocol would wipe virtually all of Google's present value.
- jdironman 8y agoOr possibly a common open resource from which services can pull regularly updated crawl data. I'm not sure if what I am thinking of exists or is feasible or not, just a thought from someone with little experience in crawling in general. Although, then that would put load on a single source and someone has to pay for those resources. Maybe if there was a distributed resource then, I wonder if block-chain technology or something similar factor into such a niche.
- dredmorbius 8y agoRight: that's "crawl, but share". Both models are fairly isomorphic, modulo the crawl / search locus. The key is on agreed data structures, practices, and access.
- mindcrime 8y agoThat sounds a lot like what CommonCrawl does, and they are a good start. But from what I've seen, if you wanted to build a really competitive search engine, you'd still have to complement that with your own crawling. Not sure though, as I haven't done a deep dive into this topic.
- DeathArrow 8y agoBuilding a better search engine than Google is doable. Maybe it is not even very expensive - something in the order of tens/hundreds of millions might be enough. The only problem is: nobody would care, people will still use Google because it's all they know. Some people even ignore the fact that underneath they use an operating system and a browser.
- kalesho 8y ago> because it's all they know. Don't you think that's what they used to say about Altavista?
- justbaker 8y ago>but could (like many crawlers) easily be modified by thoughtless or malicious people to impose a heavy burden on sites. Because of this I’ve decided to postpone (possibly indefinitely) releasing the code. I came here to state just this. What would code for the crawler in the article do that premade DDoS bots don't already do? (already does that task better too) Even in 2012 when the article was published, there were plenty of open source crawlers AND tools designed specifically for "burdening websites with traffic" that were available.