3 ms·
Perhaps there could be some kind of 'Crawler consortium'? Under this consortium, website owners would be allowed to either allow all crawlers (approved by the
by gnramires 6y ago
Perhaps there could be some kind of 'Crawler consortium'?
Under this consortium, website owners would be allowed to either allow all crawlers (approved by the consortium) or none at all (that is, none that is in the consortium, i.e. you could allow a specific researcher or something to crawl your website on a case-by-case basis).
This consortium would be composed of the search engines (Google, MS, other industry members), as well as government appointed individuals and relevant NGOs (electronic frontier foundation, etc?). There would be an approval process that simply requires your crawl to be ethical and respect bandwidth usage. Violations of ethics or bandwidth limits could imply temporary or permanent suspension. The consortium could have some bargain or regulatory measures to prevent website owners from ignoring those competitive and fairness provisions.
- dragonwriter 6y ago> Perhaps there could be some kind of 'Crawler consortium'? An industry-wide agreement not to compete for commercially valuable access to suppliers of data? Comprised of companies that are current (and in some cases perennial) focusses of antitrust attention? I think there might be a problem with that plan.
- gnramires 6y agoWell, yes, and a common solution to anti-trust cases, that I know of, is some kind of industry self-regulation. In this case I wouldn't trust the industry only to self-regulate; hence, they should at invite (while keeping a minority but not insignificant position) governments and civil society (ngos and other organizations) to participate. Could you better describe your objections?
- neolog 6y agoI don't see the problem. If a bunch of non-google companies pooled resources to make a crawl, that would reduce market concentration, not increase it.