3 ms·
I guess the google bot is now on by blocked list. There should be a public corporation, like Wikipedia, that starts scraping the Web and provides an api for an
by noosphr 5mo ago
I guess the google bot is now on by blocked list.
There should be a public corporation, like Wikipedia, that starts scraping the Web and provides an api for anyone to access.
- alex1138 5mo agoBut does Google actually 'own' the index? Have they acted anticompetitively as far as that goes?
- rockinghigh 5mo agoThey own tons of engagement data associated with that index from being the default search engine on most devices.
- jfil 5mo agoThat's a very good question that few people consider. Here is the answer: https://knuckleheads.club/how-google-distorts-the-market/ https://knuckleheads.club/how-google-distorts-the-market/
- Zambyte 5mo agoThe Internet Archive?
- ews 5mo agoyou mean this ? https://commoncrawl.org/ https://commoncrawl.org/