4 ms·
At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are be
by ddorian43 6y ago
At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ?
Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets.
How many websites actually ping google on new content ?
- echelon 6y agoThis is so fucked. We've encrusted ourselves into this walled fiefdom, and there's no way to break free. It's only going to get worse. Chrome will be the only browser. AMP the only delivery mechanism. Video will require DRM. Eventually, text content will too. Binary blobs with no ad blocking.
- einpoklum 6y agoWell, I think you're being overly alarmist, at least in the short term: DRM on Video has not really caught on, at least on-line; Non-Chrome browsers continue to have a significant share (mostly on Desktops); and ad blocking remains rather effective. In the long run, I'm definitely worried: Capitalist economies tend to see a concentration of capital, generally and in most sectors individually. And this seems to be a real danger with computing technology. Coupled with mass surveillance and the pushing of people to have their personal information held by those large tech companies, a dystopia is not inconceivable. PS - By AMP, do you mean Amazon Prime?
- AlexandrB 6y ago> DRM on Video has not really caught on, at least on-line This seems like a weird statement. All of the paid streaming services use DRM on Video, so all major browsers include the requisite black-box DRM modules. I'm actually surprised YouTube has not added Widevine DRM for all videos yet, but I'm sure it'll happen if RIAA/MPAA get annoyed enough with youtube-dl and the like. > PS - By AMP, do you mean Amazon Prime? I think he means Google AMP[1], which is slowly infecting more of the top search results on Google. [1] https://developers.google.com/amp/ https://developers.google.com/amp/
- einpoklum 6y agoI've never heard of this AMP thing. Is it really that popular? Valid point about paid streaming services - which I don't use.
- ffpip 6y agoAMP is incredibly popular. Every news site has enabled it. You have a 100% chance of seeing an AMP page in the top results for anything. Google had 2 options - make websites faster the normal way (remove bloat) or make websites faster by introducing AMP. AMP is controlled by Google. What do you think they did? They said they would reduce the site's ranking if they didnt use AMP. Within weeks, everybody except Wikipedia was introducing AMP.
- TechBro8615 6y agoAnd don't you dare scrape Google results. That's against their TOS! Rules for thee, not for me. Isn't it weird there is no machine-readable API to Google search results?
- withinboredom 6y agoI thought this is exactly how DuckDuckGo worked?
- withinboredom 6y agoYou don’t crawl unless you have to, Twitter[1], Facebook, WordPress.com[2] and other big services have a firehose you can apply to and get real-time changes. If you’re crawling the web, you’re probably doing it wrong or only servicing a particular niche. [1] https://developer.twitter.com/en/docs/twitter-api/v1/tweets/sample-realtime/overview/decahose https://developer.twitter.com/en/docs/twitter-api/v1/tweets/... [2] https://developer.wordpress.com/docs/firehose/ https://developer.wordpress.com/docs/firehose/
- ddorian43 6y agoYou missed the pricing. This is what you're doing wrong. And you missed the cloudflare part too.