6 ms·
One way Google maintains their monopoly is that many websites block all bots except for the Google indexing bot. This makes it literally impossible for anyone
by jiggawatts 3y ago
One way Google maintains their monopoly is that many websites block all bots except for the Google indexing bot.
This makes it literally impossible for anyone to make a competing search engine because millions of doors are slammed in their face. There is no practical way to negotiate access at this scale either leaving no options for small startups — e.g.: AI-based search!
One possible anti-monopoly measure would be to force Google to mask the identity of their bots. E.g.: force them to use a random IP address pool that third parties can also use without their permission.
IMHO breaking up corporations is a bit heavy-handed and not the only remedy available.
- lp0_on_fire 3y agoBeing only tangentially familiar with the indexing bots…what legal barriers are in place to prevent a competitor from impersonating a google indexing bot? Is it just a matter of the google bot originates from x subnet so that’s the only one webmasters allow? What’s to stop a competitor from running their own bot but sending user-agent: totally-the-google-indexing-bot-and-not-a-competitor?
- jiggawatts 3y agoSource addresses: the Google bot traffic comes from a small set of Google-owned IP address blocks. Third parties bake this into things like Web Application Firewall (WAF) rules. For example, Azure App Gateway WAF has a policy category for “known bots” which includes Google but excludes your tiny AI startup. It’s a moat built by giant corporations to keep tiny players in their place. Google "helpfully" publishes their bot source IP addresses: https://developers.google.com/static/search/apis/ipranges/googlebot.json https://developers.google.com/static/search/apis/ipranges/go... AWS also provides named rules such as "bot:name:googlebot": https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html https://docs.aws.amazon.com/waf/latest/developerguide/aws-ma...
- creato 3y agoI agree it’s a moat, but why would azure restrain competition from google? I think it’s just yet another example of an anti-abuse collateral damage, like email anti-spam blocking small unknown servers or Cloudflare blocking unknown IPs.
- whatshisface 3y agoMoats are anticompetitive, that's the entire idea behind why founders want them. Who else would a moat keep away, customers? ;)
- kazinator 3y agoCustomers are kept in by moats.
- whatshisface 3y agoThat's lock-in, moats are things that keep your competitors out. Source: https://www.investopedia.com/ask/answers/05/economicmoat.asp https://www.investopedia.com/ask/answers/05/economicmoat.asp
- sokoloff 3y agoBecause Azure customers want Google to be able to index their sites. Would you host your e-commerce or social media site on a cloud provider that blocked Googlebot? It's not Microsoft giving Google a handout out of the goodness of their hearts; it's Azure customers demanding that functionality. (Those Azure customers also don't care about a random little search startup and probably don't want to pay any egress fees to serve traffic to it.) AWS offers the same: https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control-example-allow-verified-bots.html https://docs.aws.amazon.com/waf/latest/developerguide/waf-bo... Oracle as well: https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_allowlist.htm https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_... Akamai as well: https://www.akamai.com/products/bot-manager https://www.akamai.com/products/bot-manager
- nova22033 3y agoFor example, Azure App Gateway WAF has a policy category for “known bots” So why should we take Satya's testimony seriously? Azure is a Microsoft product and if what you're saying is true, Microsoft is part of the problem.
- sokoloff 3y agoAlmost surely that Azure customers want that. https://news.ycombinator.com/item?id=37746480 https://news.ycombinator.com/item?id=37746480
- FractalHQ 3y agoI feel like something as fundamental to society as the internet should have a public index, no?
- qudat 3y agoThe idea sounds nice but politicians controlling an index would be an abject disaster.
- pydry 3y agoCompetitive markets > Governments > monopolies
- qudat 3y agoGovernment is the ultimate coercive monopoly; all coercive monopolies derive their power from the arbiters of force — the government.
- kaibee 3y agoGovernments are at least supposed to be accountable to their constituencies, monopolies are only accountable to their shareholders.
- gottorf 3y ago"Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience."
- esafak 3y agoBusting a monopoly is not an exercise in tyranny. Do you weep for Standard Oil or Bell? If anything, the US government is lenient towards monopolies. None of the 21st century behemoths have been busted.
- passwordoops 3y ago>IMHO breaking up corporations is a bit heavy-handed and not the only remedy available. Respectfully disagree in Google's case. Indexing is not their only advantage - having full, unfettered access and control over email, maps, Android play, cloud, and the other myriad divisions plus knows what else via side deals is too much and fully justifies breaking up the business. Your suggestion works if we were still 2002-ish, before they got too big
- 0xDEF 3y agoIt's funny how eager Americans are to break up the Big Tech companies that happen to be the only reason the US economy is not a complete dumpster fire. Look at GDP growth for the US vs. the Eurozone since 2008. The high level of integration, both horizontal and vertical, is what makes the US Big Tech companies so economically productive and valuable. But sure. Kill your software industry. Does the US even know how to do anything else? Does the US even manufacture anything physical anymore? Last I checked tiny little Denmark produces more wind turbines than the US and tiny little Switzerland produces more CNC machines than the US.
- bigyikes 3y agoI can buy this argument for Apple, but Google? To me it seems like there is an intuitive split between Search+Ads, GSuite, YouTube, and Android. Why do you believe breaking these up would adversely affect the US economy?
- anon25783 3y ago> the only reason the US economy is not a complete dumpster fire. The US economy is a complete dumpster fire for those of us worth less than seven figures. Big tech monopolization does not strengthen our productive capacity, and breaking up the big tech companies won't weaken it. If anything, it will strengthen the economy by making it more viable to found a tech startup without the explicit goal of being bought out. We aren't "killing" our software industry; we're revitalizing it.
- enraged_camel 3y ago
- didntknowya 3y agothough chatGPT seem to have had no trouble scraping data from everywhere
- PumpkinSpice 3y agoIn the past couple of years, many large tech companies developed a legal theory that scraping the public web for "internal" purposes is OK, and that any ToS-based or technical restrictions are just suggestions that they don't have to follow. These "internal" purposes included growing your social network, monitoring or reverse-engineering the algorithms of competing search engines, and now, it includes training ML. Which is funny given that when others are doing it to them, they go to great lengths to stop it, and sometimes complain loudly or threaten lawsuits. I think the main reason the big players don't sue each other is that it's a bit of a Mutual Assured Destruction kind of a deal. Google is doing it to Microsoft, Microsoft is doing it to Google...
- deprecative 3y agoTechnology patents create a complex MAD situation. At the end of the day Alphabet will be paying tons of money to Microsoft who will pay it to Apple who will pay it to Samsung who will pay it to probably Nokia or Xerox or Nintendo or whatever. And this goes on for each and every company with wealth in these spaces.
- dboreham 3y agoThey (BigCo) traditionally cross-license to benefit each other and lock out new entrants.
- hackernewds 3y agoand too late companies like Reddit and DeviantArt realized the billions of $ in UGC value that their websites had laying stagnant.
- havnagiggle 3y agoGot any reports/statistics to back this up? I highly doubt websites are not wanting major search engines to index them. AFAIK it's been standard practice to use `User-agent: *` for a long time. There are other anti-crawling measures because the bad crawlers are not going to respect your robots.txt.
- jwilliams 3y agoIt's not necessarily a lot of sites that block non-popular bots - but often it's big sites (i.e. content-centric sites such as Social Media). Think Yelp, Twitter, LinkedIn, Instagram, etc. That can add up to a serious percentage of the web.
- Ferret7446 3y agoDoes Google penalize sites for not blocking other bots? If not, then it is not Google "maintaining their monopoly", it is everyone deciding for themselves to only support Google. The government forcing restrictions on companies that people choose freely is extremely dangerous and will definitely be used for political reasons.
- horsawlarway 3y agoI would argue the takeaway here is not precisely that this behavior is "maintaining their monopoly", rather that it's incredibly strong evidence that the monopoly exists.
- zdragnar 3y agoIt's not a transaction, though, just a stated preference by third parties. It has no bearing whatsoever on the legal issue at stake.
- crazygringo 3y agoThat's not true. First of all, no website owner in their right mind is going to block Bing. Second of all, they have to detect that you're a bot in the first place. Most sites don't employ sophisticated anti-bot technology. Simply running a headless browser (which you need to do anyways for JavaScript) with a common user-agent, and slow enough browsing not to be rate-limited, will let you index 99.9+% of sites.
- jamiek88 3y agoIt is true if you respect robots.txt.
- alexey-salmin 3y agoMany website owners do explicitly block Bing and all other crawling bots in their right minds. If you have a wide low-traffic website then bots of all sorts will make a majority of your traffic and subsequently a majority of your AWS costs. If you see money spent on search engine X indexing and very few users incoming from that search engine it's a rational decision to block it. Or ask for money (that actually happens). Overall it's a systematic problem with building a search competitor: 1) Your costs are largely proportional to the size of your index 2) Your income is proportional to your userbase 3) You need a huge index to be competitive even if you don't have any users yet So, very hard to bootstrap even when you exclude all other advantages of the existing monopoly like browser-based distribution. Microsoft has the money to beat the indexing problem so they argue about distribution in court but all the small players can't even get to that level of failure.
- crazygringo 3y ago> Many website owners do explicitly block Bing and all other crawling bots in their right minds. If by "many" you mean "less than 0.1%". Nearly all sites want traffic, which search engines provide. And you're moving the goalposts from the commenter I was responding to. Nobody disputes it takes a large capital investment to create an index in the first place. But this is the business world -- that's what investors are for. But the idea that websites provide their public content to Google and leave other crawlers with no way to access it is simply untrue. That is not a factor hindering competition.
- carlosjobim 3y ago> One way Google maintains their monopoly is that many websites block all bots except for the Google indexing bot. How can that be blamed on Google?
- sph 3y ago> One way Google maintains their monopoly is that many websites block all bots except for the Google indexing bot. No, I don't think that's true. I am writing a web crawler, not for search purposes, and I haven't seen preferential treatment for GoogleBot compared to others. Sure, some might be banned outright (though bad crawlers just ignore robots.txt and do whatever they want), but in most cases new bots have the same access rights than GoogleBot. Also, your sentence doesn't pass the sniff test: you claim Google has better access than all other crawlers; but robots.txt is solely in the hands of the webmaster. How does Google coerce most website owners to block other bots? There is no conspiracy at play here.
- nubela 3y agoTry LinkedIn, or Facebook, or Twitter, or Crunchbase.
- cmrdporcupine 3y agoAgree on the first part, but I think actually this is an argument for breaking them up along those lines. I've often said that the "breaking up" of Google that makes the most sense is to split the crawling/indexing service from the rest of the company. Similar to the way telecoms companies were forced -- in some countries -- to open up access to e.g. DSL infrastructure.
- delfinom 3y ago>One way Google maintains their monopoly is that many websites block all bots except for the Google indexing bot. Because Googlebot is actually _friendly_. O boy are there bad bots out there for search engines that often DDOS your site to shit. Yandex is probably the worst out there from experience. Not only will it hammer you, they put in extremely aggressive retries such that even if you ban it temporarily for 4 hours, the literal second your IP block of their spiders expire, they will flood you with requests, and they don't even know how long the ban is, their bots just keep trying and trying.
- deafpolygon 3y agoI agree. I don't relish the idea of certain other countries' services getting too large in the vacuum that breaking up Microsoft, Apple, Amazon, Alphabet will do. Just imagine TikTok, then have that applied to Search, Mail, etc. Do you want this?
- AtNightWeCode 3y agoI think most corps allows known bots. To allow all traffic to a major site is not possible. The problem is bigger the other way around I would say. That Google, AWS, Azure and so on use the same AS numbers for private and public cloud. It is not easy to detect if it really is the Google bot or some low-lifes performing DOS-attacks from GCP. Many attacks, especially state sponsored attacks, comes from trusted clouds.