11 ms·
The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Go
by tyingq 6y ago
The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling.
Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from.
Airlines are a good example. If you search for flight status for a particular flight, Google presents that flight status in a box. As an end user, that's great. However, that sort of search used to (most times) lead to a visit to the airline web site.
The airline web site could then present things Google can't do. Like "hey, we see you haven't checked in yet" or "TSA wait times are longer than usual" or "We have a more-legroom seat upgrade if you want it".
Google took those eyeballs away. Okay, fine, that's their choice. But they don't give anything back, which removes incentives from the actual source to do things better.
You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.
In short, I don't think the crawler is the problem. And I don't think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away.
- benatkin 6y agoThat's the result of the crawling, and it preventing competition. Google would much prefer that people complain about the details while ignoring the root cause.
- tyingq 6y agoI don't understand that. The crawling access is mostly the same as it ever was. Google's SERP pages are not. A mutually beneficial search engine that respects it's sources would still crawl the same. Google used to be that. The core problem is incentives: http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html "we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm."
- benatkin 6y agoThat's incorrect. Before the search oligopolies formed, new search engines could start up. There was excite, hotbot, altavista, and more. Now they don't have access. Search these comments for census.gov.
- tyingq 6y agoThere are companies that do pretty well in this space, like ahrefs, for example. They do resort to trickery, like proxy clients that look like home computers or cell phones. But, if a small entity like ahrefs can do it, anyone can do it. In a nutshell, though, I don't see equal access for all crawlers changing anything. Maybe that's the first barrier they hit, but it isn't the biggest or hardest one by far. Bing has good crawler access, but shit market share.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- throwaway_kufu 6y agoThey are not just taking away internet traffic, but in the flights example, they actually acquired an aggregate flight/travel company and so they are actually entering markets and competing with their own ad customers. Then it comes fully circle to Google unfairly using their market position vis-a-vis data, search and advertising. It’s a win-win Google lets the data dictate which markets to enter and on one hand they can jack up advertising fees on customers/competitors and unfairly build their own service into search above both ads and organic results.
- marshmallow_12 6y agoAren't there anti trust laws to prevent this kind of thing?
- samuelizdat 6y agoThat depends, would Google let us know?
- rijoja 6y agonot if they could avoid it
- adamcstephens 6y agoYes, but they lack enforcement.
- midoBB 6y agoAnti-trust in the US tend to not hit the big tech players as much they do other sectors. Also there is actually a debate in the judicial system about the extent of Anti trust laws themselves.
- sitkack 6y agoChicago school basically published a bunch of position papers that made feudal corporations a legal entity that "aren't monopolies" because the give things away for free. Because the consumer isn't paying, it can't be bad.
- bouncycastle 6y agoIn regards to airlines, Google and Amadeus have a partnership I believe. Amadeus is the main source of data for many of these airline websites. If Google gets the data from Amadeus directly and not these websites, they are just cutting out the middleman. I don't shed a tear for any of these middleman (together with their Dark Pattern UX design).
- tyingq 6y agoAmadeus isn't a source of flight status. It is a source for (some) planned schedules and fares. Global distribution systems are a complex topic that's hard to sum up on HN. For flight status, Google is pulling from OAG and Flight Aware, and also from airline websites. Though they don't show airline sites as a source.
- dan-robertson 6y agoThe way to look at this from Google’s point of view is to realise that most websites are slow and bad[1], so if Google sent you there you would have a bad experience with a bad slow website trying to find the information you want. Google want to make it better for you. [1] it feels like Google have contributed a lot to websites being slow and bad with eg ads, amp, angular, and probably more things for the other 25 letters of the alphabet.
- deleted 6y ago[deleted]
- zentiggr 6y ago> Google want to make it better for you. Hehe, sure, nothing nefarious or greedy here... move along, move along, nothing to see...
- treis 6y ago>However, that sort of search used to (most times) lead to a visit to the airline web site. I don't think that's correct. In the old days you'd either call a travel agent or use an aggregator like expedia. Google muscles out intermediaries like Expedia, Yelp, and so on. It's not likely much better or worse for the end user or supplier. Just swapping one middleman for another.
- tyingq 6y agoI can't prove it was that way, but I spent a lot of time in the space. For a long time, the airline's site used to be the top organic result, and there was no widget. Similar for other travel related searches (not just airlines) over time. Google has been pushing down organic results in favor of ads and widgets for a long time...and slowly, one little thing at a time. Like no widgets -> small widget below first organic result -> move the widget up -> make it bigger -> etc.
- josefx 6y agoOnly if Google stays around long term. I wouldn't be surprised if each free product on its graveyard took down a dozen of competing products before it was killed of.
- pc86 6y agoThen someone can start a competitor up again, right? Assuming there's actually a market for it.
- josefx 6y agoNot every market is lucrative in the extreme and it can take a long time to recover from being "disrupted". I think it is also a common practice for larger shopping chains to dump prices when they open a new location in order to clear out the local competition, so the damage it causes is well understood to be long lasting.
- deleted 6y ago[deleted]
- veltas 6y agoI swear something like 50% of those digests are totally incorrect as well. It's amazing they have kept the feature because it has never had a very high signal-to-noise ratio. I never trust what's presented in these digests without double-checking the source page.
- tyingq 6y agoI remember when rich snippets (one type of those widgets) came out there were a lot of funny examples. One for a common query about cancer treatments that pulled data from a dodgy holistic site saying that "carrots cured most types of cancer" (or something like that). There was a similar one where Google emphatically claimed a US quarter was worth five cents in a pretty and large snippet graphic.
- Mauricebranagh 6y agoI recall in the last uk election google got the infographic of party leaders about 60-70% wrong. And quite often a people also ask refine is just some random guys comment from redit.
- BeFlatXIII 6y agoThe most memorable rich snippet humor I've seen is a horse breeder sharing a story of how her searches gave snippets with my little ponies as the preview image.
- bombcar 6y agoHave you heard the story of Thomas Running? It’s a story Google will tell you. (Search who invented running)
- jeffbee 6y agoYou're wrong on a lot of facts here. Google Flights doesn't get its data just by crawling, they get it from Sabre, the FAA, Eurocontrol, etc. Airlines are, obviously, extremely pleased to disseminate this information. Google Flights "gives back" in the exact same way as any other travel outlet: they book passengers. As for Wikipedia, the WMF is quite happy that most of their traffic is now served by Google. WMF is in the business of distributing knowledge, not in the eyeballs business. Serving traffic is just a cost for them. The main problem has been that the average cost for Wikipedia to serve a page has gone up, because many readers read it via Google, and more people who visit Wikipedia are logged-in authors, which costs them more to serve. I'm sure there's an easy solution to this problem (for example, beneficiaries of Wikipedia can donate compute facilities and services, or something along those lines).
- tyingq 6y agoThey don't get individual flight status (what I was talking about) from Sabre or the FAA or Eurocontrol. I didn't get into fares and planned schedules and Google Flights, that's a different topic. I was talking about the big widget you get for queries on status for a particular flight, which is not Google Flights. They have relented in some ways, rolling out stuff in the widget like: "The airline has issued a change fee waiver for this flight. See what options are available on American's website" But obviously, that kind of stuff isn't shown on Google for quite some time after it exists on the source site. And the widget pushes the organics off the fold unless you have a huge monitor. As for Wikipedia, I was referring to this: https://news.ycombinator.com/item?id=26487993 https://news.ycombinator.com/item?id=26487993 "Airlines are, obviously, extremely pleased to disseminate this information" In the same way that publishers love AMP, yes. They don't actually like it, but they are forced to make the best of it.
- jeffbee 6y agoOh, status. I was thinking of schedules. Still, what is the point for the consumer of being directed to an airline's terrible status page? And are they even capable of being crawled? Looking at American's site (it was the most ghastly airline that sprang to mind) I don't see how a crawler would be able to deal with it, and indeed the Google snippet for AA flight status, on the aa.com result which is far down in the results page, just says "aa.com uses cookies" which is about what you'd expect. In this case, I want to be sent literally anywhere but aa.com.
- onlyrealcuzzo 6y agoWikipedia isn't monetized. Doesn't it benefit them if Google is serving their content for free and people are finding the information they want without having to hit Wikipedia?? And also, isn't Google the largest sponsor for Wikipedia already? In 2019 - Google donated $2M [1]. In 2010, Google also donated $2m [2]. [1] https://techcrunch.com/2019/01/22/google-org-donates-2-million-to-wikipedias-parent-org/ https://techcrunch.com/2019/01/22/google-org-donates-2-milli... [2] https://en.wikipedia.org/wiki/Wikimedia_Foundation https://en.wikipedia.org/wiki/Wikimedia_Foundation
- dmitriid 6y agoGoogle was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla. 2 mln is probably Google's hourly profit. For that they get one of the biggest knowledge bases in the world. It's basically free as far as Google is concerned. The instant Google becomes confident they can supplant Wikipedia, they will.
- magicalist 6y ago> Google was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla. Google isn't a sponsor of Mozilla, they're a customer. Do people think Google is "sponsoring" Apple with $1.5 billion a year too?
- foobarian 6y ago> they're a customer. The cynic in me thinks the product is anti-trust insurance.
- dmitriid 6y agoGoogle being Apple's customer doesn't mean Google isn't sponsoring Mozilla. These are two very different companies with a very different relationship with Google. And very different influences on Google. Google wants to be on iOS. It brings customers to Google. A lot of them. iOS is possibly more profitable to Google than Android even with all the payments Apple extracts from them. Google needs Mozilla so that Google may pretend that there's competition in browser space and that they don't own standards committees. The latter already isn't really true, and Google increasingly doesn't care about the former.
- magicalist 6y ago> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Wikipedia visitors, edits, and revenue are all increasing, and the rate that they're increasing is increasing, at least in the last few years. Is this a claim about the third derivative? > Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for. The Wikimedia Enterprise thing seems like it has nothing to do with missing visitors and that companies ingesting raw Wikipedia edits are an opportunity for diversifying revenue by offering paid structured APIs and service contracts. Kind of the traditional RedHat approach to revenue in open source: https://meta.m.wikimedia.org/wiki/Wikimedia_Enterprise https://meta.m.wikimedia.org/wiki/Wikimedia_Enterprise
- tyingq 6y agoSee https://searchengineland.com/wikipedia-confirms-they-are-steadily-losing-google-traffic-228237 https://searchengineland.com/wikipedia-confirms-they-are-ste... from 2015. Google's widgets that present Wikipedia data do reduce visitors to Wikipedia. Or see page views on English Wikipedia from 2016-current: https://stats.wikimedia.org/#/en.wikipedia.org/reading/total-page-views/normal|line|2016-01-01~2021-03-26|~total|monthly https://stats.wikimedia.org/#/en.wikipedia.org/reading/total... Looks pretty flat, right? Does that seem normal? As for Wikimedia Enterprise, you do have to read between the lines a bit. "The focus is on organizations that want to repurpose Wikimedia content in other contexts, providing data services at a large scale".
- SamBam 6y agoThe first link doesn't seem quite conclusive (see the part at the bottom), and also doesn't give evidence that Google's widgets are to blame. The flattening of users could also be due to a general internet-wide reduction in long-form (or even medium-form) non-fiction reading. How are page views for The New York Times? Seems like it should be simple to A/B test, though. Obviously Google could do it themselves by randomly taking away the widget, but would could also see whether referrals from non-Google search engines (though they are themselves a tiny percentage) continue to increase while Google remains flat.
- JKCalhoun 6y ago> In short, I don't think the crawler is the problem. Except that, allow other companies to crawl/compete, and you can take eyeballs away from Google (which may well then return eyeballs to Wikipedia so long as the Google competitors don't also present scraped data).
- deleted 6y ago[deleted]
- supert56 6y agoPerhaps I am misunderstanding or over simplifying things but it always surprises me that there are legal cases brought against companies who scrape data when so many of Google's products are doing exactly this. It definitely feels like one set of rules for them and a different set for everyone else.
- lupire 6y agoGoogle doesn't scrape anything that the site owner objects to.
- Spivak 6y agoI mean it's not that weird that a company would authorize major search engines scraping them but no one else. I don't really see this as Google playing by different rules so much as economic incentives being aligned in Google's favor.
- dr-detroit 6y agoSo nobody is going to book air travel? I cant hardly follow what youre even saying besides google=bad.
- ChrisArchitect 6y agoThey're making it easier to search for flights and arrange a trip. It's UX and makes me not hate the airlines/travel process as much. And I end up buying the flight from the airline anyways, and in many cases doing the arranging on the airline site in the end once it's determined, so Google is giving that back. They're not taking stuff from the airlines, I mean what ads and stuff are on the airline sites anyways specifically during the search process. Where they are taking away is from the Expedia's and other aggregation sites that offer a garbage/hodgepodge experience that drives people crazy.
- tyingq 6y agoYou're talking about Google Flights, which is completely unrelated to flight status.
- devoutsalsa 6y agoI've noticed that sometimes Google had updated flight information before the displays at the airport.
- tyingq 6y agoFor the most part individual airports own that infrastructure. So it's hard to generalize. For most types of notable flight status/time changes, however, airlines usually know first. There are exceptions, like an airport-called ground stop.
- gtm1260 6y agoI'm not sure I agree with this. I think airline websites are so garbage filled that they've driven people to use the simple alternative of the google flights checkout. It's a bit of a vicious cycle, but In general most websites are so chock filled with crap that not having to click into them for real is a relief!
- gxs 6y agoIt's not Google's prerogative to scrape a website and display its content, no matter how awful the website.
- michaelmrose 6y agoIf 1 airline let me view information in a friendly fashion and the other didn't I would do business with the first. Lest we forget the money in that scenario is from butts in seats not clicks on a website. The particular example is ill chosen as google is actually taking on a cost, taking nothing, and gifting the airline a better ui.
- BEEdwards 6y agoIf you make an awful website that can be scrapped it's a matter of when not if someone will take your data and give it to your consumers whether your trying to upsell them or not...
- tyingq 6y agoI'm talking about flight status. Not Google Flights, shopping, or booking. There are events associated with flight status that Google doesn't know. Like change fee waivers, cash comp awards to take a later or earlier flight, seat upgrades, etc.
- creato 6y agoYeah, the Google flights issue is difficult. On one hand, the business practice is problematic. On the other hand, Google flights is so much better than its competitors it's ridiculous. If there was a way to split Google flights into a separate company and somehow ensure it wouldn't devolve into absolute trash like its competitors, that would be a good thing.
- wbl 6y agoDoes the concergie of a hotel take anything away when he informs you that your flight has been delayed?
- tyingq 6y agoIt's hard for me to make that an apt analogy. She's not well known as a portal to find websites, which is what Google had been for most of its existence. It's pretty difficult to come up with a non-computer analogy for how Google works now. Pick a different space, and the power imbalance is quite clear. If they wanted, they could destroy StackExchange very quickly with these widgets.
- mdaniel 6y agoOnly if you are presuming they are going to make a question answering widget, too, since the content on stackexchange doesn't materialize out of thin air
- 838812052807016 6y agoStandardized interoperability enables overall progress. Every airline doesn't need their own webpage. They could all provide a standard API.
- tyingq 6y ago"Every airline doesn't need their own webpage. They could all provide a standard API." That's sort of how it works in the corporate booking tool world. It is decidedly not a better experience for end users, IMO. There's quite a lot about each airline that is different, so any unified approach is a lowest common denominator. You'll notice things like loyalty points, for example, have more rich data on the airline's website. And that some fares are ONLY on the website. Or that seat maps have more useful detailed info, etc. And that's all shopping/booking. Departure control, flight status, upgrade/downgrade, check-in, seat upgrades, standby, etc, are for the most part only on the airline's website.
- ilaksh 6y agoThe way that the web has been fundamentally broken by Google and other companies is one of the reasons I am excited about an alternative protocol called Gemini. It doesn't replace the web entirely, but for basic things like exchanging information, it's great. https://gemini.circumlunar.space/ https://gemini.circumlunar.space/
- Siira 6y agoLarge swaths of web are garbage. Wasting people's time and attention on visiting pointless sites for something presentable in a small box is obviously not economical. And if some of the sources somehow die? New sources will spring up. It doesn't matter.
- lelanthran 6y ago> And I don't think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away. What makes you think they care? Killing off the sources of content might even be there goal. If they kill off sources of content, they'd be more than happy to create an easier-to-datamine replacement. Hypothetically, if they killed off wikipedia, they are best placed to use the actual wikipedia content[1] in a replacement, which they can use for more intrusive data-mining. Google sells eyeballs to advertisers; being the source of all content makes them more money from advertisers while making it cheaper to acquire each eyeball. [1] AFAIK, wikipedia content is free to reuse.
- tyingq 6y agoI'm not suggesting it's illegal. There are a great many practices that are legal that I dislike.
- stjohnswarts 6y agoI think the flight arrivals/departures is a bad example. A good example might be putting flights.google.com on the first page or even allow it to exist.
- tyingq 6y agoNot sure what you mean. Both for flight status as well as flight shopping, Google drops a huge widget at the top and pushes everything else down, below the visible fold.
- plumeria 6y ago> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for. Do you have a link of that product/service from Wikipedia?
- Camillo 6y ago> The airline web site could then present things Google can't do. Like "hey, we see you haven't checked in yet" or "TSA wait times are longer than usual" or "We have a more-legroom seat upgrade if you want it". If I'm a passenger, there's plenty of ways for airlines to notify me. If I'm searching for a flight status online, it's because I'm picking someone up. If I want more information, I'll click through. I don't see how either me or the airline are hurt. > You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for. Why is that even a problem for Wikipedia?
- tyingq 6y agoIt's a progression. It's not a huge problem, by itself, for either. But Google shareholders want to continue the same YoY gains. The only cash cow is search, so they continue to take screen real estate that used to go to others, and take it for themselves. Whether that's more ads, or more widgets, or whatever. Yes, it's legal. But it does reduce visitor interactions for those sites. Reduced visitor interactions isn't good for web sites...it takes away incentives, reduces brand value, reduces revenue. Eventually, that is not great for consumers. Ever been a middleman? Squeeze your suppliers enough, and you kill them. The next supplier will pre-emptively cut quality, features, etc, because they know you're going to try and squeeze them to death. "If I'm searching for a flight status online, it's because I'm picking someone up" That's one use case, it's not all of them. There's middle ground too, like the bones they throw Wikipedia in the form of links for more info.
- nextaccountic 6y ago> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for. What product is that?
- nathanh4903 6y ago>Google's widgets have been reducing traffic to Wikipedia pretty dramatically. But wouldn't this be a good thing? Since wikipedia is a nonprofit aiming to provide knowledge, google stealing & caching their content might help serve information to more while reducing wikipedia's server load, so IMO it might not be so bad for wikipedia.