11 ms·

> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1

• hsuduebc2 9mo ago
  I don’t think it’s comparable to today’s AI race. Google has a monopoly, an entrenched customer base, and stable revenue from a proven business model. Anyone trying to compete would have to pour massive money into infrastructure and then fight Google for users. In that game, Google already won. The current AI landscape is different. Multiple players are competing in an emerging field with an uncertain business model. We’re still in the phase of building better products, where companies started from more similar footing and aren’t primarily battling for customers yet. In that context, investing heavily in the core technology can still make financial sense. A better comparison might be the early days of car makers, or the web browser wars before the market settled.
  • ghm2199 9mo ago
    > ... stable revenue from a proven business mode... In that game, Google already won. But if they were to pour that money strategically to capture market share one of two things would happen if google was replaced/lost share: 1. it would be the start of the commoditization of search. i.e. search engine/index would become a commodity and more specialized and people could buy what they want and compete. 2. A new large tech company takes rein. In which case it would be as bad as this time. Like what I don't get is that if other big tech companies actually broke apart monopoly on search, several google dominos in mobile devices, browser tech, location capabilities would fall. It would be a massive injection of new competition into the economy, lots of people would spend more dollars across the space(and ad driven buying too) money would not accrue in an offshore tax haven in ireland To play the devils advocate, I think the only reason its not happening is because meta, apple, microsoft have very different moats/business models to profit off. They all have been stung one time or another is small or big ways for trying to build something that could compete but failed. MS with bing, Meta with facebook search, Foursquare — not big tech but still — with Maurauder's Map.
• hamdingers 9mo ago
  > If other tech companies really wanted to break this monopoly, why can't they just do it Google is a verb, nobody can compete with that level of mindshare.
  • wongarsu 9mo ago
    Xerox is a verb, but most copy machines I see are made by their competition
    • hamdingers 9mo ago
      Wonder why that could be? https://www.nytimes.com/1975/07/31/archives/xerox-settlement-is-approved-by-ftc.html https://www.nytimes.com/1975/07/31/archives/xerox-settlement...
  • eikenberry 9mo ago
    Kleenex isn't the only brand of tissues sold in stores.
  • Zyst 9mo ago
    So were AOL, and Skype
    • dylan604 9mo ago
      I don't ever recall anyone using AOL as a verb. How would you do that?
      • terespuwash 9mo ago
        Let me AOL this for you
        • dylan604 9mo ago
          said no one ever
• xnx 9mo ago
  > If other tech companies really wanted to break this monopoly, why can't they just do it Companies would rather sue than try and compete by investing their own money.
• walls 9mo ago
  A huge amount of the web is only crawlable with a googlebot user-agent and specific source IPs.
  • deleted 9mo ago
    [deleted]
  • Imustaskforhelp 9mo ago
    > And given you-know-what, the battle to establish a new search crawler will be harder than ever. Crawlers are now presumed guilty of scraping for AI services until proven innocent. I have always wondered but how does wayback machine work, is there no way that we can use wayback archive and then run a index on top of every wayback archive somehow?
    • ghm2199 9mo ago
      You can read https://hackernoon.com/the-long-now-of-the-web-inside-the-internet-archives-fight-against-forgetting https://hackernoon.com/the-long-now-of-the-web-inside-the-in... it was a nice look into their infra structure. One could theoretically build it. A few things stand out: 1. IIUC depends a lot on "Save Page Now" democratization, which could work, but its not like a crawler. 2. In absence of alexa they depend quite heavily on common crawl, which is quite crazy because there literally is no other place to go. I don't think they can use google's syndicated API, cause they would then start showing ads in their database, which is garbage that would strain their tiny storage budget. 3. Minor from a software engineering perspective but important for survival of the company: since they are an artifact of record storage, to convert that to an index would need a good legal team to battle google to argue. They do that the DoJ's recent ruling in their favor.
  • deepsquirrelnet 9mo ago
    I do not know a lot about this subject, but couldn’t you make a pretty decent index off of common crawl? It seems to me the bar is so low you wouldn’t have to have everything. Especially if your goal was not monetization with ads.
• paxys 9mo ago
  Apple had a chance to break Google's search monopoly, but they chose to take billions from them instead. Microsoft had a chance (well another chance, after they gave up IE's lead) to break up Google's browser monopoly, but they decided to use Chromium for free instead. Ultimately all these decisions come down to what's more profitable, not what's in the best interests of the public. We have learned this lesson x1000000. Stop relying on corporations to uphold freedoms (software or otherwise), becuase that simply isn't going to happen.
  • charcircuit 9mo ago
    >but they chose to take billions from them instead. They chose to use Google with a revenue sharing agreement. Google is very well monetized. It would be very difficult for Apple to monetize their own search as good as Google can. >they decided to use Chromium Windows ships with Microsoft Edge as the browser which Microsoft has full control over.
• KellyCriterion 9mo ago
  Scraping is hard. Very good scraping is even harder. And today, being a scraping business is veeery difficult; there are some "open"/public indices, but none of these other indices ever took off
  • ghm2199 9mo ago
    Well sure yes, I don't contend with the fact that its hard, but if the top tech companies joined their heads I am sure if for example, Meta, Apple, MS have enough talent between to make an open source index if only to reap gains from the de-monopolization of it all.
    • Imustaskforhelp 9mo ago
      I mean, doesn't microsoft have bing?
      • ghm2199 9mo ago
        Yeah but no one uses it. I am not even sure people that are forced to use it like using it because it was productized it pretty poorly. After all who wants another google? They invested 100 Billion dollars, which is a lot of wasted money TBH. Search indexes are hard, surely, but if you were to strip it to just a good index on the browser, made it free, kept it fresh, it cannot be 100 billion dollars to build. Then you use this DoJ decision and fight against google to not deny a free index to have equal rights on chrome you can have a massive shot at a win for a LOT less money.
        • Imustaskforhelp 9mo ago
          > Yeah but no one uses it. I am not even sure people like using it because it was productized it pretty poorly. They invested 100 Billion dollars, which is a lot of wasted money TBH. I mean... Duckduckgo uses bing api iirc and I use duckduckgo and many people use duckduckgo. I also used bing once because bing used to cache websites which weren't available in wayback archive, I don't know how but It was pretty cool solution for a problem. I hate bing too and I am kind of interested in ecosia/qwant's future as well (yes there's kagi too and good luck to kagi as well! but I am currently still staying on duckduckgo)
• oh_fiddlesticks 9mo ago
  > 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it? FTA: > Context matters: Google built its index by crawling the open web before robots.txt was a widespread norm, often over publishers’ objections. Today, publishers “consent” to Google’s crawling because the alternative - being invisible on a platform with 90% market share - is economically unacceptable. Google now enforces ToS and robots.txt against others from a position of monopoly power it accumulated without those constraints. The rules Google enforces today are not the rules it played by when building its dominance.
  • baggachipz 9mo ago
    A classic case of climbing the wall, and pulling the ladder up afterward. Others try to build their own ladder, and Google uses their deep pockets and political influence to knock the ladder over before it reaches the top.
    • dylan604 9mo ago
      Why does Google even need to know about your ladder? Build the bot, scale it up, save all the data, then release. You can now remove the ladder and obey robots.txt just like G. Just like G, once you have the data, you have the data. Why would you tell G that you are doing something? Why tell a competitor your plans at all? Just launch your product when the product is ready. I know that's anathema to SV startup logic, but in this case it's good business
      • Nextgrid 9mo ago
        Running the bot nowadays is hard, because a lot of sites will now block you - not just by asking nicely via robots.txt, but by checking your actual source IP. Once they see it's not Google, they send you a 403.
        • eloisius 9mo ago
          Cloudflare’s ubiquity makes bootstrapping a search index via crawler virtually impossible, but what about data sources like Common Crawl?
• jeromechoo 9mo ago
  Building an index is easy. Building a fresh index is extremely hard. Ranking an index is hard. It's not just BM25 or cosine similarity. How do you prioritize certain domains over others? How do you rank homepages that typically have no real content in them for navigational queries? Changing the behavior of 90% of the non-Chinese internet is unraveling 25 years and billions of dollars spent on ensuring Google is the default and sometimes only option. Historically, it takes a significant technological counter position or anti-trust breakup for a behemoth like Google to lose its footing. Unfortunately for us, Google is currently competing well in the only true technological threat to their existence to appear in decades.
  • AlienRobot 9mo ago
    Good news! Google doesn't know how to rank pages either!
    • pas 9mo ago
      yet ... it works "ok" most of the time. not to mention that people mostly need wikipedia, the news, navigating the infuriating world of websites of big service providers (gov sites, or try to find anything on Microsoft's dark corner of the web), porn and brainrot but it's awfully hard to make traction on a business that provides this.
• citizenpaul 9mo ago
  >why can't they just do it Money. Google controls 99% of the adverting market. That's why its called a monopoly. No one else can compete because they can never make enough money to make it worth the costs of doing it themselves.
• w10-1 9mo ago
  Other comments mention difficulty, cost, conditions, etc. Also, competitive agreements: of the big players like Apple, Microsoft, Facebook/Meta, Amazon, etc., only Google is in the ad business. But it has credible threats of digging into their businesses - GCP, Android, (not to mention software licenses and competitive access to e.g., Samsung), etc. So they agree to cede the ad world to Google, to keep Google out of their businesses. The injunctions cannot be effective. Google ads are essentially a tax at a fine scale that rational people chose when it didn't change site behavior. But then Google ads changed the nature of the web itself, converting every snippet of information into an opportunity to monetize. Neither would change with a public search.org, and injunctions to license ad-free indexes won't change site behavior or publishers' self-interest in selling access to their content to Google alone. Google knows the injunctions are unworkable and ultimately ineffective. The only question is what price they have to pay to the Trump judiciary to counter them.