5 ms·

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entirely proprietary engine and you need to CONSTANTLY refine all the above until it isn't garbage. Maybe you could muster something passable for $1B over 5 years with a strong core team that attracts great talent. If Apple actually does this, as they are rumored to, I bet they dump $10b into it just for the initial version.

kilroy123 6y ago
I agree and I have been hoping Apple builds a serious competitor. I welcome any competition at this point. Let's be real, not many people are using bing. People _would_ actually use apple search.
- ur-whale 6y ago
  >I welcome any competition at this point. Microsoft tried and failed to build a competitor and it's not like they have shallow pockets. They grossly underestimated a number of aspects: - The huge number of man-years invested in Google's search quality stack hand-tuning and what it would take to replicate it. - The fact that the machine learning field was simply not ready to tackle the search quality problem. - The infrastructure required to build a crawler / indexer stack as good as Google's I think in 2020, the second problem is within reach of many companies technically. It's mostly a matter of throwing enough money at optimized infrastructure. However, replicating the search quality stack is going to be very hard, unless someone makes a huge breakthrough in machine learning / language modeling / language understanding at a thousandth of the cost it currently takes to run something like GPT-3. The most likely candidate to execute properly on that last bit is - unfortunately - Google.
  - Krasnol 6y ago
    Sure but Microsoft has tried and failed to build a competitor to google not to DDG. I don't see it that hopeless. I feel it kinda is like starting Open Street Maps. It won't be perfect for a long time but there will be people who'd prefer it and help out.
- _underfl0w_ 6y ago
  > People would actually use apple search. Of course they would - it would be set as the default search on their iPhones with no clear-cut way to change it. You know, "security". The users don't know what's best for them, etc. as Apple seems to think.
AsyncAwait 6y ago
I think that is why the idea would be to have the tier 1 sites so you don't have to index as much.
- dumbfounder 6y ago
  I said ISN'T a pile of garbage :)
Aeolun 6y ago
If you want a good engine there is no need to index 100B pages, since 99% of the pages are blogspam.
- bmurphy1976 6y ago
  How are you going to identify what's blogspam and what's legitimate without indexing it all in the first place?
  - Aeolun 6y ago
    You load, detect and discard it? We’re already talking about building a search engine, might as well talk about a model to convincingly detect blogspam too.
    - account42 6y ago
      if(has_ads) { spam = likely; }
ricardo81 6y ago
Agreed, it's going to require significant investment in hardware and software. The recent UK Competitions and Market Authority report evaluating Google and the UK search market came to the conclusion a new entrant would require about 18 Billion GBP in capital to become a credible alternative search engine, in terms of size, quality, hardware, man hours making it. Remember Cuil? Had the size, the fanfare but unfortunately not the quality.
nostromo 6y ago
> If Apple actually does this, as they are rumored to, I bet they dump $10b into it just for the initial version. Google pays Apple more than that every year just to set Google as the default search engine on iPhones. In a way, Google is funding its future competitor.
lemax 6y ago
Open Street Map is a nice analogy for what could work. Aside from the open source maintenance of the map, there's also tons of corporate help in the background. Companies that are delivering OSM as a service or relying on it for their own services have an interest in making it better. MapBox, for example, apparently pays tons of people a salary who are contributing upstream to Open Street Map. If we can get an Apple/Microsoft/Other players collab maybe a venerable alternative can actually be built.
tibyat 6y ago
No, you just want a faster horse.

kilroy123 6y ago

I agree and I have been hoping Apple builds a serious competitor. I welcome any competition at this point. Let's be real, not many people are using bing. People _would_ actually use apple search.

ur-whale 6y ago
>I welcome any competition at this point. Microsoft tried and failed to build a competitor and it's not like they have shallow pockets. They grossly underestimated a number of aspects: - The huge number of man-years invested in Google's search quality stack hand-tuning and what it would take to replicate it. - The fact that the machine learning field was simply not ready to tackle the search quality problem. - The infrastructure required to build a crawler / indexer stack as good as Google's I think in 2020, the second problem is within reach of many companies technically. It's mostly a matter of throwing enough money at optimized infrastructure. However, replicating the search quality stack is going to be very hard, unless someone makes a huge breakthrough in machine learning / language modeling / language understanding at a thousandth of the cost it currently takes to run something like GPT-3. The most likely candidate to execute properly on that last bit is - unfortunately - Google.
- Krasnol 6y ago
  Sure but Microsoft has tried and failed to build a competitor to google not to DDG. I don't see it that hopeless. I feel it kinda is like starting Open Street Maps. It won't be perfect for a long time but there will be people who'd prefer it and help out.
_underfl0w_ 6y ago
> People would actually use apple search. Of course they would - it would be set as the default search on their iPhones with no clear-cut way to change it. You know, "security". The users don't know what's best for them, etc. as Apple seems to think.

AsyncAwait 6y ago

I think that is why the idea would be to have the tier 1 sites so you don't have to index as much.

dumbfounder 6y ago
I said ISN'T a pile of garbage :)

Aeolun 6y ago

If you want a good engine there is no need to index 100B pages, since 99% of the pages are blogspam.

bmurphy1976 6y ago
How are you going to identify what's blogspam and what's legitimate without indexing it all in the first place?
- Aeolun 6y ago
  You load, detect and discard it? We’re already talking about building a search engine, might as well talk about a model to convincingly detect blogspam too.
  - account42 6y ago
    if(has_ads) { spam = likely; }

ricardo81 6y ago

Agreed, it's going to require significant investment in hardware and software. The recent UK Competitions and Market Authority report evaluating Google and the UK search market came to the conclusion a new entrant would require about 18 Billion GBP in capital to become a credible alternative search engine, in terms of size, quality, hardware, man hours making it. Remember Cuil? Had the size, the fanfare but unfortunately not the quality.

nostromo 6y ago

> If Apple actually does this, as they are rumored to, I bet they dump $10b into it just for the initial version. Google pays Apple more than that every year just to set Google as the default search engine on iPhones. In a way, Google is funding its future competitor.

lemax 6y ago

Open Street Map is a nice analogy for what could work. Aside from the open source maintenance of the map, there's also tons of corporate help in the background. Companies that are delivering OSM as a service or relying on it for their own services have an interest in making it better. MapBox, for example, apparently pays tons of people a salary who are contributing upstream to Open Street Map. If we can get an Apple/Microsoft/Other players collab maybe a venerable alternative can actually be built.

tibyat 6y ago

No, you just want a faster horse.