6 ms·

>I think a small operation is exactly the sort of outfit to do it. A small operation in terms of number of developers yes. But not being able to subsidise user

>I think a small operation is exactly the sort of outfit to do it. A small operation in terms of number of developers yes. But not being able to subsidise user growth means they will never be able to build their own full web index as the fixed cost of doing that is too high for a small number of users. As a consequence, they will always be at the mercy of Google/Bing. The range of things they can innovate on will always be limited. And the situation can only get worse as people start to expect more AI functionality. I doubt that you can build a sustainable niche product if the effort you have to put into it is just as big as if you're building for billions of users. Having few users is not what defines a niche. Niches are defined by specialisation.

marginalia_nu 3y ago
> A small operation in terms of number of developers yes. But not being able to subsidise user growth means they will never be able to build their own full web index as the fixed cost of doing that is too high for a small number of users. What do you reckon the cost of doing this would be? Just doing the napkin math for say a Mojeek sized index (couple of billion docs) doesn't seem to justify a particularly astronomical budget.
- ColinHayhurst 3y ago
  7 billion+ actually, and our investment level mentioned here: https://news.ycombinator.com/item?id=36517505 https://news.ycombinator.com/item?id=36517505
  - marginalia_nu 3y ago
    Shit, man. That number is larger every time I check :D
- fauigerzigerk 3y ago
  >What do you reckon the cost of doing this would be? That is indeed the key question. I tried to find out before commenting but the information I found is very vague. Internet Archive spends millions per year, but their index is updated far too slowly for a search engine. I have no idea what it costs to create a Google-size index. Do you think the Mojeek index is good enough to compete with Google?
  - marginalia_nu 3y ago
    > Do you think the Mojeek index is good enough to compete with Google? As a back-end for something like Kagi, it's sure getting there. Most of what sets Google apart is their exceptional level of user profiling. The actual indexing technology is likely on par with most of their competition. Of course very little of that enters into API queries.
  - neurostimulant 3y ago
    Internet archive snapshot the whole page while google only index the first few kb of text data on every page (forgot the number, was it 100kb max per page?), so maybe it'll cost lower to build a search engine index compared to an internet preservation project? Assuming they'll need to index 820 billion pages (the number of pages preserved in the internet archive), at 100kb each, and assuming they use a database with 0.3x text data compression efficiency, they'll need at least 24600 TB to store those text data. Assuming $300 per 16TB disk, then they'll need to spend at least $7,380,000 for disk alone. This is a lot of money just for storage and we haven't included stuff like replication and backup, indexing metadata overhead, etc.
    - DeathArrow 3y ago
      > Internet archive snapshot the whole page while google only index the first few kb of text data Is that a good way to build an index?
    - ricardo81 3y ago
      >100kb max per page? They changed that limit somewhere in the mid-2000s. Just as well, there's some CMS's out there where there's several hundred kilobytes of inline JS and CSS before any body text.
    - marginalia_nu 3y ago
      A search engine doesn't index the HTML code. You're looking at a few Kb per document. You also don't need multiple historical snapshots of the document like WM retains. So you're looking at maybe 20 bn docs, 4 Kb each. 100 Tb, before compression.
petra 3y ago
//As a consequence, they will always be at the mercy of Google/Bing. The range of things they can innovate on will always be limited. Sure. But Google/Bing has shown lack of interest in innovating for power searchers. I believe they'll continue doing so. And even so, this space is big, I'm sure there are many niches that would fit a scrappy, subscription based player.
alwaysbeconsing 3y ago
> not being able to subsidise user growth You're ignoring that Kagi runs on subscriptions, so this funding is not all there is. If the subscription covers basic user costs (which it should, because they just adjusted the pricing) or at least covers most of them then the 670K can be used for growth.

marginalia_nu 3y ago

> A small operation in terms of number of developers yes. But not being able to subsidise user growth means they will never be able to build their own full web index as the fixed cost of doing that is too high for a small number of users. What do you reckon the cost of doing this would be? Just doing the napkin math for say a Mojeek sized index (couple of billion docs) doesn't seem to justify a particularly astronomical budget.

ColinHayhurst 3y ago
7 billion+ actually, and our investment level mentioned here: https://news.ycombinator.com/item?id=36517505 https://news.ycombinator.com/item?id=36517505
- marginalia_nu 3y ago
  Shit, man. That number is larger every time I check :D
fauigerzigerk 3y ago
>What do you reckon the cost of doing this would be? That is indeed the key question. I tried to find out before commenting but the information I found is very vague. Internet Archive spends millions per year, but their index is updated far too slowly for a search engine. I have no idea what it costs to create a Google-size index. Do you think the Mojeek index is good enough to compete with Google?
- marginalia_nu 3y ago
  > Do you think the Mojeek index is good enough to compete with Google? As a back-end for something like Kagi, it's sure getting there. Most of what sets Google apart is their exceptional level of user profiling. The actual indexing technology is likely on par with most of their competition. Of course very little of that enters into API queries.
- neurostimulant 3y ago
  Internet archive snapshot the whole page while google only index the first few kb of text data on every page (forgot the number, was it 100kb max per page?), so maybe it'll cost lower to build a search engine index compared to an internet preservation project? Assuming they'll need to index 820 billion pages (the number of pages preserved in the internet archive), at 100kb each, and assuming they use a database with 0.3x text data compression efficiency, they'll need at least 24600 TB to store those text data. Assuming $300 per 16TB disk, then they'll need to spend at least $7,380,000 for disk alone. This is a lot of money just for storage and we haven't included stuff like replication and backup, indexing metadata overhead, etc.
  - DeathArrow 3y ago
    > Internet archive snapshot the whole page while google only index the first few kb of text data Is that a good way to build an index?
  - ricardo81 3y ago
    >100kb max per page? They changed that limit somewhere in the mid-2000s. Just as well, there's some CMS's out there where there's several hundred kilobytes of inline JS and CSS before any body text.
  - marginalia_nu 3y ago
    A search engine doesn't index the HTML code. You're looking at a few Kb per document. You also don't need multiple historical snapshots of the document like WM retains. So you're looking at maybe 20 bn docs, 4 Kb each. 100 Tb, before compression.

petra 3y ago

//As a consequence, they will always be at the mercy of Google/Bing. The range of things they can innovate on will always be limited. Sure. But Google/Bing has shown lack of interest in innovating for power searchers. I believe they'll continue doing so. And even so, this space is big, I'm sure there are many niches that would fit a scrappy, subscription based player.

alwaysbeconsing 3y ago

> not being able to subsidise user growth You're ignoring that Kagi runs on subscriptions, so this funding is not all there is. If the subscription covers basic user costs (which it should, because they just adjusted the pricing) or at least covers most of them then the 670K can be used for growth.