5 ms·

> The difference with a web search engine is that nobody today can build such a thing completely from scratch I'm sorry, but can you substantiate this claim? I

> The difference with a web search engine is that nobody today can build such a thing completely from scratch I'm sorry, but can you substantiate this claim? I've seen no indication that a search engine is not buildable from scratch at all.

fho 3y ago
Sigh ... Of to the silicone mines it is then ... again /s
- isoprophlex 3y ago
  You mean, the beach?
  - sethammons 3y ago
    It is a joke based on needing to define "from scratch". Similar to, "how do you bake a cake from scratch? First you must create the universe." Op is gathering the raw ingredients to start fabricating his chips.
    - enneff 3y ago
      Yeah, that’s sand. :)
      - fho 3y ago
        Fair enough :-)
crazygringo 3y ago
Assuming you get the TCP/IP stack for free, you still need to build fully-featured HTTPS and a "webscale" multi-server database for document storage from scratch. The crawler is easy and so is something like PageRank, but then building the sharded keyword text search engine itself that operates at webscale is a whole other project... The point is that it's too much work for a single person to build all the parts themselves. It's only feasible if you rely on pre-existing HTTPS libraries and database and text search technologies.
- YetAnotherNick 3y ago
  Simple methods of search like exact matching are very fast using textbook algorithms. There are well known algorithm like suffix tree which could search in millions of documents in milliseconds.
  - crazygringo 3y ago
    That's not enough. It needs to be sharded and handle things like 10 search terms, each of which match a million documents and you're trying to find the intersection. Across shared results from 20 servers. Quickly. That's not a textbook algorithm.
    - marginalia_nu 3y ago
      Intersecting postingslists is a solved problem. You can do it in sublinear time with standard search engine algorithms. The problem space is embarassingly parallel, so sharding is no problem. Although, realistically, you probably only need 1 server to cope with the load and storage needs. This isn't 2004. Servers are big and fast now, as long as you don't try to use cloud compute.
      - crazygringo 3y ago
        1 server to index the whole internet? I don't care how big your server is -- that's going not to happen. And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch. It would only be feasible to build on top of existing libraries, which is the whole point here.
        marginalia_nu 3y ago
        > 1 server to index the whole internet? I don't care how big your server is -- that's going not to happen. Well let's do the math! 1 billion bytes is 1 GB. The average text-size of a document is ~5 KB, that is the HTML, uncompressed. 1 billion documents is thus about 5 TB raw HTML. In practice the index would be smaller, say 1 TB, because you're not indexing the raw HTML, but a fixed width representation of the words therein, typically with some form of compression as well. (In general, the index is significantly smaller than the raw data unless you're doing something strange) You could buy a thumbdrive that would a search index for 1 billion documents. A real server uses enterprise SSDs for this stuff, but even so, you don't even have to dip into storage server specs to outfit a single server with hundreds of terabytes of SSD storage. > And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch. I'm still not seeing why this is impossible. > It would only be feasible to build on top of existing libraries, which is the whole point here. I don't think there are existing libraries for most of this stuff. Most off-the-shelf stuff in the search space is for indexing smaller corpuses, they generally aren't built to scale up to corpus sizes in the billions (e.g. using 32 bit keys etc.)

fho 3y ago

Sigh ... Of to the silicone mines it is then ... again /s

isoprophlex 3y ago
You mean, the beach?
- sethammons 3y ago
  It is a joke based on needing to define "from scratch". Similar to, "how do you bake a cake from scratch? First you must create the universe." Op is gathering the raw ingredients to start fabricating his chips.
  - enneff 3y ago
    Yeah, that’s sand. :)
    - fho 3y ago
      Fair enough :-)

crazygringo 3y ago

Assuming you get the TCP/IP stack for free, you still need to build fully-featured HTTPS and a "webscale" multi-server database for document storage from scratch. The crawler is easy and so is something like PageRank, but then building the sharded keyword text search engine itself that operates at webscale is a whole other project... The point is that it's too much work for a single person to build all the parts themselves. It's only feasible if you rely on pre-existing HTTPS libraries and database and text search technologies.

YetAnotherNick 3y ago
Simple methods of search like exact matching are very fast using textbook algorithms. There are well known algorithm like suffix tree which could search in millions of documents in milliseconds.
- crazygringo 3y ago
  That's not enough. It needs to be sharded and handle things like 10 search terms, each of which match a million documents and you're trying to find the intersection. Across shared results from 20 servers. Quickly. That's not a textbook algorithm.
  - marginalia_nu 3y ago
    Intersecting postingslists is a solved problem. You can do it in sublinear time with standard search engine algorithms. The problem space is embarassingly parallel, so sharding is no problem. Although, realistically, you probably only need 1 server to cope with the load and storage needs. This isn't 2004. Servers are big and fast now, as long as you don't try to use cloud compute.
    - crazygringo 3y ago
      1 server to index the whole internet? I don't care how big your server is -- that's going not to happen. And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch. It would only be feasible to build on top of existing libraries, which is the whole point here.
      - marginalia_nu 3y ago
        > 1 server to index the whole internet? I don't care how big your server is -- that's going not to happen. Well let's do the math! 1 billion bytes is 1 GB. The average text-size of a document is ~5 KB, that is the HTML, uncompressed. 1 billion documents is thus about 5 TB raw HTML. In practice the index would be smaller, say 1 TB, because you're not indexing the raw HTML, but a fixed width representation of the words therein, typically with some form of compression as well. (In general, the index is significantly smaller than the raw data unless you're doing something strange) You could buy a thumbdrive that would a search index for 1 billion documents. A real server uses enterprise SSDs for this stuff, but even so, you don't even have to dip into storage server specs to outfit a single server with hundreds of terabytes of SSD storage. > And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch. I'm still not seeing why this is impossible. > It would only be feasible to build on top of existing libraries, which is the whole point here. I don't think there are existing libraries for most of this stuff. Most off-the-shelf stuff in the search space is for indexing smaller corpuses, they generally aren't built to scale up to corpus sizes in the billions (e.g. using 32 bit keys etc.)