4 ms·

Internet archive snapshot the whole page while google only index the first few kb of text data on every page (forgot the number, was it 100kb max per page?), so

Internet archive snapshot the whole page while google only index the first few kb of text data on every page (forgot the number, was it 100kb max per page?), so maybe it'll cost lower to build a search engine index compared to an internet preservation project? Assuming they'll need to index 820 billion pages (the number of pages preserved in the internet archive), at 100kb each, and assuming they use a database with 0.3x text data compression efficiency, they'll need at least 24600 TB to store those text data. Assuming $300 per 16TB disk, then they'll need to spend at least $7,380,000 for disk alone. This is a lot of money just for storage and we haven't included stuff like replication and backup, indexing metadata overhead, etc.

DeathArrow 3y ago
> Internet archive snapshot the whole page while google only index the first few kb of text data Is that a good way to build an index?
ricardo81 3y ago
>100kb max per page? They changed that limit somewhere in the mid-2000s. Just as well, there's some CMS's out there where there's several hundred kilobytes of inline JS and CSS before any body text.
marginalia_nu 3y ago
A search engine doesn't index the HTML code. You're looking at a few Kb per document. You also don't need multiple historical snapshots of the document like WM retains. So you're looking at maybe 20 bn docs, 4 Kb each. 100 Tb, before compression.
- fauigerzigerk 3y ago
  True, but then Google doesn't just download the page source and index that. They run JavaScript in some cases to get to the actual content. This must come at a significant cost. Their index is enormous as well: "The Google Search index contains hundreds of billions of web pages and is well over 100,000,000 gigabytes in size." https://www.google.com/intl/en_uk/search/howsearchworks/how-search-works/organizing-information https://www.google.com/intl/en_uk/search/howsearchworks/how-... Doesn't mean you have to be as big as Google to do something useful of course.
  - marginalia_nu 3y ago
    Sure, download and run the javascript, but then you can snapshot the DOM, grab the text, and discard all the rest. The HTML and js is of little practical value for the index after that point. Google's index is likely very large because they don't have any real economic incentives to keeping it small.
    - fauigerzigerk 3y ago
      >... but then you can snapshot the DOM, grab the text, and discard all the rest Yes, absolutely, I didn't mean to imply otherwise. But first you have to figure out what you can discard beyond the HTML tags themselves to avoid indexing all the garbage that is on each and every page. When I tried to do this I came to the conclusion that I needed to actually render the page to find out where on the page a particular piece of text was, what font size it had, if it was even visible, etc. And then there's JavaScript of course. So what I'm saying is that storing a couple of kilobytes is probably not the most costly part of indexing a page.
      - marginalia_nu 3y ago
        You don't need to store it indefinitely though, and there's not much point in crawling faster than you can process the data. The couple of kilobytes per document is the actual storage footprint. Sure you need to massage the data, but that almost entirely CPU bound. You also need a lot of RAM for keeping the hot parts of the index.
      - akiselev 3y ago
        > When I tried to do this I came to the conclusion that I needed to actually render the page to find out where on the page a particular piece of text was, what font size it had, if it was even visible, etc. And then there's JavaScript of course. Are there open source projects devoted to this functionality? It’s becoming more and more a sticking point for working with LLMs. Grabbing the text without navigation and other crap but while maintaining formatting and links, etc
- DeathArrow 3y ago
  You don't index HTML code, but you have to process HTML and eventually run Javascript to get the text content. Then you have to compute the word frequencies.And that means you have to use more compute power.
leoedin 3y ago
This seems like a good use case for some sort of spam filtering. Maybe the web is that big, but what percentage of it is data someone will ever care about? I wonder if you could make a good-enough search with agressive filtering of pages before they enter the index.
DeathArrow 3y ago
>so maybe it'll cost lower to build a search engine index compared to an internet preservation project? An index is just hashmap of words and list of urls. So you have to parse the page, and add the urls and word frequencies to the list. In terms of storage is cheaper, in terms of computing power is more expensive.
- marginalia_nu 3y ago
  A hash table is not a good backing structure for a search engine. Hash tables almost guarantee worst case disk read patterns. You use something like a skip list or a b-tree, since that's makes much better use of the hardware, and on top of that allows you to do incredibly fast joins.
  - DeathArrow 3y ago
    That's true, but I was referring to a hash table as a mere mental model, not an actual implementation. A better name would be a dictionary instead of hash table.
    - fomine3 3y ago
      Perl culture here.

DeathArrow 3y ago

> Internet archive snapshot the whole page while google only index the first few kb of text data Is that a good way to build an index?

ricardo81 3y ago

>100kb max per page? They changed that limit somewhere in the mid-2000s. Just as well, there's some CMS's out there where there's several hundred kilobytes of inline JS and CSS before any body text.

marginalia_nu 3y ago

A search engine doesn't index the HTML code. You're looking at a few Kb per document. You also don't need multiple historical snapshots of the document like WM retains. So you're looking at maybe 20 bn docs, 4 Kb each. 100 Tb, before compression.

fauigerzigerk 3y ago
True, but then Google doesn't just download the page source and index that. They run JavaScript in some cases to get to the actual content. This must come at a significant cost. Their index is enormous as well: "The Google Search index contains hundreds of billions of web pages and is well over 100,000,000 gigabytes in size." https://www.google.com/intl/en_uk/search/howsearchworks/how-search-works/organizing-information https://www.google.com/intl/en_uk/search/howsearchworks/how-... Doesn't mean you have to be as big as Google to do something useful of course.
- marginalia_nu 3y ago
  Sure, download and run the javascript, but then you can snapshot the DOM, grab the text, and discard all the rest. The HTML and js is of little practical value for the index after that point. Google's index is likely very large because they don't have any real economic incentives to keeping it small.
  - fauigerzigerk 3y ago
    >... but then you can snapshot the DOM, grab the text, and discard all the rest Yes, absolutely, I didn't mean to imply otherwise. But first you have to figure out what you can discard beyond the HTML tags themselves to avoid indexing all the garbage that is on each and every page. When I tried to do this I came to the conclusion that I needed to actually render the page to find out where on the page a particular piece of text was, what font size it had, if it was even visible, etc. And then there's JavaScript of course. So what I'm saying is that storing a couple of kilobytes is probably not the most costly part of indexing a page.
    - marginalia_nu 3y ago
      You don't need to store it indefinitely though, and there's not much point in crawling faster than you can process the data. The couple of kilobytes per document is the actual storage footprint. Sure you need to massage the data, but that almost entirely CPU bound. You also need a lot of RAM for keeping the hot parts of the index.
    - akiselev 3y ago
      > When I tried to do this I came to the conclusion that I needed to actually render the page to find out where on the page a particular piece of text was, what font size it had, if it was even visible, etc. And then there's JavaScript of course. Are there open source projects devoted to this functionality? It’s becoming more and more a sticking point for working with LLMs. Grabbing the text without navigation and other crap but while maintaining formatting and links, etc
DeathArrow 3y ago
You don't index HTML code, but you have to process HTML and eventually run Javascript to get the text content. Then you have to compute the word frequencies.And that means you have to use more compute power.

leoedin 3y ago

This seems like a good use case for some sort of spam filtering. Maybe the web is that big, but what percentage of it is data someone will ever care about? I wonder if you could make a good-enough search with agressive filtering of pages before they enter the index.

DeathArrow 3y ago

>so maybe it'll cost lower to build a search engine index compared to an internet preservation project? An index is just hashmap of words and list of urls. So you have to parse the page, and add the urls and word frequencies to the list. In terms of storage is cheaper, in terms of computing power is more expensive.

marginalia_nu 3y ago
A hash table is not a good backing structure for a search engine. Hash tables almost guarantee worst case disk read patterns. You use something like a skip list or a b-tree, since that's makes much better use of the hardware, and on top of that allows you to do incredibly fast joins.
- DeathArrow 3y ago
  That's true, but I was referring to a hash table as a mere mental model, not an actual implementation. A better name would be a dictionary instead of hash table.
  - fomine3 3y ago
    Perl culture here.