10 ms·

Writing your own search engine is hard (2004)

161 points by georgehill 4y ago

t_mann 4y ago
Would be interesting to see stats from that time how many people were working on search engines and how it turned out for them. Did they end up getting acquired, at least funded for a while, exited, or just bootstrapped themselves until they realized there'll only be one winner.
Xeoncross 4y ago
Yeah, there are certainly more problems these days. For one, the size of the web is larger, more of it is spam causing issues with pure page rank to detect networks that heavily link to each other. Important sites have a bunch of anti-crawling detection set up (especially news sites). It's even worse that the best user-generated content is behind walled gardens in facebook groups, slack channels, quora threads, etc... The rest of the good sites are javascript-heavy and you often have to run chrome headless to render the page and find the content - but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. On the upside, https://commoncrawl.org/ https://commoncrawl.org/ now exists and makes the prototype crawling work much easier. It's not the full internet, but gives you plenty to work with and test against so you can skip to the part where you figure out if you can produce anything useful should you actually try to crawl the whole internet.
- nonrandomstring 4y ago
  > but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. The established idea of a "search engine" seems stuck, limited and based on some 90's technology that worked on a 90's web that no longer exists. Surely after 30 years we can build some kind of content discovery layer on top of what's out there?
  - Xeoncross 4y ago
    https://blogsurf.io/ https://blogsurf.io/ is an example of a small search engine that just stuck to a directory of known blogs instead of indexing the big sites or randomly crawling the web and ending up with mostly gibberish pages from all the spam sites.
    - mannyistyping 4y ago
      thank you for sharing this! I read through the site's about and I really enjoy how the creator wanted to stick to a specific area for quality over quantity.
  - jeffbee 4y ago
    I think that is not what they mean. I think what they meant is the site will detect your headless robot and serve it good content, while serving spam and malware to everyone else. The crawlers need their own distributed networks of unrelated addresses to prevent or detect this behavior.
  - thanksgiving 4y ago
    > Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source. I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...
    - ALittleLight 4y ago
- ArrayBoundCheck 4y ago
  I don't know how people can use the data. There's so much of it! I don't see any harddrives that are 80TB. It seems like people would need some kind of raid setup that can handle 200+TB of uncompressed data
  - Xeoncross 4y ago
    You don't need to download the whole thing. You can parse the WARC files from S3 to only extract the information you want (like pages with content). It's a lot smaller when you only keep the links and text.
  - francoismassot 4y ago
    A search index is often made of smaller independent pieces often called segments. So you can download & process progressively the data locally and upload it to an object storage. And run queries on it. That's what we did here for this project: https://quickwit.io/blog/commoncrawl https://quickwit.io/blog/commoncrawl Also an interesting blog post here: https://fulmicoton.com/posts/commoncrawl/ https://fulmicoton.com/posts/commoncrawl/
    - marginalia_nu 4y ago
      Huh, that's pretty expensive. For comparison, my search engine has an operational cost of ~$50/mo.
- Xeoncross 4y ago
  There isn't a one-size-fits all approach, but I've never worked on a project that encompasses as many computer science algorithms as a search engine. - Tries (patricia, radix, etc...) - Trees (b-trees, b+trees, merkle trees, log-structured merge-tree, etc..) - Consensus (raft, paxos, etc..) - Block storage (disk block size optimizations, mmap files, delta storage, etc..) - Probabilistic filters (hyperloloog, bloom filters, etc...) - Binary Search (sstables, sorted inverted indexes) - Ranking (pagerank, tf/idf, bm25, etc...) - NLP (stemming, POS tagging, subject identification, etc...) - HTML (document parsing/lexing) - Images (exif extraction, removal, resizing / proxying, etc...) - Queues (SQS, NATS, Apollo, etc...) - Clustering (k-means, density, hierarchical, gaussian distributions, etc...) - Rate limiting (leaky bucket, windowed, etc...) - text processing (unicode-normalization, slugify, sanitation, lossless and lossy hashing like metaphone and document fingerprinting) - etc... I'm sure there is plenty more I've missed. There are lots of generic structures involved like hashes, linked-lists, skip-lists, heaps and priority queues and this is just to get 2000's level basic tech. - https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy - https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic - https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx - https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch - https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve - https://github.com/thomasjungblut/go-sstables https://github.com/thomasjungblut/go-sstables A lot of people new to this space mistakenly think you can just throw elastic search or postgres fulltext search in front of terabytes of records and have something decent. That might work for something small like a curated collection of a few hundred sites.
  - kreeben 4y ago
    Yes, yes, yes :D There are so many topics in this space that are so interesting it's like a dream. I would add to your list - sentiment analysis - roaring bitmaps - compression - applied linear algebra - ai In a vent diagram intersecting all of these topics, is search. Coding a search engine from scratch is a beautiful way to spend ones days, if you're into programming.
  - boyter 4y ago
    > That might work for something small like a curated collection of a few hundred sites. Probably more like a few million but otherwise 100% true. Once you really need to scale you have to start losing some accuracy or correctness. It helps that the goal of a search engine is not to find all the results but instead delight the user by finding the things they want.
- deleted 4y ago
  [deleted]
- speedgoose 4y ago
  Are the Google crawling bots using similar IP ranges than Google Cloud?
- tim-- 4y ago
  Even though Common Crawl exists, if you wanted to get the data locally to have a bit of a poke around, it's practically impossible. The next search engine will never be made by a few kids playing around in the shed. The latest Common Crawl is roughly 120TB, enough to fit on 48 LTO-6 tapes. LTO-6 is the perfect middle ground expense-wise if you did want to play with the entire dataset at home. The tapes cost ~$30 (AUD) each, a cheap loader from eBay can be had for around ~$1000-4000 (like a Dell PowerVault or Fujitsu Eternus). Your looking at around $8k just to have your data close to your computer. Either that, or you are going to just run everything on AWS/Google Cloud/Azure. Two of these are your competitors. No one is going to copy 48 tapes for you, either :) 120TB would take around three months to download - if my ISP didn't cut me off first for probably being the heaviest consumer user of bandwidth. https://www.chatnoir.eu/ https://www.chatnoir.eu/ has a search engine that is built from the CommonCrawl data running on Elasticsearch, and it runs on 130 nodes [0]. I would love to be able to run my own search engine, but unless there are a number of breakthrough algorithms, I don't see how it can be easily achieved. I did ask some guys from the Internet Archive if they would be happy to copy some data for me onto some tapes. I wonder if there is a service for this? [0] https://webis.de/downloads/publications/papers/bevendorff_2018.pdf https://webis.de/downloads/publications/papers/bevendorff_20...
  - cosmolev 4y ago
    Why dealing with LTO-6 tapes? Isn't it easier to deploy 8 x 16TB HDDs?
    - tim-- 4y ago
      Tapes are cheaper when sneaker footing the data around.
  - marginalia_nu 4y ago
    I've sort of mostly dismissed common crawl for the same reason when building my search engine. It's simply too unwieldy. It's far easier (and cheaper) to do my own crawling at a manageable scale, than it is to work with CC's datasets. I'm also not entirely sure which problem CC is intending to solve. It's not like their data is in any way more complete than my own crawl sets, what I can't access to crawl, they don't seem to crawl either.
ldjkfkdsjnv 4y ago
Theory I have: Text search on the web will slowly die. People will search video based content, and use the fact that a human spoke the information, as well as comments/upvotes to vet it as trustworthy material. Google search as we know it will slowly die, and then will decline like Facebook. TikTok will steal search marketshare as their video clips span all of human life.
- xnx 4y ago
  Returning text results in response to queries will continue to decline in favor of returning answers and synthesized responses directly. I don't want Google to point me to a page that contains the answer somewhere, when it could provide me an even better summary based on thousands of related pages it has read.
  - ldjkfkdsjnv 4y ago
    Right but the main flaw with Google is people increasingly dont trust the result whether it is synthesized or not. And Google is in the adversarial position of wanting to censor certain answers as well as present answers that maximize their own revenue. An answer (like video based TikTok), will arise and crush them eventually.
wizofaus 4y ago
Doesn't mention the hardest part I found when developing a crawler - dealing with pages whose content is mostly dynamic and generated client side (SPA's). Even using V8 it's hard to do reliably and performantly at scale.
- sanjayts 4y ago
  > Doesn't mention the hardest part ... dealing with pages whose content is mostly dynamic and generated client side (SPA's) Given this is from 2004 I'm not surprised.
  - wizofaus 4y ago
    That was about when I was writing my crawler (not for search but for rules-based analysis). Even in 2004 a lot of key DOM elements were created/modified client side.
    - wizofaus 4y ago
      Though I do remember now that we solved it by having a separate mechanism for accessing pages that required logging in or had significant client-side rendering by allowing the user to record a macro that was played back in a headless browser. Within a few years though it was obvious a crawler would need to be able to automatically handle client scripts.
    - deleted 4y ago
      [deleted]
- marginalia_nu 4y ago
  Is this an actual problem, though? It seems like not even Google deals well with these types of pages. Often those types of pages are difficult to link to as well, as they're often highly stateful, applications rather than documents, and what you're seeing may not be what you get when you click the link. Not really what you want in a search engine.
  - 4y ago
wolfgang42 4y ago
I’ve been puttering away at making a search engine of my own (I should really do a Show HN sometime); let’s see how my experience compares with 18 years ago: Bandwidth: This is now also cheap; my residential service is 1 Gbit. However, the suggestion to wait until you’ve got indexing working well before optimizing crawling is IMO still spot-on; trying to make a polite, performant crawler that can deal with all the bizzare edge cases (https://memex.marginalia.nu/log/32-bot-apologetics.gmi https://memex.marginalia.nu/log/32-bot-apologetics.gmi) on the Web will drag you down. (I bypassed this problem by starting with the Stack Exchange data dumps and Wikipedia crawls, which are a lot more consistent than trying to deal with random websites.) CPU: Computers are really fast now; I’m using a 2-core computer from 2014 and it does what I need just fine. Disk: SATA is the new thing now, of course, but the difference these days is HDD vs SSD. SSD is faster: but you can design your architecture so that this mostly doesn’t matter, and even a “slow” HDD will be running at capacity. (The trick is to do linear streaming as much as possible, and avoid seeks at all costs.) Still, it’s probably a good idea to store your production index on an SSD, and it’s useful for intermediate data as well; by happenstance more than design I have a large HDD and a small SSD and they balance each other nicely. Storing files: 100% agree with this section, for the disk-seek reasons I mention above. Also, pages from the same website often compress very well against each other (since they’re using the same templates, large chunks of HTML can be squished down considerably), so if you’re pressed for space consider storing one GZIPped file per domain. (The tradeoff with zipping is that you can’t arbitrarily seek, but ideally you’ve designed things so you don’t need to do that anyway.) Also, WARC is a standard file format that has a lot of tooling for this exact use case. Networking: I skipped this by just storing everything on one computer; I expect to be able to continue doing this for a long time, since vertical scaling can get you very far these days. Indexing: You basically don’t need to write anything to get started with this these days! I’m just using bog-standard Elasticsearch with some glue code to do html2text; it’s working fine and took all of an afternoon to set up from scratch. (That said, I’m not sure I’ll continue using Elastic: it has a ton of features I don’t need, which makes it very hard to understand and work with since there’s so much that’s irrelevant to me. I’m probably going to switch to either straight Lucene or Bleve soon.) Page rank: I added pagerank very early on in the hopes that it would improve my results, and I’m not really sure how helpful it is if your results aren’t decent to begin with. However, the march of Moore’s law has made it an easy experiment: what Page and Brin’s server could compute in a week with carefully optimized C code, mine can do in less than 5 minutes (!) with a bit of JavaScript. Serving: Again, ElasticSearch will solve this entire problem for you (at least to start with); all your frontend has to do is take the JSON result and poke it into an HTML template. It’s easier than ever to start building a search engine in your own home; the recent explosion of such services (as seen on HN) is an indicator of the feasibility, and the rising complaints about Google show that the demand is there. Come and join us, the water’s fine!
boyter 4y ago
Glad to see this on the front page. One of those posts I reread every now and then. Better yet it’s written by Anna Patterson, who in addition to the mentioned searches at the bottom wrote chunks of Cuil (interesting even if it failed) and works on parts of Googles index both before Cuil and I think now. Sadly it’s a little out of date. I’d love to see a more modern post by someone. Perhaps the authors of mojeek, right dao or someone Elise running their own custom index. Heck I’d pay for some by Matt Wells of Gigablast or those behind Blekko. The whole space is so secretive that for those really interested in the space only crumbs of information are ever really released. If you are into this space or just curious the videos about bitfunnel which forms parts of the bing index are an excellent watch https://www.youtube.com/watch?v=1-Xoy5w5ydM https://www.youtube.com/watch?v=1-Xoy5w5ydM and https://www.clsp.jhu.edu/events/mike-hopcroft-microsoft/#.YT_6UC0Rpf0 https://www.clsp.jhu.edu/events/mike-hopcroft-microsoft/#.YT...
- ColinHayhurst 4y ago
  Thanks for the mention @boyter. Maybe we should write something like "Why Writing Your Own Search Engine to Index Billions of Pages is Hard". It would make interesting reading about challenges we have overcome. For us it's not being secretive as much as finding bandwidth. Getting awareness is a massive challenge too, which is why we've written a lot more content in the last two years. I'm sure you are looking for something more meaty than this: https://blog.mojeek.com/2021/05/no-tracking-search-how-does-it-work.html https://blog.mojeek.com/2021/05/no-tracking-search-how-does-...
deleted 4y ago
[deleted]
amelius 4y ago
The article doesn't touch upon the hardest and most interesting part: NLP and finding the most relevant results. I would like to see a post on this.
- frozencell 4y ago
  - https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/ - https://blogs.bing.com/search-quality-insights/september-2020/Introducing-the-next-wave-of-AI-at-Scale-innovations-in-Bing https://blogs.bing.com/search-quality-insights/september-202...
streets1627 4y ago
Hey folks, I am one of the co-founders of neeva.com While writing a search engine is hard, it is also incredibly rewarding. Over the past two years, we have brought up a meaningful crawl / index / serve pipeline for Neeva. Being able to create pages like https://neeva.com/search?q=tomato%20soup https://neeva.com/search?q=tomato%20soup or https://neeva.com/search?q=golang+struct+split https://neeva.com/search?q=golang+struct+split which are so much better than what is out there in commercial search engines is so worth it. We are private, ads free and customer paid.
- kbyatnal 4y ago
  any plans for an engineering blog? sounds like you and the team would have some interesting additions to the post!
  - marginalia_nu 4y ago
    They've been doing twitter threads. A bit hit and miss, but sometimes they've been pretty interesting. https://twitter.com/Neeva/status/1541506420318822406 https://twitter.com/Neeva/status/1541506420318822406 https://twitter.com/Neeva/status/1547293389539852288 https://twitter.com/Neeva/status/1547293389539852288
    - vivekraghunatha 4y ago
      (Other co-founder of Neeva) We wrote an op-ed last month in Fast Company re: the challenges of crawling at scale in today's world: https://www.fastcompany.com/90759792/with-google-dominating-search-the-internet-needs-crawl-neutrality https://www.fastcompany.com/90759792/with-google-dominating-... tldr; there are two big challenges in crawling the web: * quantity -- how do you crawl the web at O(B) pages per day * quality -- how do you make sure you are crawling the high quality parts On the quantity side, the web is a much trickier place to crawl than 10-15 years ago. Even if you build a system that is well behaved, respects rate limits and work w/ webmasters and CDNs to make sure you are treated as a good bot, the following things will bite you: * some sites only allow googlebot/bingbot via robots * even when sites allow all search crawlers, you'll see 429s, 503s, crawl delay directives that limit you to very low qps, and other mysterious errors. (most retailers and aggregators) * working w/ webmasters works only if you manage to get a response from them (good luck) * many sites require JS rendering (which is 100x more expensive in terms of the number of assets you are crawling) (OTOH, Cloudflare is a force for good: https://radar.cloudflare.com/verified-bots https://radar.cloudflare.com/verified-bots) On the quality side, crawl prioritization works best when you have click data, which most small search engines don't have enough of. In the absence of that, good seed sets and high quality inlinks are your friend. In other news, eng blogs are being written up as we speak. Will post here on HN as they roll out. Also, like marginalia_nu pointed out, we share whatever we've been working on on a weekly basis on our Twitter account.
Rockjodd 4y ago
For those interested in search engines, recommend checking out Vespa.ai[1][2] - the engine behind several features at Yahoo. [1] https://vespa.ai/ https://vespa.ai/ [2] https://docs.vespa.ai/en/getting-started.html https://docs.vespa.ai/en/getting-started.html
jefftk 4y ago
This is pretty reasonable for 2004, but the problems have changed. Everything on that page is totally doable for a serious engineering team, and has been done many times. The real hard part is what they briefly touch on with: Don’t do page rank initially. Actually don’t do it at all. For this observation I risk being inundated with hate mail, but nonetheless don’t do page rank. If you four guys in your garage can’t get something decent-looking up without page rank, you’re not going to get anything decent up with page rank. Use the source, Luke—the HTML source, that is. Page rank is lengthy analysis of a global nature and will cause you to buy more machines and get bogged down on this one complicated step—this one factor in ranking. Start by exploiting everything else you can think of: Is the word in the title? Is it in bold? etc. Spend your time thinking about anything you can exploit and try it out. The web is full of sites that want to rank, since traffic makes money and appearing high in search results gets you traffic. Simple handling of HTML source is incredibly gameable, and while it might have worked okay on the web of 2004 it definitely is not enough now. It's you and your team verse an enormous number of SEO people.
- cratermoon 4y ago
  As best I recall, one of the first SEO tricks took advantage of simple algorithms that ranked pages based more or less on word frequency, with certain synonyms collapsed, eg "movies, films, flicks, and cinema" would all be consider synonyms.[1] It took exactly seven minutes for site authors to figure out keyword stuffing. 1. Synonym collapsing isn't even necessarily a good thing, As Clay Shirky noted in "Ontology is Overrated -- Categories, Links, and Tags", people who enjoy watching movies and people into cinema are different crowds. https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Overrated.pdf https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Over...
forgotmypw17 4y ago
This is from the "doesn't scale" quadrant, but if you are not confident that your bot will behave well, shouldn't you supervise everything it does closely until you become confident?
cratermoon 4y ago
This bring back memories. Pentiums. IDE vs SCSI. Yahoo & dmoz. 10 kilobytes per web page (Even HN is 34K). Colo facilities.
jillesvangurp 4y ago
The hardest part of building your own search engine is not that it is technically hard (it's actually pretty easy) but that the bar for success is just really high. There are a few existing engines and they offer their services for free. So, not only is the bar really high, you have no viable revenue model and this stuff gets expensive quickly. Or put differently, if you are going to replicate what existing search engines already do, you are probably not going to be as good initially and you are going to struggle making money. Fixing the money part is the hard part.
kragen 4y ago
Useful context for this is that Anna Patterson started the search engine company Cuil in 02008, four years after writing this article (when she was still at Google). Its results were bad enough that the "Cuil Theory" meme was launched on Reddit and Tumblr making fun of it: https://knowyourmeme.com/memes/sites/cuil-theory https://knowyourmeme.com/memes/sites/cuil-theory: > One Cuil = One level of abstraction away from the reality of a situation. > Example: You ask me for a Hamburger. > 1 Cuil: if you asked me for a hamburger, and I gave you a raccoon. > 2 Cuils: If you asked me for a hamburger, but it turns out I don't really exist. Where I was originally standing, a picture of a hamburger rests on the ground. > 3 Cuils: You awake as a hamburger. You start screaming only to have special sauce fly from your lips. The world is in sepia. > 4 Cuils: Why are we speaking German? A mime cries softly as he cradles a young cow. Your grandfather stares at you as the cow falls apart into patties. You awake only to see me with pickles for eyes, I am singing the song that gives birth to the universe. http://cuiltheory.wikidot.com/ http://cuiltheory.wikidot.com/ Two years later, in 02010, the founders shut down the search engine, laid off all the employees, sold its patents to Google, and became Google employees again. Now all that remains of Cuil is the Cuil Theory Wiki. I'd be really interested to see a retrospective on what went wrong. I guess writing your own search engine really is hard, but I'd like to know what turned out to be so much harder than they expected.
- SQueeeeeL 4y ago
  Wait, you just made me understand what this video was actually about after all this time. I felt a weird dreadful nostalgia just hearing it again https://youtu.be/nfdEdE96En0 https://youtu.be/nfdEdE96En0
  - kragen 4y ago
    This video is great, despite its absence of video content.

161 points by georgehill 4y ago

Would be interesting to see stats from that time how many people were working on search engines and how it turned out for them. Did they end up getting acquired, at least funded for a while, exited, or just bootstrapped themselves until they realized there'll only be one winner.

Xeoncross 4y ago

Yeah, there are certainly more problems these days. For one, the size of the web is larger, more of it is spam causing issues with pure page rank to detect networks that heavily link to each other. Important sites have a bunch of anti-crawling detection set up (especially news sites). It's even worse that the best user-generated content is behind walled gardens in facebook groups, slack channels, quora threads, etc... The rest of the good sites are javascript-heavy and you often have to run chrome headless to render the page and find the content - but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. On the upside, https://commoncrawl.org/ https://commoncrawl.org/ now exists and makes the prototype crawling work much easier. It's not the full internet, but gives you plenty to work with and test against so you can skip to the part where you figure out if you can produce anything useful should you actually try to crawl the whole internet.

nonrandomstring 4y ago
> but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. The established idea of a "search engine" seems stuck, limited and based on some 90's technology that worked on a 90's web that no longer exists. Surely after 30 years we can build some kind of content discovery layer on top of what's out there?
- Xeoncross 4y ago
  https://blogsurf.io/ https://blogsurf.io/ is an example of a small search engine that just stuck to a directory of known blogs instead of indexing the big sites or randomly crawling the web and ending up with mostly gibberish pages from all the spam sites.
  - mannyistyping 4y ago
    thank you for sharing this! I read through the site's about and I really enjoy how the creator wanted to stick to a specific area for quality over quantity.
- jeffbee 4y ago
  I think that is not what they mean. I think what they meant is the site will detect your headless robot and serve it good content, while serving spam and malware to everyone else. The crawlers need their own distributed networks of unrelated addresses to prevent or detect this behavior.
- thanksgiving 4y ago
  > Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source. I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...
  - ALittleLight 4y ago
ArrayBoundCheck 4y ago
I don't know how people can use the data. There's so much of it! I don't see any harddrives that are 80TB. It seems like people would need some kind of raid setup that can handle 200+TB of uncompressed data
- Xeoncross 4y ago
  You don't need to download the whole thing. You can parse the WARC files from S3 to only extract the information you want (like pages with content). It's a lot smaller when you only keep the links and text.
- francoismassot 4y ago
  A search index is often made of smaller independent pieces often called segments. So you can download & process progressively the data locally and upload it to an object storage. And run queries on it. That's what we did here for this project: https://quickwit.io/blog/commoncrawl https://quickwit.io/blog/commoncrawl Also an interesting blog post here: https://fulmicoton.com/posts/commoncrawl/ https://fulmicoton.com/posts/commoncrawl/
  - marginalia_nu 4y ago
    Huh, that's pretty expensive. For comparison, my search engine has an operational cost of ~$50/mo.
Xeoncross 4y ago
There isn't a one-size-fits all approach, but I've never worked on a project that encompasses as many computer science algorithms as a search engine. - Tries (patricia, radix, etc...) - Trees (b-trees, b+trees, merkle trees, log-structured merge-tree, etc..) - Consensus (raft, paxos, etc..) - Block storage (disk block size optimizations, mmap files, delta storage, etc..) - Probabilistic filters (hyperloloog, bloom filters, etc...) - Binary Search (sstables, sorted inverted indexes) - Ranking (pagerank, tf/idf, bm25, etc...) - NLP (stemming, POS tagging, subject identification, etc...) - HTML (document parsing/lexing) - Images (exif extraction, removal, resizing / proxying, etc...) - Queues (SQS, NATS, Apollo, etc...) - Clustering (k-means, density, hierarchical, gaussian distributions, etc...) - Rate limiting (leaky bucket, windowed, etc...) - text processing (unicode-normalization, slugify, sanitation, lossless and lossy hashing like metaphone and document fingerprinting) - etc... I'm sure there is plenty more I've missed. There are lots of generic structures involved like hashes, linked-lists, skip-lists, heaps and priority queues and this is just to get 2000's level basic tech. - https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy - https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic - https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx - https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch - https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve - https://github.com/thomasjungblut/go-sstables https://github.com/thomasjungblut/go-sstables A lot of people new to this space mistakenly think you can just throw elastic search or postgres fulltext search in front of terabytes of records and have something decent. That might work for something small like a curated collection of a few hundred sites.
- kreeben 4y ago
  Yes, yes, yes :D There are so many topics in this space that are so interesting it's like a dream. I would add to your list - sentiment analysis - roaring bitmaps - compression - applied linear algebra - ai In a vent diagram intersecting all of these topics, is search. Coding a search engine from scratch is a beautiful way to spend ones days, if you're into programming.
- boyter 4y ago
  > That might work for something small like a curated collection of a few hundred sites. Probably more like a few million but otherwise 100% true. Once you really need to scale you have to start losing some accuracy or correctness. It helps that the goal of a search engine is not to find all the results but instead delight the user by finding the things they want.
deleted 4y ago
[deleted]
speedgoose 4y ago
Are the Google crawling bots using similar IP ranges than Google Cloud?
tim-- 4y ago
Even though Common Crawl exists, if you wanted to get the data locally to have a bit of a poke around, it's practically impossible. The next search engine will never be made by a few kids playing around in the shed. The latest Common Crawl is roughly 120TB, enough to fit on 48 LTO-6 tapes. LTO-6 is the perfect middle ground expense-wise if you did want to play with the entire dataset at home. The tapes cost ~$30 (AUD) each, a cheap loader from eBay can be had for around ~$1000-4000 (like a Dell PowerVault or Fujitsu Eternus). Your looking at around $8k just to have your data close to your computer. Either that, or you are going to just run everything on AWS/Google Cloud/Azure. Two of these are your competitors. No one is going to copy 48 tapes for you, either :) 120TB would take around three months to download - if my ISP didn't cut me off first for probably being the heaviest consumer user of bandwidth. https://www.chatnoir.eu/ https://www.chatnoir.eu/ has a search engine that is built from the CommonCrawl data running on Elasticsearch, and it runs on 130 nodes [0]. I would love to be able to run my own search engine, but unless there are a number of breakthrough algorithms, I don't see how it can be easily achieved. I did ask some guys from the Internet Archive if they would be happy to copy some data for me onto some tapes. I wonder if there is a service for this? [0] https://webis.de/downloads/publications/papers/bevendorff_2018.pdf https://webis.de/downloads/publications/papers/bevendorff_20...
- cosmolev 4y ago
  Why dealing with LTO-6 tapes? Isn't it easier to deploy 8 x 16TB HDDs?
  - tim-- 4y ago
    Tapes are cheaper when sneaker footing the data around.
- marginalia_nu 4y ago
  I've sort of mostly dismissed common crawl for the same reason when building my search engine. It's simply too unwieldy. It's far easier (and cheaper) to do my own crawling at a manageable scale, than it is to work with CC's datasets. I'm also not entirely sure which problem CC is intending to solve. It's not like their data is in any way more complete than my own crawl sets, what I can't access to crawl, they don't seem to crawl either.

ldjkfkdsjnv 4y ago

Theory I have: Text search on the web will slowly die. People will search video based content, and use the fact that a human spoke the information, as well as comments/upvotes to vet it as trustworthy material. Google search as we know it will slowly die, and then will decline like Facebook. TikTok will steal search marketshare as their video clips span all of human life.

xnx 4y ago
Returning text results in response to queries will continue to decline in favor of returning answers and synthesized responses directly. I don't want Google to point me to a page that contains the answer somewhere, when it could provide me an even better summary based on thousands of related pages it has read.
- ldjkfkdsjnv 4y ago
  Right but the main flaw with Google is people increasingly dont trust the result whether it is synthesized or not. And Google is in the adversarial position of wanting to censor certain answers as well as present answers that maximize their own revenue. An answer (like video based TikTok), will arise and crush them eventually.

wizofaus 4y ago

Doesn't mention the hardest part I found when developing a crawler - dealing with pages whose content is mostly dynamic and generated client side (SPA's). Even using V8 it's hard to do reliably and performantly at scale.

sanjayts 4y ago
> Doesn't mention the hardest part ... dealing with pages whose content is mostly dynamic and generated client side (SPA's) Given this is from 2004 I'm not surprised.
- wizofaus 4y ago
  That was about when I was writing my crawler (not for search but for rules-based analysis). Even in 2004 a lot of key DOM elements were created/modified client side.
  - wizofaus 4y ago
    Though I do remember now that we solved it by having a separate mechanism for accessing pages that required logging in or had significant client-side rendering by allowing the user to record a macro that was played back in a headless browser. Within a few years though it was obvious a crawler would need to be able to automatically handle client scripts.
  - deleted 4y ago
    [deleted]
marginalia_nu 4y ago
Is this an actual problem, though? It seems like not even Google deals well with these types of pages. Often those types of pages are difficult to link to as well, as they're often highly stateful, applications rather than documents, and what you're seeing may not be what you get when you click the link. Not really what you want in a search engine.
- 4y ago

wolfgang42 4y ago

I’ve been puttering away at making a search engine of my own (I should really do a Show HN sometime); let’s see how my experience compares with 18 years ago: Bandwidth: This is now also cheap; my residential service is 1 Gbit. However, the suggestion to wait until you’ve got indexing working well before optimizing crawling is IMO still spot-on; trying to make a polite, performant crawler that can deal with all the bizzare edge cases (https://memex.marginalia.nu/log/32-bot-apologetics.gmi https://memex.marginalia.nu/log/32-bot-apologetics.gmi) on the Web will drag you down. (I bypassed this problem by starting with the Stack Exchange data dumps and Wikipedia crawls, which are a lot more consistent than trying to deal with random websites.) CPU: Computers are really fast now; I’m using a 2-core computer from 2014 and it does what I need just fine. Disk: SATA is the new thing now, of course, but the difference these days is HDD vs SSD. SSD is faster: but you can design your architecture so that this mostly doesn’t matter, and even a “slow” HDD will be running at capacity. (The trick is to do linear streaming as much as possible, and avoid seeks at all costs.) Still, it’s probably a good idea to store your production index on an SSD, and it’s useful for intermediate data as well; by happenstance more than design I have a large HDD and a small SSD and they balance each other nicely. Storing files: 100% agree with this section, for the disk-seek reasons I mention above. Also, pages from the same website often compress very well against each other (since they’re using the same templates, large chunks of HTML can be squished down considerably), so if you’re pressed for space consider storing one GZIPped file per domain. (The tradeoff with zipping is that you can’t arbitrarily seek, but ideally you’ve designed things so you don’t need to do that anyway.) Also, WARC is a standard file format that has a lot of tooling for this exact use case. Networking: I skipped this by just storing everything on one computer; I expect to be able to continue doing this for a long time, since vertical scaling can get you very far these days. Indexing: You basically don’t need to write anything to get started with this these days! I’m just using bog-standard Elasticsearch with some glue code to do html2text; it’s working fine and took all of an afternoon to set up from scratch. (That said, I’m not sure I’ll continue using Elastic: it has a ton of features I don’t need, which makes it very hard to understand and work with since there’s so much that’s irrelevant to me. I’m probably going to switch to either straight Lucene or Bleve soon.) Page rank: I added pagerank very early on in the hopes that it would improve my results, and I’m not really sure how helpful it is if your results aren’t decent to begin with. However, the march of Moore’s law has made it an easy experiment: what Page and Brin’s server could compute in a week with carefully optimized C code, mine can do in less than 5 minutes (!) with a bit of JavaScript. Serving: Again, ElasticSearch will solve this entire problem for you (at least to start with); all your frontend has to do is take the JSON result and poke it into an HTML template. It’s easier than ever to start building a search engine in your own home; the recent explosion of such services (as seen on HN) is an indicator of the feasibility, and the rising complaints about Google show that the demand is there. Come and join us, the water’s fine!

boyter 4y ago

Glad to see this on the front page. One of those posts I reread every now and then. Better yet it’s written by Anna Patterson, who in addition to the mentioned searches at the bottom wrote chunks of Cuil (interesting even if it failed) and works on parts of Googles index both before Cuil and I think now. Sadly it’s a little out of date. I’d love to see a more modern post by someone. Perhaps the authors of mojeek, right dao or someone Elise running their own custom index. Heck I’d pay for some by Matt Wells of Gigablast or those behind Blekko. The whole space is so secretive that for those really interested in the space only crumbs of information are ever really released. If you are into this space or just curious the videos about bitfunnel which forms parts of the bing index are an excellent watch https://www.youtube.com/watch?v=1-Xoy5w5ydM https://www.youtube.com/watch?v=1-Xoy5w5ydM and https://www.clsp.jhu.edu/events/mike-hopcroft-microsoft/#.YT_6UC0Rpf0 https://www.clsp.jhu.edu/events/mike-hopcroft-microsoft/#.YT...

ColinHayhurst 4y ago
Thanks for the mention @boyter. Maybe we should write something like "Why Writing Your Own Search Engine to Index Billions of Pages is Hard". It would make interesting reading about challenges we have overcome. For us it's not being secretive as much as finding bandwidth. Getting awareness is a massive challenge too, which is why we've written a lot more content in the last two years. I'm sure you are looking for something more meaty than this: https://blog.mojeek.com/2021/05/no-tracking-search-how-does-it-work.html https://blog.mojeek.com/2021/05/no-tracking-search-how-does-...

deleted 4y ago

[deleted]

amelius 4y ago

The article doesn't touch upon the hardest and most interesting part: NLP and finding the most relevant results. I would like to see a post on this.

frozencell 4y ago
- https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/ - https://blogs.bing.com/search-quality-insights/september-2020/Introducing-the-next-wave-of-AI-at-Scale-innovations-in-Bing https://blogs.bing.com/search-quality-insights/september-202...

streets1627 4y ago

Hey folks, I am one of the co-founders of neeva.com While writing a search engine is hard, it is also incredibly rewarding. Over the past two years, we have brought up a meaningful crawl / index / serve pipeline for Neeva. Being able to create pages like https://neeva.com/search?q=tomato%20soup https://neeva.com/search?q=tomato%20soup or https://neeva.com/search?q=golang+struct+split https://neeva.com/search?q=golang+struct+split which are so much better than what is out there in commercial search engines is so worth it. We are private, ads free and customer paid.

kbyatnal 4y ago
any plans for an engineering blog? sounds like you and the team would have some interesting additions to the post!
- marginalia_nu 4y ago
  They've been doing twitter threads. A bit hit and miss, but sometimes they've been pretty interesting. https://twitter.com/Neeva/status/1541506420318822406 https://twitter.com/Neeva/status/1541506420318822406 https://twitter.com/Neeva/status/1547293389539852288 https://twitter.com/Neeva/status/1547293389539852288
  - vivekraghunatha 4y ago
    (Other co-founder of Neeva) We wrote an op-ed last month in Fast Company re: the challenges of crawling at scale in today's world: https://www.fastcompany.com/90759792/with-google-dominating-search-the-internet-needs-crawl-neutrality https://www.fastcompany.com/90759792/with-google-dominating-... tldr; there are two big challenges in crawling the web: * quantity -- how do you crawl the web at O(B) pages per day * quality -- how do you make sure you are crawling the high quality parts On the quantity side, the web is a much trickier place to crawl than 10-15 years ago. Even if you build a system that is well behaved, respects rate limits and work w/ webmasters and CDNs to make sure you are treated as a good bot, the following things will bite you: * some sites only allow googlebot/bingbot via robots * even when sites allow all search crawlers, you'll see 429s, 503s, crawl delay directives that limit you to very low qps, and other mysterious errors. (most retailers and aggregators) * working w/ webmasters works only if you manage to get a response from them (good luck) * many sites require JS rendering (which is 100x more expensive in terms of the number of assets you are crawling) (OTOH, Cloudflare is a force for good: https://radar.cloudflare.com/verified-bots https://radar.cloudflare.com/verified-bots) On the quality side, crawl prioritization works best when you have click data, which most small search engines don't have enough of. In the absence of that, good seed sets and high quality inlinks are your friend. In other news, eng blogs are being written up as we speak. Will post here on HN as they roll out. Also, like marginalia_nu pointed out, we share whatever we've been working on on a weekly basis on our Twitter account.

Rockjodd 4y ago

For those interested in search engines, recommend checking out Vespa.ai[1][2] - the engine behind several features at Yahoo. [1] https://vespa.ai/ https://vespa.ai/ [2] https://docs.vespa.ai/en/getting-started.html https://docs.vespa.ai/en/getting-started.html

jefftk 4y ago

This is pretty reasonable for 2004, but the problems have changed. Everything on that page is totally doable for a serious engineering team, and has been done many times. The real hard part is what they briefly touch on with: Don’t do page rank initially. Actually don’t do it at all. For this observation I risk being inundated with hate mail, but nonetheless don’t do page rank. If you four guys in your garage can’t get something decent-looking up without page rank, you’re not going to get anything decent up with page rank. Use the source, Luke—the HTML source, that is. Page rank is lengthy analysis of a global nature and will cause you to buy more machines and get bogged down on this one complicated step—this one factor in ranking. Start by exploiting everything else you can think of: Is the word in the title? Is it in bold? etc. Spend your time thinking about anything you can exploit and try it out. The web is full of sites that want to rank, since traffic makes money and appearing high in search results gets you traffic. Simple handling of HTML source is incredibly gameable, and while it might have worked okay on the web of 2004 it definitely is not enough now. It's you and your team verse an enormous number of SEO people.

cratermoon 4y ago
As best I recall, one of the first SEO tricks took advantage of simple algorithms that ranked pages based more or less on word frequency, with certain synonyms collapsed, eg "movies, films, flicks, and cinema" would all be consider synonyms.[1] It took exactly seven minutes for site authors to figure out keyword stuffing. 1. Synonym collapsing isn't even necessarily a good thing, As Clay Shirky noted in "Ontology is Overrated -- Categories, Links, and Tags", people who enjoy watching movies and people into cinema are different crowds. https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Overrated.pdf https://oc.ac.ge/file.php/16/_1_Shirky_2005_Ontology_is_Over...

forgotmypw17 4y ago

This is from the "doesn't scale" quadrant, but if you are not confident that your bot will behave well, shouldn't you supervise everything it does closely until you become confident?

cratermoon 4y ago

This bring back memories. Pentiums. IDE vs SCSI. Yahoo & dmoz. 10 kilobytes per web page (Even HN is 34K). Colo facilities.

jillesvangurp 4y ago

The hardest part of building your own search engine is not that it is technically hard (it's actually pretty easy) but that the bar for success is just really high. There are a few existing engines and they offer their services for free. So, not only is the bar really high, you have no viable revenue model and this stuff gets expensive quickly. Or put differently, if you are going to replicate what existing search engines already do, you are probably not going to be as good initially and you are going to struggle making money. Fixing the money part is the hard part.

kragen 4y ago

Useful context for this is that Anna Patterson started the search engine company Cuil in 02008, four years after writing this article (when she was still at Google). Its results were bad enough that the "Cuil Theory" meme was launched on Reddit and Tumblr making fun of it: https://knowyourmeme.com/memes/sites/cuil-theory https://knowyourmeme.com/memes/sites/cuil-theory: > One Cuil = One level of abstraction away from the reality of a situation. > Example: You ask me for a Hamburger. > 1 Cuil: if you asked me for a hamburger, and I gave you a raccoon. > 2 Cuils: If you asked me for a hamburger, but it turns out I don't really exist. Where I was originally standing, a picture of a hamburger rests on the ground. > 3 Cuils: You awake as a hamburger. You start screaming only to have special sauce fly from your lips. The world is in sepia. > 4 Cuils: Why are we speaking German? A mime cries softly as he cradles a young cow. Your grandfather stares at you as the cow falls apart into patties. You awake only to see me with pickles for eyes, I am singing the song that gives birth to the universe. http://cuiltheory.wikidot.com/ http://cuiltheory.wikidot.com/ Two years later, in 02010, the founders shut down the search engine, laid off all the employees, sold its patents to Google, and became Google employees again. Now all that remains of Cuil is the Cuil Theory Wiki. I'd be really interested to see a retrospective on what went wrong. I guess writing your own search engine really is hard, but I'd like to know what turned out to be so much harder than they expected.

SQueeeeeL 4y ago
Wait, you just made me understand what this video was actually about after all this time. I felt a weird dreadful nostalgia just hearing it again https://youtu.be/nfdEdE96En0 https://youtu.be/nfdEdE96En0
- kragen 4y ago
  This video is great, despite its absence of video content.