So, I clearly need to write a blog post about this one day - there are so many answers and they won't all fit in an HN reply. So I'll give some higher level con

So, I clearly need to write a blog post about this one day - there are so many answers and they won't all fit in an HN reply. So I'll give some higher level concepts that I found helpful. Not to dodge your questions, but it gives a way to think about things. * Do Things That Don't Scale. There's a Paul Graham essay on this, but the general idea I took is that a naive, brute force solution will get you quite a long way, and you'll learn lots from it. PHP & SQLite on shared web hosting should never work, but, it goes SO much further than you'd think. * Wait Until Something Is A Problem. You'll eventually feel the parts that need scaling. Worry about them then. For a long time I didn't even have an inverted index, didn't even have FTS enabled on SQLite. Things still worked. (But you really do want an inverted index & full text search.) * Do You Actually Need This? You probably don't need to index the entire internet. You only want the parts of the internet that are relevant to you. You can use your browser's history to get a list of the websites / pages that are actually important to you. It is not that many. This is roughly the idea behind asciimoo's Hister project, which makes a local index of the internet while you browse the web. On headers & proxies & javascript etc... so far, I don't bother. If a website doesn't want me indexing them, I respect that. There's still large parts of the web that do want to be indexed. Read their robots.txt files so you can learn what websites would like you to do, especially Crawl-Delay. I have never used AWS, so I can't help there. My stuff is all just Apache, PHP & SQLite, and a bunch of self-made Go command-line programs. For further research, read all of the Marginalia Search blog, you'll learn lots from that. Remember Marginalia is run by just one person. Also look at Seirdy's list of Search Engines With Their Own Indexes. https://www.marginalia.nu/log/ https://www.marginalia.nu/log/ https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ https://seirdy.one/posts/2021/03/10/search-engines-with-own-...