11 ms·
Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of c
by marginalia_nu 5y ago
Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)
- santamex 5y agoWhich software do you use to index the sites?
- marginalia_nu 5y agoI wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke. A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays. So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that. Then it goes to the words file and looks up the URL range of the first word. Then it goes to the words file and looks up the URL range of the second word. Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word. Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result. I'm skipping over a few steps, but that's the very crudest of outlines.
- rvnx 5y agoIt's a great project!
- q3k 5y agoGood stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration. Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
- marginalia_nu 5y agoWell just crunching the numbers should indicate what is possible and what isn't. For the moment I have just south of 20 million URLs indexed. 1 x 20 million bytes = 20 Mb. 10 x 20 million bytes = 200 Mb. 100 x 20 million bytes = 2 Gb. 1,000 x 20 million bytes = 20 Gb. 10,000 x 20 million bytes = 200 Gb. 100,000 x 20 million bytes = 2 Tb. 1,000,000 x 20 million bytes = 20 Tb. This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data. How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?
- freediver 5y agoWhat crawler are you using and what kind of crawling speeds are you achieving? How do you rank the results (is it based on content only) or you have external factors too? What is your personal preferred search option of the 7 and why? Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
- marginalia_nu 5y ago> What crawler are you using and what kind of crawling speeds are you achieving? Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper. >How do you rank the results (is it based on content only) or you have external factors too? I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query. > What is your personal preferred search option of the 7 and why? I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement. > Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way. Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible. And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing. I'm honestly a bit at a loss for words.
- Aeolun 5y agoI’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?
- marginalia_nu 5y agoThe trick is that the list of URLs for each word already is in the URLs file. The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence. Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8. The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8. If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap. This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)
- bullen 5y agoHm, ok I understand more but how do you perform the "binary search", just loop over the URL ids? Funny I also selected "hello" and "world" above! Xo My system is also written in Java btw! Here are example results of my word search: http://root.rupy.se/node/data/word/four http://root.rupy.se/node/data/word/four http://root.rupy.se/node/data/word/only http://root.rupy.se/node/data/word/only etc.
- marginalia_nu 5y agoI'll get back to your email in a while, I've got a ton of messages I'm working through. But yeah, in short pseudocode: for url in range-for-"hello": if binary-search (range-for-"world", url): yield url I do use streams, but that is the bare essence of it.
- 5y ago
- bullen 5y agoInteresting, in my database (http://root.rupy.se http://root.rupy.se) I have one file per word that contains the ids (long) of the nodes (URLs), so to search many words together I have to go through the first file and one by one see if I find matches in the second. How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?
- axelroze 5y agoHi, Interesting idea. Definitely see an overlap with eReader markets and looking at text only contents. How does it work? It ignores pages on which it detects frameworks for ui and ads or any javascript code at all?
- bluefox 5y agoThis is a very cool project! Thank you.
- BugsJustFindMe 5y agoI love this, and I love (many of) the results so far! What I can't find on the site is detail about what "too many modern web design features" means. Is it just penalizing sites with tons of JavaScript?
- marginalia_nu 5y agoJavascript tags are penalized the hardest, but it also takes into consideration density of text per HTML. There's also some adjustments based on text length, which words occur in the page, etc.
- ad404b8a372f2b9 5y agoVery cool project! How many websites do you have in your index? And how did you go about building it? I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.
- marginalia_nu 5y agoAbout 21 million, and I'm crawling myself. Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text. I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.
- ad404b8a372f2b9 5y agoIf you're open to sharing your index I could make a classifier for you, I do this for a living. It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.
- asah 5y agoLove it, kudos! This is great for developers and others who Just Need Answers and not shopping or entertainment. If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get traction. If you're Serious(tm), dog into the publisher structure formats and format those, create a test suite, etc. A word of caution: if this takes off, as a business it's vulnerable to Google shifting its algorithms slightly to identify the segment of users+queries who prefer these results and give the same results to those queries. Hope this helps!
- marginalia_nu 5y agoIf Google starts showing interesting text-heavy links instead of vapid listicles and storefronts, I have accomplished everything I ever could dream of.
- 0xbadcafebee 5y agoThank you for doing this important work.
- palijer 5y agoHaha, reminds me exactly of this. https://xkcd.com/810/ https://xkcd.com/810/
- aaron5 5y agohaha, great answer! thanks for your work on this :)
- MaysonL 5y agoGoogle Info - for when you're looking for information, not shopping advice or lists!
- addandsubtract 5y agoGoogle info? Can you give me a sample query of what you mean?
- artembugara 5y agoNice, what are you using to crawl the web?
- marginalia_nu 5y agoIt's pretty much all bespoke. I use external libraries for parsing HTML (JSoup) and robots.txt; but that's about it.
- soheil 5y agoWhat was the starting site you fed to the crawler to follow the links from to build the index?
- marginalia_nu 5y agoJust my (swedish) personal website. The first iteration of the search engine was probably mainly seeded by these links: https://www.marginalia.nu/00-l%C3%A4nkar/ https://www.marginalia.nu/00-l%C3%A4nkar/ But I've since expanded my websites, so now I think these play a decent role in later iterations, although they are virtually all of them pages I've found eating my own dogfood: https://memex.marginalia.nu/links/fragments-old-web.gmi https://memex.marginalia.nu/links/fragments-old-web.gmi https://memex.marginalia.nu/links/bookmarks.gmi https://memex.marginalia.nu/links/bookmarks.gmi
- ColinHayhurst 5y agoGreat work. Working on an alternative search engine too. Take a look at my profile.
- blondin 5y agofantastic project, thank you!
- androceium 5y agoPretty neat!!! You may already be aware of this, but the page doesn't seem to be formatted correctly on mobile. The content shows in a single thin column in the middle.
- marginalia_nu 5y agoHmm, which OS? I only have a single Android phone so I've only fixed the CSS for that.
- ant6n 5y agoFor example Firefox on Android.
- androceium 5y agoI was seeing it on Android w/ Firefox. Seems like it's fixed now though. :)
- marginalia_nu 5y agoCurious, I haven't touched the stylesheets.
- khimaros 5y agoFennec F-Droid on Android 11 has some rendering issues.
- bityard 5y agoThis is awesome. I've been looking for a long time for a search engine that basically takes everything Google does and does the opposite. Thank you for doing this, I will definitely be bookmarking it. Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.
- Wronnay 5y agoI would also love to add some sites which I were missing... Unfortunately this doesn't seem to be a feature which new search engines are focusing on - Brave Search also misses that feature...
- crocodiletears 5y agoIt's very rare that I see a project on HN I can see myself using. This is one. Like others have said, the results can be a little rough. But they're rough in a way I think is much more manageable than the idiosynchrosies of more 'clever' search engines.
- marginalia_nu 5y agoI think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine. Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.
- crocodiletears 5y agoThis is exactly how I prefer to use my search engines.
- quaintdev 5y agoI searched like this all my life and always got expected results. But just a week ago I found out that these "how", "what" questions give better and faster results on Google.
- LeftHandPath 5y agoThat switch happened some years ago. I've been unlearning and relearning how to use google for what feels like at least three or four years now. The main pain-point, though, is that a lot of long-tail searches you could've used to find different results in years past, now seem to funnel you to the same set of results based on your apparent intent. At least, it has felt that way -- I'm not entirely sure how the modern google algorithm works.
- eitland 5y agoI realized this a few years ago when I observed my wife find things faster on Google than me. I appreciate that it is easier for newcomers but I still hate it personally after years and especially that they cannot even avoid meddling with my queries even when I try to accept the new system and use the verbatim option.
- agumonkey 5y agois there a json endpoint ? I'd love to make an emacs bridge :)
- maddyboo 5y agoSeconded, I’d like to incorporate it into a project of mine.
- habibur 5y agoHow are you doing the crawling without getting blocking? -- the hardest part.
- judge2020 5y agoNot OP but crawling is easy if you don't try scanning 5+ pages a second - almost all rate limiting/heuristic based 'keep server costs low' engines, including Cloudflare, don't care if you request every page, but will take action if you do something like burst every page and take up just as many server resources as a hundred concurrent users. Now, that is assuming you aren't on some VPS provider. If you're going to crawl, you'll have the best chance when you use your own IPs on your own ASN, with DNS and reverse DNS set up correctly. This makes it so the IP reputation systems can detect you as a crawler but not one that hammers every site it visits. Also, I imagine that, for a search engine like this, it doesn't expect content to change much anyways - so it can take its time crawling every site only once every month or two, instead of the multiple times a week (or day) search engines like Google have to for the constantly-updated content being churned out.
- eigengrau5150 5y agoI like this. Thanks for doing it.
- davegauer 5y agoThis is absolutely wonderful. I am LOVING the results I'm getting back from it: the sort of content-rich sites that have become nigh unreachable using traditional search engines. Thank you for building this!
- noduerme 5y agoI love this idea, and admire the work you put into it. I'm a fan of long reads and historical non-fiction, and Google's results are truly garbage. I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were: * A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also making the top search results with other wild claims, e.g. that the Egyptians discovered Arizona); * Another site claiming the first inhabitants of Australia were a lost tribe of Israel; * A third site claiming that Australia was discovered and founded by members of a secret society of Rosicrucians who had infiltrated the Dutch East India Company and planned to build an Australian utopia... These were all pages heavy with HTML4 tags and virtually devoid of Javascript, the kinds of pages you'd frequently see in the late 1990s from people who had built their own static websites in a text editor, or exported HTML from MS Word. At that time, there were millions of those sites with people paying for their own unique domain names, and so the proportion of them that were home to wild-eyed conspiracy theories was relatively small. What I think has happened is that kooks continued to keep these sites up - to the point where it's almost a visual trope now to see a red <h1> tag in Times New Roman and think, uh oh, I've stumbled on an "ancient aliens" site. Whereas scholars and journals offering higher quality information have moved to more modern platforms that rely more heavily on modern browsers - with or without their own domain names. So as a result what seemed to surface here were the fragments of the old web that remain live - possibly because people living in cabins in Montana forget to cancel their web hosting, or because the nature of old-school conspiracy theorists is to just keep packing their old sites with walls of text surrounded by <p> tags. Arguably, this seems to rank the way Google's engine used to, since it couldn't run JS and they wanted to punish sites that used code to change markup at render time. At least, when I used to have to do onsite SEO work, it was always about simple tag hierarchies. I wonder whether there isn't some better metric of validity and information quality than what markup is used. Some of the sites that surfaced further down could be considered interesting and valuable resources. I think not punishing simple wall-of-text content is a good thing. But to punish more complicated layouts may have the perverse effect of downranking higher-quality sources of information - i.e. people and organizations who can afford to build a decent website, or who care to migrate to a modern blogging platform.
- 5y ago
- scrollaway 5y agoI searched Warcraft and got a gold selling/ level boosting site. Some things never change :)
- soheil 5y agoAwesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?
- marginalia_nu 5y ago> How are you able to keep the site running after HN kiss of death? I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware. My day job is also developing this sort of highly performance java applications, I guess it helps. > What is your stack, elastic search or something simper? It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries. > How did you crawl so many websites for a project this size? It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains). > Did you use any APIs like duck duck go or data from other search engines? Nope, it's all me. > Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor? I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains. It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.
- edbaskerville 5y agoThis has amazing potential. I'd encourage you to form a non-profit, turn this into something that can last as an organization without becoming what you're trying to avoid becoming. This is a good enough start that I bet you could raise a sizeable startup fund very soon from a combination of crowdfunding and foundation grants—I bet the Sloan Foundation would love this!
- zizee 5y agoLove the idea. A little feedback: layout needs tweaking for mobile. FWIW: I'm on mobile Firefox for Android.
- peterburkimsher 5y agoThank you so much for creating such a useful search engine! Is there any way that you can get an HTTP certificate? I use an old iPhone 4S, and most of the modern web is inaccessible due to TLS. Hacker News and mbasic.facebook are two of the last sites I can use. Usually text-based sites are more accessible, so this could be really useful to help me continue using my antique devices!
- karlzt 5y agoI can't see the letters in the disturbing white search box, I'm on duckduckgo, brave, monocles, jquarks, smartcookieweb on Android.
- drpixie 5y agoReally like it, and love that you've done this yourself. I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator & converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals. Thanks.
- marginalia_nu 5y agoI've mostly added these converters and frills because I'm trying to eat my own dogfood, and google is deeply engrained as my go-to source for unit conversions and back-of-envelope calculations. Don't worry, this stuff is easy, and doesn't even remotely take away from the work on the harder problems.
- aoms 5y agoWhat's the tech stack?
- marginalia_nu 5y agoIt's custom java code, for the most part. I'm using mariadb for some ancillary information, but the index and the crawler and everything is written from scratch. It's hosted on my consumer-equipment server (Ryzen 3900x, 128Gb ram, Optane 900p+a few IronWolf drives), bare bones on Debian.
- agencies 5y agoI continue to want a communal set of 50-100 million urls and data that are "good" (i.e. for any value of good and more complete than common crawl) that are accessible enough to be easy to work with that can be used to experiment with different web search tech. There are enough separate problems to tackle in web search that breaking it down would maybe move the needle. We have lots of kaggle competitions about ranking, but using closed data. What other types of kaggle projects would help web search? What can we do to foster a sustainable bazaar of projects to make it easier to build web search engines?
- powerchord 5y agoThis is really cool! A spacer between the links would help my old eyes; I keep getting lost in which link goes with which description. :-)
- niutech 5y agoNice project, but have you heard of FrogFind? it also presents lightweight search results.