49 ms·

A search engine that favors text-heavy sites and punishes modern web design

• camillomiller 5y ago
  Great idea, awful UI
  • muxator 5y ago
    How so? It's intuitive and super fast. I whish there were more websites with such a simple UI.
    • marginalia_nu 5y ago
      Some people like a flashy UI, the modern look is important for them. It's ok to have aesthetic preferences, let's not pretend we don't all have them. In the end, it's a niche search engine I've made, the intended audience is the long tail. It just isn't for everyone, and if it was for everyone, it probably would be lesser for it.
      • ravenstine 5y ago
        I'm glad you aren't trying to please anyone. I'd like a return to an internet with fewer colors, gadgets and gizmos, custom fonts, TypeKit, JavaScript requirements, and so on. Most of the time I'm reading articles, so just give me more text and less fluff!
      • silent_cal 5y ago
        I like the UI.
    • feikname 5y ago
      it's too "uncompact". Font size too big and could use a bit more horizontal space. I find it comfortable to use at 60% zoom level
  • jbj 5y ago
    What makes you find the user interface awful? it is litterally a search website with a text box for a search term and a button to do the search.
  • stronglikedan 5y ago
    Ironic comment, considering that this is a search engine to weed out sites with awful UIs. This gives us exactly what we need in a search UI - no more, no less - in a clean and intuitive way.
  • fouc 5y ago
    Yeah the design could use some work. The search results are not compact - I only see 1 result without scrolling, not counting the related wikipedia link that apparently has no description. I don't particularly like that it seems to be a column constrained to 550px width, instead of being responsive and taking advantage of greater widths. to the author of the site, if you're not really into design/css, take a look at tailwindcss, it makes it fairly easy to produce a minimal amount of css that is responsive.
  • r00t4ccess 5y ago
    The page isn't prompting for cookie preferences, asking to allow notifications, popping up a mailing list or coupon half way do the page, playing a full page video with sound, or load 97million lines of javascript. I'd say its pretty much perfect.
  • IggleSniggle 5y ago
    Huh. I think it's a great UI. What did you not like about it?
• marginalia_nu 5y ago
  Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)
  • santamex 5y ago
    Which software do you use to index the sites?
    • marginalia_nu 5y ago
      I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke. A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays. So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that. Then it goes to the words file and looks up the URL range of the first word. Then it goes to the words file and looks up the URL range of the second word. Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word. Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result. I'm skipping over a few steps, but that's the very crudest of outlines.
      • rvnx 5y ago
        It's a great project!
      • q3k 5y ago
        Good stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration. Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
• fsflover 5y ago
  See also: https://wiby.me/ https://wiby.me/
  • Tade0 5y ago
    The "surprise me..." button is adequately labelled.
    • tpmx 5y ago
      Great link to drag to the bookmark bar.
    • twobitshifter 5y ago
      Shades of stumbleupon.
  • deleted 5y ago
    [deleted]
• gtmb 5y ago
  As everything in life flows in cycle, I predict the search engine that will de-throne Google will be like Google when it started - a simple variation of page rank. No smarts, no bubble, no signals decided by over fitting to a biased engineer preference.
  • jerrre 5y ago
    I wouldn't say the existence of this page proves your prediction right (as it's not dethroning Google anytime soon). It's easy to forget that the goal of Google isn't to provide a useful search engine (at least not anymore), but the search engine is a by product of them wanting to show ads.
    • popcube 5y ago
      now Google try immitate a document system on your computer, usually I rely on Google know what I need:(
    • marcos100 5y ago
      If Google isn't useful then nobody will use it. The search engine and the ads are tightly coupled. A better search engine means it can predict with more accuracy what you are looking for and can serve you an even more targeted ad that increases the chance you'll click.
      • rchaud 5y ago
        > If Google isn't useful then nobody will use it. Or they'll continue to use it out of sheer inertia. Google is paying Apple $15 billion to keep its place as iOS default search engine. IE6 didn't die overnight when Firefox arrived.
  • antupis 5y ago
    As dev I would love search engine which would only do search to stackoverflow github issues, documentation etc.
    • 5y ago
• samsaga2 5y ago
  Where does the data come from? Do you index the whole web yourself? I see it totally impossible for a personal project. I'm very curious about that.
  • marginalia_nu 5y ago
    I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort). You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7 million URLs. I've gone as far as 50 million and could probably double that if I really wanted to. The difficulty isn't having a small enough index, but rather having a relevant enough index, weeding out the link farms and stuff that just take space. I only index N-grams of up to 4 words, carefully chosen to be useful. The search engine, right now, is backed by a 317 Gb reverse index and a 5.2 Gb dictionary.
    • c0wb0yc0d3r 5y ago
      How did you go about seeding your web crawler with URLs to crawl?
      • dannyw 5y ago
        Not OP, but if I was to do this, I'd start by downloading Wikipedia and all its external links and references, and crawling from there. You should eventually reach most of the publicly visible internet.
        • c0wb0yc0d3r 5y ago
          I feel a little embarrassed that I didn't think of something like that. When I did some crawler experimenting in my younger years, I thought I was pretty clever using sites that would let you perform a random Google searches. I would just crawl all the pages from the results returned. Your method would undoubtedly be more interesting I think. It would certainly lead to interesting performance problems quicker, I bet.
• michaelcampbell 5y ago
  (Old man yells at cloud.)
  • deleted 5y ago
    [deleted]
• 300bps 5y ago
  What we need now is a search engine that weeds out sites that have been SEO optimized for keyword density. I’m tired of searching for “generic keyword” and getting a page with an extremely low signal to noise ratio written like this: “Many people search for generic keyword. That is why you can find all about generic keyword here. In fact we specialize in generic keyword and slight alterations of generic keyword.” It’s like Google stopped caring that people were gaming it.
• earthbee 5y ago
  I love this! I've been searching random words with no aim in particular and keep finding lots of interesting tiny personal webpages. It feels like the old web
• timdaub 5y ago
  I'm developing a text-heavy site and philosophically I'm trying to view documents as just that... documents [1]. But I don't get good results for "rug pull". - 1 https://rugpullindex.com https://rugpullindex.com
  • marginalia_nu 5y ago
    Yeah it's hosted by cloudflare. I'm currently IP-blocking them, as because they keep prompting my crawler with a captcha, presumably because it's made millions of requests from their CDN. Some rigmarole getting recognized as a good bot by the CDNs. I've submitted a request fairly recently, but haven't heard back from them yet. Like I would like to be on good terms with them, and other websites that block small independent crawlers. I can't blame them though, there's a lot of bad bots out there. But I'm doing my best not be part of the problem.
    • petercooper 5y ago
      Aha, I was going to ask how you were coping with CDNs like Cloudflare blocking bots. It's sad we've got to this point where basically only the established search engines are grandfathered in to be able to crawl sites.
    • deleted 5y ago
      [deleted]
    • Aachen 5y ago
      I wonder if google had to plead the same way or if already-big players are given a free advantage.
  • BugsJustFindMe 5y ago
    > I'm developing a text-heavy site I looked at the source for your site's front page. That's not text-heavy; that's markup-heavy. I didn't bother looking at the rest of the pages because it appears to be yet another crypto market site.
• deadalus 5y ago
  Very interesting because of the interesting results from random websites. It's a great discovery tool. Now hoping for search engine that favors text-heavy sites and punishes paywalls
  • the__alchemist 5y ago
    I built one!
• kebos 5y ago
  This is really cool, it filters out all fluff. It's not always taking me to totally relevant sites but the results contain my favourite type of content. Full of writing and pure html - usually the hallmark of someone who knows what they are doing, wants to communicate but doesn't want to waste their time.
• ape4 5y ago
  Fast and doesn't crash when on the front page of Hacker News!
  • marginalia_nu 5y ago
    Well,... yet. Load average is at 1.2, not that bad. But the services are getting a solid workout.
    • marginalia_nu 5y ago
      The real test is in now, index server is reconstructing its index. It does this every 6 hours if there is new pages. Takes half an hour or so usually. It's supposed to be able to handle searches at the same time, but jeepers, it's gonna have to chew through nearly 400 Gb of data while dealing with over 1 request per second.
      • 0xbadcafebee 5y ago
        Is your site/code on GitHub? I would be happy to give performance tips/tweaks. Also Fyi, https://marginalia.nu/ https://marginalia.nu/ gives a certificate error (I know that's not the search site)
  • ricardo81 5y ago
    That crossed my mind too, considering vanilla webpages sometimes struggle with a top page HN thread, never mind a search engine backend.
• mattowen_uk 5y ago
  Nice! Typing my name in, gets my own site back as 3 of the top 5 results. I suddenly feel important ;)
• mumblemumble 5y ago
  I like it. Coincidentally, the other day I was daydreaming about a search engine that favors sites that are updated less frequently. The thought being, the kinds of labors of love that characterized the 1990s Web that I still sometimes miss are still out there, it's just harder to find them amidst the flood of SEO dreck. So perhaps they could be made discoverable again with the help of a contrarian search engine that specifically looks for the kinds of things that Google and Bing don't like to see.
  • potatoman22 5y ago
    I like the idea. Tangentially, I wonder how one would find the right 'penalty' for more updated sites?
  • gibspaulding 5y ago
    Million Short [1] offers an option to omit results from popular domains. It's a different approach from what you describe, but I think the goal is similar. [1] https://millionshort.com/ https://millionshort.com/
    • Phiwise_ 5y ago
      I think I like this method the most, honestly. There's something to be said for minimalist web design, but there's also something to be said for everything we've learned since the 90s.
  • itzworm 5y ago
    I had this problem recently trying to fix an Atari. There's a guy out there who has ton's of guides on doing video out mods but newer guide references the older. However googling the OG guide didn't find it so I manually scoured his old web page.
  • capableweb 5y ago
    Similarly, I wish there was a recommendation engine (for web, music, movies, whatever) that can show you what is the furthest away from your existing tastes. I've learned to re-create my Spotify account once every 6 months or so, as their recommendation engine becomes a boring machine after using it daily for some months. I'd love to discover new content that is different from what I read/watch/listen to now, but it's really hard to know about genres you don't know about.
• platz 5y ago
  modern design = low information density?
  • monkeybutton 5y ago
    Definitely low signal to noise. Looking at you recipe websites and cooking blogs.
• cyral 5y ago
  I tried a few queries and got extremely irrelevant results
  • marginalia_nu 5y ago
    It really depends on what you search for. A major drawback is that there needs to be text-heavy sites to find, in order for the search engine to find them. Compare for example the results for "Duke Nukem 3D" with those for "Cyberpunk 2077".
• xipho 5y ago
  Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most (all?) of the major data sources that have accumulated data on it. When Wolfram Alpha added species names, I checked with the name, boring, Duck Duck, boring, Google (well we know Google isn't for search anymore, it's absolutely horrible) boring, Bing, boring... you get the idea. This was a little different, extremely few results, but a couple of them really made me grin, and all(?) made me curious or raise an eyebrow or reflect on who/what might have been the source of the link, or remember some obscure connection from grad-school. So, if anything a crawled list of results worthy of ponder, thanks for this!
  • duckmysick 5y ago
    I'm intrigued by this experiment but I can't visualize it. What do you mean by boring results? Would combing through a library (the one with paper books) also produce boring results? What's your ideal results?
    • xattt 5y ago
      There’s certain grey literature that’s not captured in university library federated searches nor easily found with mainstream search engines.
      • xipho 5y ago
        There are decades of academic research not digitized. The digitization window used to only hit around 1990, I haven't looked at it hard recently, but I suspect this still remains true for many important journals. This is grey only to those who do not know how to use a library.
    • xipho 5y ago
      Perhaps a counter example, something that is interesting. Anecdotally. This, of all things, is the top result in my search: https://tft.brainiac.com/archive/0303/msg00037.html https://tft.brainiac.com/archive/0303/msg00037.html. Which is strange to me because I don't recognize tft.brainiac. I click, it's a list of biological relationships among Hymenoptera, including a reference to genus of the wasps I studied, presumably in a biological relationship (host/parasite) context. I cataloged every relationship known at one point, so my brain wants to know where this come from, is it something I caught. Then I go look for more context, and find it's part of a thread about D&D(?) and hymenoptera, and it's epic, and a chunk of my morning is lost figuring out why and how this came to be.
• pietroppeter 5y ago
  from About page: > If you search for "Plato", you might for example end up at the Canterbury Tales. Go looking for the Canterbury Tales, and you may stumble upon Neil Gaiman's blog. I know it is just a suggestion, but had to try searching both, with no luck in getting the expected unexpected.
  • marginalia_nu 5y ago
    Yeah I did some work very recently aimed at improving the relevance a bit. It was a bit too random in the state it was before. Now it, perhaps, isn't random enough anymore.
    • pietroppeter 5y ago
      It looks very nice anyway, great job! I did try with other queries and results were in general interesting.
• yewenjie 5y ago
  Related question - suppose I want to create a meta search engine for myself, and I want it to be as fast as possible. What are the things I should be optimizing for?
• streamofdigits 5y ago
  Let a thousand search engines bloom. btw, interesting how many http (as opposed to https) sites show up...
• michaelgrafl 5y ago
  I just looked up my last name and found a World class heavyweight weightlifter named Josef Grafl born in 1872 who has an awesome portrait of him on Wikipedia. Never before have I read about that man. I love this.
• prionassembly 5y ago
  The website itself seems generated with some kind of kick-ass generator from template files (.gmi?) I feel like I'm stuck with Wordpress.com because it brings me some traffic (whereas something hand-rolled on nsfspeech or digital ocean or whatever would literally be off the edge of the web), but the structure of that is so cool.
  • NhanH 5y ago
    That would be gemini protocol!
  • huijzer 5y ago
    You can easily do proper SEO with static site generators too. Even more, static sites can be hosted via GitHub or GitLab Pages, Netlify or CloudFlare and in all cases the speed will outperform Wordpress in almost all cases. Also, you have way more control over the output than with Wordpress.
• lbriner 5y ago
  My pet peeve with search results is simply that there are ancient technical results that in many cases are irrelevant. If I am searching for a Window error message, I don't want some old forum post from 2001, especially if it didn't have any answers! What would be cool would be for people who host old stuff to "archive" it at some point so it doesn't appear in normal results, only if you tick "include archives".
  • athenot 5y ago
    As much as the release names for macOS over the years were marketing gimmicks, it does make it a lot easier to zero in on the correct version when doing these types of searches.
• egberts1 5y ago
  I tried “Error 49” as a search phrase. It’s rudimentary but no IT-related result.
• zimpenfish 5y ago
  Searched for my initials - got back a bunch of raw binary results (mp4, pdf, img, txz, etc.) which was disconcerting. Although it did find one reference to actual-me which is better than Google manages on the first 4 pages... https://imgur.com/a/n2xro2Y https://imgur.com/a/n2xro2Y
  • marginalia_nu 5y ago
    Yeah there was unfortunately a problem with the content-type code recently, it unfortunately categorized some binary data as HTML and tried to process it best-effort. So there's some binary soup in the index. The bug has since been fixed, but it won't come into effect in a few weeks.
• marcosdumay 5y ago
  I've got some results where the same site is has than 70% of the links. It was a very on topic and high quality site, but still, all the results shouldn't point to the same place. I think some grouping by site (and capping to only the few most relevant links there) would improve the engine.
  • marginalia_nu 5y ago
    It shouldn't give you more than three per domain on paper, but I've got issues with deduplication so sometimes you get six :-( Still a lot of work needs to be done.
• arnaudsm 5y ago
  I wish we could configure Google's algorithm to our needs, and blacklist websites.
  • MarioMan 5y ago
    It could get tedious depending on how many sites you want to block, but you can add "-site:google.com" to exclude google.com, for instance.
    • arnaudsm 5y ago
      I mean a blacklist system like Twitter's, where you block a website forever. Pinterest would be the first to go.
• yakubin 5y ago
  Wikpedia links point to <https://encyclopedia.marginalia.nu/ https://encyclopedia.marginalia.nu/> instead, which to my eyes is less readable. The justified text, done with CSS, instead of the LaTeX algorithm, looks wild. The font used for quotations is even worse (very thin). Wikipedia is perfectly usable without JavaScript and it's one of the nicest sites out there typography-wise, so I'd reconsider this redirection.
  • marginalia_nu 5y ago
    I guess it's a matter of taste. I can barely read anything on regular wikipedia because the inline links disrupt my flow.
• eitland 5y ago
  Tested with the first person to settle on Island: https://search.marginalia.nu/search?query=Ingolf+Arnarson https://search.marginalia.nu/search?query=Ingolf+Arnarson and it worked surprisingly well. Anyone else has good examples?
• pjs_ 5y ago
  This kicks ass!!
• platz 5y ago
  > Don't be afraid to scroll down in the search results I never knew it was fear that was preventing me from scrolling
• OneEyedRobot 5y ago
  Very cool. A person can really appreciate simple web design looking at something like Luke Smith's recipe page. So how on earth do you take an idea like this and scale it for both broad web coverage and high traffic? For that matter, just how much 'useful' text is there on the net?
• hdjjhhvvhga 5y ago
  Congratulations, great work!
• thrtythreeforty 5y ago
  > New: You can now look up dictionary definitions for words. If you for example don't know what the definition of is is, you can inquire thus: define:is. Oh man, I love subtle jabs and tongue in cheek writing like this. Very Robin Williams-esque.
  • marginalia_nu 5y ago
    I am the first to admit it's a pretty dated reference.
• fhackernewz 5y ago
  fuck you hacker news
• silent_cal 5y ago
  Wonderful work (':
• FractalHQ 5y ago
  Ok this is great if all I want to do is read text, but often times that is very much not all I want to do. The web is much more than text and images these days. I can appreciate this as long as it’s branded as a search engine for blogs and articles specifically, as opposed to being touted as a drop-in replacement for the modern search engine.
  • lukas099 5y ago
    Is this a criticism? It doesn't at all seem touted as a drop-in replacement for the modern search engine.
    • FractalHQ 5y ago
      I do think this is a great tool, I didn’t intend to come off as being critical. I think I had some misdirected frustration from people on HN talking about how the modern web is bad and should be replaced with a text protocol. I aspire to create web apps that push the boundaries of what is possible today, and feel disappointed whenever I encounter people advocating for regression. But I digress. In hindsight, my comment was obnoxious and under-appreciative of the tool being shared, and my rant was only tangentially related.
• leephillips 5y ago
  This is wonderful and stupendous. I’ve often thought that Google could be turned back into a good search engine by simply eliminating the crap and letting the useful sites float to the top of the results. marginalia.nu seems to like my sites, so it must be good! Some results are prefixed with ! or an arrow dingbat. What does that mean?
• mattchew 5y ago
  Oh, I dream of a day where there are multiple useful search engines, specialized for different purposes. You're doing God's work here. Thanks and good luck.
  • brian_herman 5y ago
    Kind of reminds me of the past like alta vista and dogpile.
  • scns 5y ago
    The Flying Spaghetti Monster wants to have a word with you. (edit) That is a nice dream though.
  • BoxOfRain 5y ago
    I wonder if there's any mileage in an extension of something like uBlock Origin's lists of ad networks to block but instead it's a list of known content mills and SEO spam factories to remove from search results?
• snakeboy 5y ago
  Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totally useless.
  • hdjjhhvvhga 5y ago
    As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.
    • jandrese 5y ago
      This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now. Still, the best way to break SEO is to have actual competition in the search space. As long as SEO remains focused on Google there is an opportunity for these companies to thrive by evading SEO braindamage.
      • xtracto 5y ago
        That's why I use Saffron [1], it magically converts those sites into a page in my recipe book. I found it when the developer commented here in HN. Also, a lot of cooking website have started to add a link with "jump to recipe" functionality allowing you to skip all the crap. [1] https://www.mysaffronapp.com/ https://www.mysaffronapp.com/
        • Funes- 5y ago
          There's also https://based.cooking https://based.cooking.
          • eigengrau5150 5y ago
• criddell 5y ago
  Have you given any thought on what you will do if you get a DMCA take down request or a request from a person asking you to remove them from search results?
• schmorptron 5y ago
  I've also found that brave search gets much better results than google for some programming related topic, simply by not being targeted by blogspam SEO as much. It's refreshing to not have to click through 3 auto generated "articles" but to either a) get the documentation straight away or b) find actually human written blog entries.
• arethuza 5y ago
  I did a quick check using the name of the Scottish village I am originally from (or as I should say "far am fae") and this produced a much more interesting set of links for me than that produced by Google
• titzer 5y ago
  I think I want a BBS. Text mode, fixed width font, keyboard-driven menus, no (or very little) bitmapped graphics. I've been thinking about the UIs for a lot of sites that I use to "do things" on the web. E.g. search for flights. Do I need any of that "beautiful" web design with pretty forms and fonts, bevelled edges, drop shadows, drop-down menus, hovers? Hell, do I even need a map? Heck no, I need three text entry fields and output a bulleted list, maybe table of results. Just give me the raw data and do as little presentation as possible, thanks. I really think I want an internet console, not an animated magazine.
• javajosh 5y ago
  This is really good; I'll actually use it!
• lumost 5y ago
  This is a fascinating tool, I estimated that the corpus of the factual web was between 1 and 10 TB when I last played around with BigQuery using domain names which had low amounts of click bait. Seeing these search results I suspect my estimate was off by a couple orders of magnitude. Although a search for "Fractional Reserve Banking" shows that some further ranking improvements can be made to exclude unrelated results, and potentially penalize old conspiracy sites. https://search.marginalia.nu/search?query=fractional+reserve+banking https://search.marginalia.nu/search?query=fractional+reserve...
• 0xbadcafebee 5y ago
  I love it. Even though it didn't give me the results I was looking for. I searched "new york fishing license", and it didn't give me any links to the actual new york fishing license websites. But it did give me a ton of really cute little websites related to lakes and fishing in New York. This one has amazing information about fishing all over Western New York: http://www.huntfishnyoutdoors.com/fishing.php http://www.huntfishnyoutdoors.com/fishing.php
• mrpf1ster 5y ago
  I searched "c strtok" and got one result saying '"strtok" could be spelled "stroke", "stork", "sarto", "strop"'. Cool concept though!
  • marginalia_nu 5y ago
    The spelling suggestions are presented whenever there isn't any results, but sometimes they can be pretty misleading. What happens is that C, as a word, isn't indexed because it's deemed too short, and the bigram "c strtok" can't be found anywhere. Try 'strtok' instead.
• skyfaller 5y ago
  This is a fantastic search engine. It delivers on its promise of "serendipity". I found pages featuring my name that I'm not sure I've ever seen before, after many years of searching myself to test out search engines. Perhaps more importantly, it delivers the most correct result when searching for my username: the first result is not any of my social media accounts, or even my own blog, but the text of the obscure science fiction story that I took my username from! Well done. I've immediately added this as a search keyword in Firefox, and I'll be using it more in the future. Could meta search engines like DuckDuckGo include this as a source? Should they?
• rfrey 5y ago
  This is stunning. I searched "winemaking" because it's my latest obsession, and turned up dozens of links to high-quality pages I'd never seen despite spending an hour a day for three months cruising Google on the topic. Please do announce it here if you ever decide to solicit help or contributors. My stab at this problem was to have a search index of only ad-free pages, on the hypothesis it would turn up self-hosted blogs, university personal pages, that sort of thing. But the results were too thin, your approach is much better.
  • rc_mob 5y ago
    or if he wants investors. I'd invest few hundred k in this
• winddude 5y ago
  hmm, I dream of recipes search engine that punishes recipes pages with too much text. lol
  • mint2 5y ago
    Yeah, recipes sites have both too much text and too many pictures. But they do illustrate what this search engine needs to watch out of. If they rank more text higher and their search site becomes popular, won’t everyone just spam recipe site word salad, maybe even ai generated word salad. But in the interval, until that day comes, they are going to have a very useful service.
• tbojanin 5y ago
  this is sweet
• winddude 5y ago
  curious how do you afford the infrastructure? I found that the hardest part of running a search engine.
  • marginalia_nu 5y ago
    I'm self-hosting, and the server is a Ryzen 7 3900x with 128 Gb of non-ECC RAM. It sits in my living room next to a cheap UPS. I did snag one of the last remaining Optane 900Ps off Amazon, and it powers the index and the database--and I really do think this is among the best hardware choices for this use case. But beyond that it's really nothing special, hardware-wise. Like it's less than a month's salary. It runs Debian, and all the services run bare metal with zero containerization. Modern consumer hardware can be absurdly powerful if you let it. Like I have no doubt a thousand engineers could spend a hundred times as much time building a search engine that did pretty much the same thing mine does, it would require a full data center to stay running and be much slower. But that's just a cost of large scale software development I don't have to pay as a solo developer with no deadline, no planning and a shoestring budget.
• sailorganymede 5y ago
  Love this! Is there any way someone could help contribute to this?
• ncfausti 5y ago
  This is incredible. I just got goosebumps as I stumbled upon https://solitaryroad.com https://solitaryroad.com after searching for "linear algebra homomorphism". It reminds me of the magical feelings of the early Internet. Keep up the great work!
• abdullahkhalids 5y ago
  Can we submit text-heavy sites for possible inclusion? Assuming they pass your filters.
• snuser 5y ago
  Just searching for 'dogs' gave me more interesting results than I've seen from google in years
• josefresco 5y ago
  It you like wacky search engines, there's also Million Short: https://millionshort.com https://millionshort.com where you can search and remove the top 100/1K/10k/100K/1M results.
• jarbus 5y ago
  This engine is fantastic for recipes
• isaacgreyed 5y ago
  A common use case, how to do random thing in programming: I searched python make a bar chart and it returned a live coding video with an AI generated text transcript and two articles which mentioned a different kind of bar. I then narrowed it down to just python bar chart, and got a blog post about scripting with a bar chart in it, this http://www.nitcentral.com/voyager4/hellyear.htm http://www.nitcentral.com/voyager4/hellyear.htm with monty python, bars, and charts from 1996 and among some other things I found this https://python-course.eu/naive_bayes_classifier_introduction.php https://python-course.eu/naive_bayes_classifier_introduction..., which had an example of a python bar chart even though the title of the page made me think it wasn't what I wanted. So for what I imagine to be a difficult search because of all the different meanings of the words, I found my result on the second query pretty quickly, and found some cool unrelated stuff too. I like mostly that I get what I type in, and not exactly what I want, but what I want is there too.
  • CapmCrackaWaka 5y ago
    I would probably use this if I wanted to find interesting blog posts/websites about a topic I want to learn more about in general. It seems less useful for returning exact answers to specific questions.
• kews 5y ago
  There's probably a more suitable term than "modern" that we should generally be using, since "modern" consistently has a positive connotation.
  • marginalia_nu 5y ago
    Dunno, I prefer to use as neutral or positive terminology even when I talk about things I don't like. I think it very easily comes off as juvenile ranting when you start throwing around terms with strong negative connotations.
• greggturkington 5y ago
  Wouldn't this just skew towards really old sites? The third search result for "dog" is this page on how to remove AOL Instant Messenger, published in 2002. https://sillydog.org/netscape/kb/removeaim.html https://sillydog.org/netscape/kb/removeaim.html No one wants to see newsletter signup popovers, but "modern web design" includes good performance and relevant content. (The search engine itself takes about 2 seconds to first contentful paint, not great.)
  • bityard 5y ago
    This search engine pretty much takes everything that Google is doing and does the opposite. For instance, Google has decided that "relevant" usually also means "recent". Thus, when searching for something on Google, you mainly get results from blogspam farms and almost never do you see anything more than a few years old. An implication of this is that old sites tend to disappear (either into obscurity or by being taken down) because Google penalizes them in search rankings. The author of this search engine says, however: > If a webpage has been around for a long time, then odds are it has fundamental redeeming quality that has motivated keeping it around all for that time. I don't know that I agree 100% with this (there was lots of crap on the "old" web too), but it makes a certain amount of sense.
    • greggturkington 5y ago
      What "fundamental redeeming quality" about uninstalling AIM from Windows 3.x motivated making that the 3rd result for "dog"? The 5th result is a tutorial on CSS. This search engine decided it's relevant because it has "dog" in the URL. Is that a better reasoning than Google's? https://htmldog.com/guides/css/beginner/ https://htmldog.com/guides/css/beginner/ Core Web Vitals ranks sites higher that perform well. Text-heavy sites that are also optimized and relevant would already perform well.
      • marginalia_nu 5y ago
        What are you searching for when you enter the query "dog", keeping in mind the search engine deliberately does not examine synonyms or and deliberately seeks out the path less taken? Dog facts? Then search "dog facts" Famous dogs? Then search "famous dogs" Rappers? Try "snoop dogg"
• claytn 5y ago
  Searching for your own name will turn up some interesting results! I got some early 90s webpages that just contain obituaries or marriage records. I never knew cities maintained these records online!
• timvisee 5y ago
  Cool! There do seem to be some text encoding issues though. For example: https://search.marginalia.nu/search?query=tim+visee https://search.marginalia.nu/search?query=tim+visee
  • marginalia_nu 5y ago
    Yeah I think the charset detection needs work. It understands the "Content-type: text/html;charset=utf-8" -header, and <meta charset="UTF-8"> but not <meta http-equiv="content-type" content="text/html; charset=utf-8"> It turns out HTML has a lot of corner cases. I'm constantly marveling at how web browsers hold together as well as they do.
    • timvisee 5y ago
      Thanks for your response! Hope you can implement this as well without too much trouble. I wonder if you could just assume UTF-8 to be the default these days. I imagine that to fix many other cases as well.
      • marginalia_nu 5y ago
        Haha! I did actually assume UTF-8 at first, but being a search engine has a lot of older websites, I sadly got a lot of encoding errors doing that, too.
        • soheil 5y ago
          Maybe just like js-heavy sites also punish non-conforming-encoding sites.
• thetanil 5y ago
  Yes please! More of this!
• jordache 5y ago
  how about a search engine that bans all pinterest content. I hate pinterest with a passion. I may need to get a "his" laptop separate from my wife, since she needs that darn pinterest extension for pinning photos.
• abhiminator 5y ago
  Absolute textgasm. Wonder how 'text-only first' prioritization is being implemented, algorithmically speaking?
  • ryankrage77 5y ago
    This post - https://news.ycombinator.com/item?id=28551183 https://news.ycombinator.com/item?id=28551183 - suggests it's a simple set of hueristics, looking for things like javascript, link/SEO spam, language, amount of text content, etc, filtering out unwanted results and only indexing wanted ones.
    • abhiminator 5y ago
      Thank you!
• asjdflakjsdf 5y ago
  You should monetise this with amazon affiliate links that are relevant to each search. And then use that money to keep this project going. Google is fantastic, but it has become something different from what it was, the company and the product. It is so refreshing to see a modern tool that encourages exploration of the actual world wide web.
  • marginalia_nu 5y ago
    I might add a donate button or something if people want to help support the project, hardware isn't cheap and all. But I have a job and decent income. I think if this search engine became the way I earned money, it would influence the project in a bad way, and corrupt its purpose, which is to help people explore the less-commercial internet.
    • kews 5y ago
      Appreciated. The more things fill up with monetizing shit, the more I stay away. There's something beautiful in having higher purposes than grubbing for cash.
    • eigenhombre 5y ago
      +1 for a donate button; much preferred over affiliate links or ads of any kind. Thank you for making this beautiful little(?) product!
    • RistrettoMike 5y ago
      I'd donate to continued expansion/development of something like this. Where is somewhere good to follow you for any thoughts/updates?
      • marginalia_nu 5y ago
        I have something of a blog here, with an Atom feed. https://memex.marginalia.nu/log/ https://memex.marginalia.nu/log/ It's not very well optimized for mobile, really it's more of a bridge for my geminispace content.
• abhinav22 5y ago
  Great work and congrats!
• Phileosopher 5y ago
  Wow, if this catches on, my original content will actually matter![1] I've always had a love-hate relationship with modern web design principles because my design choices have all the excitement and polish of what we get on HN. I'm sure I'm not the only one, either. Content-rich sites need more love. [1] https://adequate.life https://adequate.life
• aabajian 5y ago
  Gotta say, sometimes the results really are nice. I searched for "Land Cruiser 70." The first result is a simple, short blog post about a couple who traveled across Europe and Asia in their Troop Carrier (http://www.destoop.com/trip/1%20PREPARATION/2%20Vehicle%20specification/OUR_CAR.HTM http://www.destoop.com/trip/1%20PREPARATION/2%20Vehicle%20sp...). The first results on Google are Australian site for buying a LC70 (news-flash, I can't buy one in the USA). There is also a MotorTrend article about the LC70...also irrelevant since it's only sold in Australia.
• afrcnc 5y ago
  except it doesn't actually return that many results
• hulitu 5y ago
  "Search results Search "alt.sysadmin.recovery" needs to be a word Those were all the results," No comment.
• spandrew 5y ago
  All of my searches are turning up unrelated results ("college life after the pandemic", "post-pandemic teaching in higher education", "football news NFL" etc.) NFL one had 'some' decently related results, but the websites were all strangely disreputable.
  • mmmpop 5y ago
    > the websites were all strangely disreputable Interesting you'd feel that way when sites without "modern design" are encountered. Is this your own bias perhaps creating a judgment or are they sites that you already know have a bad reputation?
    • typon 5y ago
      Or perhaps the websites being returned are garbage? I have the same experience trying a few searches and following the top 5 links. Besides wikipedia, I haven't found a single useful website.
      • spandrew 5y ago
        This! Maybe modern or 'non-modern' web design just isn't a great litmus test for quality content? Could just need some work. At any rate I wasn't clicking on the results.
• exabrial 5y ago
  I would like a search that punishes 'modern' SPOs that load 87mb of the author's pet JS projects to display simple text. Basically every modern SPO.
• api 5y ago
  This is the most amazing thing I have seen on here in at least a year! It's... no... it can't be... a search engine that finds actual information instead of 5 megabyte blobs of tracking code and SEO crap!
• pajko 5y ago
  Does it filter out ad-heavy copy-paste/autogenerated fake sites? Tired of seeing those on the first few pages of Google. Bing gets more and more usable, but far from perfect.
  • marginalia_nu 5y ago
    It tries.
• Nicksil 5y ago
  - Semantic HTML; not everything is a div; correct use of markup. - Search results are not overrun with commercial, SEO stuffing, "content" farms. I don't know what to say. This is such a refreshing sight. Well done.
• kodeninja 5y ago
  Ivermectin (marginalia): https://search.marginalia.nu/search?query=ivermectin+ https://search.marginalia.nu/search?query=ivermectin+ Ivermectin (Google): https://www.google.com/search?q=ivermectin https://www.google.com/search?q=ivermectin The difference in the overall _thrust_ of the results is remarkable. Very interesting! Thanks for building it.
  • silent_cal 5y ago
    Lol!
  • typon 5y ago
    The Google results tell you why Ivermectin is not a good replacement for vaccination against Covid, the Marginalia results tell you that Ivermectin is a miracle drug for treating Covid 19. Really shows how much technology has the power to change reality in today's world.
    • lame-robot-hoax 5y ago
      Google links to the FDA, CDC, WHO, NIH, WebMD, drugs.com, and a pro ivermectin journal article from the American Journal of Therapeutics. The Marginalia results point you mostly to random blogs.
    • marginalia_nu 5y ago
      Part of what I wanted to show with this project is that there is no such thing as an objective search engine. Even seemingly irrelevant technological decisions drastically impact the narrative.
      • Drew_ 5y ago
        Well the focus on text content isn't the only technical difference here. Google is obviously weighing hundreds of signals in its search results that your engine is not accounting for. These omitted signals are also relevant.
        • 5y ago
• NotAnOtter 5y ago
  "Don't be afraid to scroll down in the search results, unlike in many other search engines, depending on what you are looking for, you may find the best results in the middle of the listing." This is a very polite way of saying "this engine isn't very good" Overall impressed with the project but I thought the word play there was funny
  • marginalia_nu 5y ago
    I felt I needed to add it to help people taught by other search engines that they only get 1-2 good results, and the rest is useless. The reason I'm providing a hundred results is that there are often a lot of results to choose from. If the point is to find something unexpected, and that indeed is the entire point, then that is the only sane design choice. Like you search for something on Google and similar, and you know what you are going to find. They are so good at searching the Internet and predicting what you are going to click on that you never see something new. It's a great feat of engineering, but a huge tragedy, because discovering new things, outside of what you our your demographic has previously demonstrated an interest in, it can be absolutely life changing.
• rc_mob 5y ago
  blessings upon you sir for making this
• mrkramer 5y ago
  Awesome work! I had similar idea in mind but I'm glad to see someone else was able to pull it off.
• yrds96 5y ago
  Damn that's is interesting search engine, this is great for search simple terms and find a bunch of blog articles about the term.
• nagyf 5y ago
  This is great, I like the results. Couple of things I noticed: - Search results often very old, from the early 2000s (I guess because back then more websites were text oriented). Are you taking into account the age of the page when showing results? It would be great to see more up-to-date results at the top - I noticed a few results which directed me to websites with security risks, Firefox didn't even let me open them. Is it possible to filter these out from the results?
• bovermyer 5y ago
  I adore this. Unfortunately, searching for my own name - with or without quotes - doesn't actually find my site. It does find a handful of references to me from over twenty years ago, though, which I thought was fascinating.
  • tgv 5y ago
    My name retrieved the "dead pornstar list". Unexpected.
• gen_greyface 5y ago
  Hi, It'd be nice if you could add a OpenSearch description document for your site. https://developer.mozilla.org/en-US/docs/Web/OpenSearch https://developer.mozilla.org/en-US/docs/Web/OpenSearch
  • gen_greyface 5y ago
    until then i'll keep the site bookmarked. :-)
• lightsurfer 5y ago
  thank you!
• palijer 5y ago
  This has been needed in my life for a while. I am growing really apathetic about the internet lately, but I realize that is because my entry point is always a google search. I miss finding blog posts and scholarly articles in long form. I hate the SEO sites with unreadable UI because the information in them is often a lot lower quality as well.
  • marginalia_nu 5y ago
    I think the small web and gemini protocol might be up your alley too, if you feel this way about the Internet.
• IlliOnato 5y ago
  Pretty cool. I am not sure yet how useful, but cool it is. However, it seems that it currently does not support non-Latin alphabets. Which I understand in an early version. Still, it's handling of such "exception cases" could be improved: when I search for a Russian word, say "Аквариум", I get <<Search "Аквариум" needs to be a word>>, which is rather rude...
  • foxfluff 5y ago
    "It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort)." https://news.ycombinator.com/item?id=28551183 https://news.ycombinator.com/item?id=28551183
    • IlliOnato 5y ago
      Fine; still "Not a supported language" would be much better response than "not a word".
• dukeofdoom 5y ago
  "corporate speak" bs detector and filter on google search engine would be nice.
• agumonkey 5y ago
  Very nice. Start a trend :)
• pkamb 5y ago
  Great results for "sauna". Lots of Web 1.0 pages discussing building plans and displaying pictures of individually built, traditional, unique, old saunas on some property. The Google result are all blogspam or sales pages for cheap shipped saunas. Lots of "IR" results. Phony health benefit pages. Stock photos solely of beautiful new hotel gyms. I've noticed this problem with Google results for quite some time. Sadly, the new content being created of the top variety is mostly being done within private Facebook groups that can't be easily searched, linked, or archived.
• justinzollars 5y ago
  It works. Nice job.
• slim 5y ago
  This is a search engine indexing the internet on a mariadb database hosted on consumer hardware maintained by a single person as a hobby and it does not suffer from HN hug of death
  • swyx 5y ago
    how on earth do you index so much on consumer hardware? my frontend developer mind is blown.
    • foxfluff 5y ago
      Wait till you learn that modern CPUs run billions of cycles per second. With multiple cores in parallel! And they can reach transfer rates of tens of gigabytes per second to RAM, or around a terabyte per second into L3.
      • ThalesX 5y ago
        And then you add a single HTTP request and everything tones down to the speed of the web. Or I/O. Or DB call.
        • goodpoint 5y ago
          More like: you add some javascript library and now the browser needs 5 seconds to run 10 MB of javascript.
        • knuthsat 5y ago
          You can still support millions of these requests per second if you just bake all of the dependencies directly in a small OS running on your fastest raspberry pi.
    • paxys 5y ago
      Consumer hardware today is simply what was cutting-edge and crazy expensive 5 years ago.
  • marginalia_nu 5y ago
• gjm11 5y ago
  I tried a few searches. <<javascript pipe syntax>>: none of the search results appeared to have anything to do with Javascript pipe syntax. (Which doesn't exist yet, but it's under discussion.) Google gives a bunch of highly-relevant results. <<hans reichenbach relativity>>: first result is a list of books about relativity, one of which is Reichenbach's "Philosophy of space and time"; good, but there's no real information there. Second is about Reichenbach but nothing to do with relativity or even, really, philosophy of science. Third is about philosophy of science and mentions some of Reichenbach's work but not related to relativity. Fourth mentions Reichenbach's "Philosophy of space and time" as part of a list of books relevant to a seminar on "time and eternity". None of this is bad, but it's not great either. Google gives a couple of online philosophy encyclopaedia entries, then a journal article on "Hans Reichenbach's relativity of geometry", then the Wikipedia article on Reichenbach ... much more informative. <<luna lovegood actress>>: I thought this would be an easy one. It was easy for Google, which gave me her name in large friendly letters at the top, then her IMDB entry, and a bunch of other relevant things. Literally nothing in the Marginalia results was relevant to the query. I guess maybe popular culture is just too monetizable, so no one is going to write about it on the sites that Marginalia crawls? Let's try some slightly less popular culture. <<wilde "a handbag">>: First result is kinda-relevant but weird: it's about a musical adaptation of The Importance of Being Earnest. It doesn't mention that famous line from the play, but one of the numbers in the musical has the words "a handbag" in the title. Second result is a review of a CD of musicals, including the same work. Third is a bunch of short reviews of theatrical items from the Buxton Festival Fringe, one of which is a three-man adaptation of TIOBE. Next four are 100% irrelevant. Next is a list of names of plays. Last one is actually relevant; it's an article about "Lady Bracknell through the decades". Google puts that one first (after, sigh, a bunch of YouTube videos which look as if they might actually be relevant). I really like the idea of this, and many of the things it turns up look like they might be interesting, but it isn't doing very well at producing results that are actually relevant to the thing being searched for.
  • silent_cal 5y ago
    To be fair, those searches are pretty weird.
• pomian 5y ago
  Congratulations. Truly impressive search results. I tried two, one word searches. The results were interesting, useful, and would have been impossible (well, really really hard) to find, on standard search engines. Plus, no garbage, ads, recommendations, etc etc. As another commenter suggested, it is what World Wide Web searches results were like, twenty years ago!
  • pomian 5y ago
    PS. I added Marginalia as a search option (even the default for now) in Firefox Nightly (on Android). In case others want to, under settings for search, you can add other, then name, and then: https://search.marginalia.nu/search?query=%s https://search.marginalia.nu/search?query=%s
    • freddref 5y ago
      After a good amount of searching it doesn't seem possible to add Marginalia as default search in firefox (84.0b8) on Debian. I did not expect this to not be available.
      • mmphosis 5y ago
        I am running Linux Mint, and I did not expect adding a custom search engine to be missing from Firefox. There are plug-ins, but I don't like adding plugins. https://mmphosis.netlify.app/search.marginalia.nu/ https://mmphosis.netlify.app/search.marginalia.nu/
      • TeMPOraL 5y ago
        Open a search, right-click on the address bar. The author added the magic metadata needed, so Firefox should be able to pick it up and offer "add search engine" option in the context menu. I know. It's an idiotic UX regression over being able to configure search engines yourself in the Settings screen. Takes control away from the user.
• dumbfounder 5y ago
  Based on a few searches it seems to favor sites with very long passages of text. Search for a name and you get pages with massive lists of names. It quite simply isn’t very good at everyday searches. But it does bring up the point, shouldn’t I be able to tell my search engine I want results like this? It should be a feature of google I can turn on and off. It should be one of many ways to impact relevance.
• high_byte 5y ago
  I'd like a chrome extension that marks links that target text-heavy vs "modern" so I know beforehand what to expect - paywall, ads, popups, clickbaits, etc.
• marto1 5y ago
  super! How far would you say are you in indexing the blogosphere ? I tried the engine a few times, but I mostly get academic papers and I know most (good) blogs are in fact text-heavy.
  • marginalia_nu 5y ago
    It's not indexed particularly deeply. Blogs typically have a decent amount of javascripts.
    • Aissen 5y ago
      I have a static website (granted, it's not well linked), with no JS, and it does not seem to be in the index. But I saw in a sibling post that you had index limits, so it makes sense IMO.
      • marginalia_nu 5y ago
        Probably just hasn't been discovered then. The Internet is big. I wish more small websites would be better at linking to other small websites :-(
        • Aissen 5y ago
          It must be close, I found a page that links to it. Only one or two more other of magnitudes to index it :-)
• pjspycha 5y ago
  This is really refreshing work, and we can all benefit from other search engines focused on improving the field. I tried a bunch of searches and some of them were quite wonderful, others were a little dry on results. But overall I enjoyed going through it. Here is some critiques if you don't mind: I did search for "Daria Bilodid" and the results were a bit troublesome. First the Wikipedia result did not work: https://en.wikipedia.org/wiki/Daria_Bilodid https://en.wikipedia.org/wiki/Daria_Bilodid vs https://encyclopedia.marginalia.nu/wiki/Daria_Bilodid https://encyclopedia.marginalia.nu/wiki/Daria_Bilodid Secondly the results matched a few judoinside.com results which is ok, including sites to her competitors, but seemed to miss the judoinside website for her: https://www.judoinside.com/judoka/92660/Daria_Bilodid https://www.judoinside.com/judoka/92660/Daria_Bilodid. The design is hard on my eyes, I have a average size screen and its using less than half of the width. The line-height is enormous and seems to breakup flow making it uncomfortable for me to read. The spacing around each result is the same as between titles and paragraph items, which again was unpleasant to read.
  • ASalazarMX 5y ago
    > Secondly the results matched a few judoinside.com results which is ok, including sites to her competitors, but seemed to miss the judoinside website for her: https://www.judoinside.com/judoka/92660/Daria_Bilodid https://www.judoinside.com/judoka/92660/Daria_Bilodid. The title says this search engine punishes modern websites (images, videos, MB of JS, I suppose), and this site looks scarce on text and heavy on images, maybe that's confusing the ranking. I certainly find the results very refreshing, but you'll have to complement with other search engines if they're not enough. In fact, I think the days when we could use a single search engine have already passed.
• dzink 5y ago
  One use case to always test: "online wishlist" or "make a wishlist". If you start seeing tools like https://www.DreamList.com https://www.DreamList.com or others, you are on the right path. If you start seeing random web pages linking to individual wish lists, then people are likely not able to find tools on your search engine.
• dmje 5y ago
  Love it. You should provide a link to Patreon / whatever so people can support you financially. Hosting is probably not cheap for you. Given the love here on HN I suspect you'd do well.
  • marginalia_nu 5y ago
    Hosting is actually surprisingly cheap, but that's because I'm hosting it on consumer hardware in my living room, off my domestic broadband connection. That's both a blessing and a curse. It works okay as long as I don't touch it, but I can't do maintenance without shutting it down. I can't implement crawler changes without a week of shitty results as it needs to visit half the Internet before it gets decently good. I can only afford a production machine, so all testing that can't be done with unit tests gets done there. Anyway I added a patreon in case anyone wants to toss a coin.
• rchaud 5y ago
  Is it fair to assume that text-heavy sites that are inactive (but still online) don't have SSL? If so, would you ever tweak the parameters to surface sites that that aren't served with "HTTPS"?
• runnerup 5y ago
  Seems like this is still very very hard! I searched for "hart protocol" hoping to find this: http://www.romilly.co.uk/ http://www.romilly.co.uk/
• dfdz 5y ago
  I like the concept, but I did not work on any of the search phrases I entered consisting of the full title of a computer science article or book. It also does not work for subjects. For example, if you search "discrete math" it links to academic webpages, but most of them do not have any notes posted. It is just a plain text website with the syllabus of a class.
• Paul_S 5y ago
  Looking for an arm assembly instruction, instead I get this strange website as the result http://mailstar.net/coronavirus.html http://mailstar.net/coronavirus.html Is that accidental or is this website promoted because it's text heavy and will surface for any search without many results?
  • marginalia_nu 5y ago
    Looks like that page just has an absurd amount of keywords. Those sometimes surface when there isn't any good results. Haven't found a foolproof detection method that doesn't unjustly punish innocent pages with large amounts of content.
• sealthedeal 5y ago
  lol this is great, reminds me of the old school search engines we would use in school back in the day before Google haha.
• enduku 5y ago
  Fantastic project! Found very interesting links to a lot of compiler related keywords. A similar service, yet different in their approach to cut through the e-commerce and seo optimized websites I found useful is MillionsShort[0] millionshort.com
• fsckboy 5y ago
  I've read most of the comments here and people are evaluating the search results: all good information. I'm looking at "punishes modern web design"... This thing IS modern web design. I think it's called "marginalia" in reference to the huge margins they chose! I'm using a browser on a linux desktop and side-by-side, HN's page design is old-fashioned tasteful making pretty good use of space, and maginalia has a font that's more than twice the 2D pointsize and is so spread out with whitespace that the "Tips" on the home page are off the bottom of my window.
• wolpoli 5y ago
  I wish niche search engines has an option to group results by domain names. There are a few major sites that dominate Google search results with low effort content. As long as Google stands as the largest search engine, it's unlikely that these major sites will want to rearchitect itself into different domain names.
• throwawaysea 5y ago
  Is it possible to also make a site that favors a diverse set of information sources? For instance a lot of searches turn up results from Pinterest or Wikipedia or Amazon or whatever else. I wonder if there's room for a search engine that is all about favoring a greater diversity of smaller sources, for those who are less interested in staying within walled gardens.
• jteppinette 5y ago
  This is awesome! We should definitely move in this direction.
• optimalsolver 5y ago
  I predict it will return a disproportionate amount of sites by schizophrenic conspiracists.
• turtlebits 5y ago
  Great for a text-focused site- however, the results are a bit confusing. Would help if there were more details on the criteria used for a site to be included in the index. Suggestion - Use system fonts (the site downloads almost 300k of fonts)
• ryankrage77 5y ago
  The results are fantastic, but I can't see how the excerpts relate to the search term. For example, a search term for 'Scotichronicon' returns some fascinating results, but the search term itself doesn't appear in the title or excerpts of most of the results. This makes it harder to judge how relevant they are.
  • marginalia_nu 5y ago
    The excerpts are static and very best effort. You just have to visit the website and find out I'm afraid. I can do a lot with what I have, but I can't do full text search on millions of documents with dynamic excerpts off a single computer in my living room.
• b-x 5y ago
  Too bad it rejects non-Latin words, as if the definition of "text" is a sequence of alphabetical letters originated from Latin. I thought that we've reached the time to embrace all cultures in the world, but this retrogressive engine proves that most modern tech designers are myopic about other civilizations in the globe.
  • fghfghfghfghfgh 5y ago
    It’s one guy. Making a useful tool. It even has an altruistic purpose. Shame on you for twisting a well intended effort into a negative statement that suits your narrow identity political world view.
    • b-x 5y ago
      Less insulting error messages would be more welcome than casting out others without any consideration. You may distribute "shame" however you want, but this only helps enforcing the damaging insults and amplifying them.
  • hombre_fatal 5y ago
    No, it just proves that a one-man hobby project with finite resources found it reasonable to restrict the scope. Maybe when they find out they're an immortal billionaire they can build all the additional things you've entitled yourself to expect from the freely shared work of others.
  • marginalia_nu 5y ago
    Understand that this is something I built for myself, by myself, so it focuses on languages I understand. It hosted on a single consumer grade computer in my living room. I built it out of pocket and anyone is free to use it. Does this make me a villain? If I can do this, what's preventing some guy in Japan or India or Peru from doing the same, of course focusing on their languages?
    • b-x 5y ago
      Maybe a better suited choice for errors than an insulting message: when I provided a query in my native language it regurgitated the error "needs to be a word" instead of more acceptable "not a supported language". When you claim that a word in some other culture is not "a word", just because it's not recognized by your machine, that's demeaning to say the least.
• llbeansandrice 5y ago
  Is there anyway to add this as a favored search engine in the browser? I currently use google as it's set as the default search when I type in the address bar but would love to switch and move google/ddg to a added character like "<search terms> @g"
  • tomerv 5y ago
    All the major browsers support adding custom search engines. You just need to specify the URL template to do the search. The common format is to put "%s" as the search term. You can use it for any site, not just things that are considered search engines. Firefox is a bit different, since you do it by adding a bookmark, and giving that bookmark a keyword. The other browsers I checked have an option under the search engine settings. After defining the custom search engine, you just type "<keyword> <search term>" in the URL bar.
    • llbeansandrice 5y ago
      Ah I'm on FF so I'll have to do the weird bookmark method. A little annoying since it supports other search engines.
    • freddref 5y ago
      Is there a way to set marginalia as default search in firefox?
• tomaszs 5y ago
  I like the concept of a search engine that does not try to figure out what I should learn based on what I search..I know what I search for
• sabujp 5y ago
  effort is good, but needs some work, no results here : https://search.marginalia.nu/search?query=rxjava+2+api+docs https://search.marginalia.nu/search?query=rxjava+2+api+docs https://www.google.com/search?q=rxjava+2+api+docs&oq=rxjava+2+api+docs https://www.google.com/search?q=rxjava+2+api+docs&oq=rxjava+...
  • marginalia_nu 5y ago
    It is very much a work in progress, still struggling with some areas. I only really got into the territory of "sometimes actually useful" like this weekend. Wasn't planning on blowing up on HN just yet.
• prewett 5y ago
  Kudos for taking on this project, and I like the idea! I think it'll be a big project to take it to the next level, but would love to have a search engine that's more useful. Some reactions: - The font is really big and the columns really narrow, so I get 3 - 4 entries per page, something like 8 words per line, and huge spacings between lines, which makes it a frustrating experience. I've been using the recommendations in https://practicaltypography.com/ https://practicaltypography.com/, which recommends 60 - 90 characters in a line I think, and line spacing of 120% - 140% (I like 125%). The line lengths here might technically fall within the lower bound, but it's really short, and for search results I'm going to try scanning the text to see if there's something relevant, so I think going on the long side is better here. At least make the width somewhat variable so that I can shrink the rather large font and fit more on the line. - The results are eclectic, but I'm not sure it's usable at the moment. "scala append list" did not get me much that's helpful, while Google will usually at least put up some click-farming tutorial that although minimal effort does tend to answer the question. Both "mapo doufu recipe" and "ma po do fu recipe" had very few recipes, although the latter did have one. Unfortunately, recipe websites are some of the worst, with about 10 pages of description, ads, pictures, what-have-you until the recipe at the very bottom. "collection unmitigated pedantry" did return the acoup.blog entry at the top, though. Good luck on the project! -
• PieUser 5y ago
  searching for covid gives a bunch of bogus crap of fake news
• deleted 5y ago
  [deleted]
• ephbit 5y ago
  Quoted from the linked site: > Convenience functions have been added, and the search engine can now perform simple calculations and unit conversions. Try 1 pint in cubic centimeters, or 50+sqrt(pi). This functionality is still under development, be patient if it doesn't work. Why would you make any ever so small effort to implement calculations? I don't get it. If your search engine enabled me to find more useful search results to my queries than google or yacy or whatever, I wouldn't care one tiny bit about being able to do calculations with it. Why not focus on the search functionality?
  • exporectomy 5y ago
    How else do you do unit conversions? I use Google because it's far easier than any other software I've tried. Mainly because it's more forgiving of errors. It knows that "34 fset in msters" is 10.3632 meter. This search engine isn't, though, so I wouldn't waste time trying to discover its unit conversion syntax rules.
    • samhh 5y ago
      On macOS for example I'd use Spotlight.
    • soheil 5y ago
      You can also use the Chrome address bar without hitting the enter, just start typing
  • marginalia_nu 5y ago
    I implemented calculations because easily 80% of my google queries are calculations, unit conversions, etc. Search functionality is larger priority. Calculations and unit conversions were an afternoon's break from the search functionality :-)
    • ephbit 5y ago
      Ok, I guess people use google (or search engines in general) differently ... I rarely if ever use a search enginge to calculate stuff or do unit conversions. I use google only to search and when ecosia/bing doesn't return anything useful.
• exporectomy 5y ago
  Wow. I tested it on recipes which Google has destroyed and this was the first result, a simple clear recipe: http://demont.myds.me/leerecipes/mainmeals/mainmeals1/chickengarlicveg.html http://demont.myds.me/leerecipes/mainmeals/mainmeals1/chicke... compared to Google's endless drivel of "This chicken stir fry recipe will become a staple in your home. It’s so quick to make and you can use whatever vegetables you have on hand. It tastes wonderful regardless of how you alter the ingredients. ... " JUST SHUT UP AND GIVE ME THE RECIPE! https://natashaskitchen.com/chicken-stir-fry-recipe/ https://natashaskitchen.com/chicken-stir-fry-recipe/
  • srcreigh 5y ago
    Fwiw this website, Natasha's kitchen, seems to be one of the more performant ad filled recipe websites. Make use of the "Jump to recipe" button to get to the recipe faster.
  • 1f60c 5y ago
    This isn’t entirely Google’s fault. Recipes on their own aren’t copyrighted in the US, and adding this fluff text is a way around that.
    • AdamN 5y ago
      Yes it is. They're giving a lower quality result to the user (their customer ... but not for long if competitors can get just a little bit better)
      • isubkhankulov 5y ago
        A small nit: I think Google’s customers are actually the companies paying for ads. Its an important distinction and probably explains why their search results’ quality has gone down
    • mdoms 5y ago
      It is entirely Google's fault. Google knows people don't want this dreck (everyone knows it) but still serves it up.
• beepbooptheory 5y ago
  need a duck duck go `bang!` for this
• drusepth 5y ago
  Interesting approach. I always search myself on new search engines to compare the results. Most engines return my personal blog/website, books/stories I've written, news stories, my github projects/contributions, social links, etc. This search engine surfaces just three obscure IRC logs that contain my nick in join/part messages (nothing said from me!) from 2009. And nothing else. There's probably some things this approach is really good at but I'm not sure what they'd be for me off hand. Always cool to see new approaches to search, though.
• leavenotracks 5y ago
  Really impressed with the results I’m seeing so far. In all searches I have done so far, the results are truly lightweight, and haven’t had to click through any modals, subscription pop-ups or any other junk thus far! Will be using more in the days to come.
• arduinomancer 5y ago
  Wow this is immediately useful If you figure out some sort of funding model (maybe even just Patreon) I could totally see this as a viable side project Already discovered this recipe site: https://based.cooking/ https://based.cooking/ I love how adding recipes is through pull requests: https://github.com/LukeSmithxyz/based.cooking/pulls https://github.com/LukeSmithxyz/based.cooking/pulls
  • MrBoomixer 5y ago
    Thank you for this, it really makes me love the web and the people making things like this, Forked!
  • marginalia_nu 5y ago
    I've added a patreon, and a few people have hopped onboard. I'm honestly a bit stunned at the reception this has gotten.
• rafael_c 5y ago
  I liked this one... I searched for 'George Harrison' and among the first results there was a page with interesting comments about Harrison's solo career; someone reminiscing about the time they got to talk to him about guitars for half an hour at a bar at the airport; a transcript for an interview he gave on TV... Whereas on GOOGLE: an instrusive 'People also ask' which I was not interested; thumbnails for videos on youtube that I was not looking for; previews to garbage clickbaity news articles; and then finally for the search items: a bunch of websites for lyrics; his Instagram (!) and fb pages; his imdb page; some more news articles I was not looking for... Granted, google's web results above are perhaps what people are looking for 75% of the time, but how limiting and boring. I'm also a sucker for the simplistic text-centric, information-laden pages from the pre-facebook era. For 'global warming', however - since Marginalia excludes modern web-design pages - the results are of dubious relevance and interest, since they are, well, 'old'. I see myself using this engine a lot.
• amznbyebyebye 5y ago
  Wow. Love this. Searched for “Ramanujan”, one of my heros. Found this gem- https://math.ucr.edu/home/baez/ramanujan/ https://math.ucr.edu/home/baez/ramanujan/ Ramanujan’s “easiest” formula. Awesome!!
• rukuu001 5y ago
  This is great. The results for my search were like a suggested reading list.
• dzhiurgis 5y ago
  Search for playwright waitForSelector and you land in pretty useless page. I'm all in for text websites, but something like playwright.dev documentation is top notch - fuzzy search being key thing.
  • marginalia_nu 5y ago
    Yeah I wasn't really planning for this to blow up like it did today. It's currently sitting at about 35% of the index size I usually aim for, so besides the stuff I can't index because it's behind CDNs, there's a lot of pages it just hasn't gotten to yet. playwright.dev is pretty low on the priority list because it has a metric crap-ton of javascript on its front page. The crawler has visited it, looked at it, and put it very far down the priority queue.
    • soheil 5y ago
      Even though some sites have a metric crap-top of js they sometimes render very minimally for certain screen sizes or mobile devices without any of the js crap. Does your crawler pay attention to any of that?
      • marginalia_nu 5y ago
        It doesn't look at what the javascript does, just how much there is.
• phreack 5y ago
  One nitpick that kind of bothered me - on a large desktop monitor, the results page was like 70% whitespace margins with the results squished in the middle like a portrait cellphone. Hopefully it's easy to fix, I like to research at home and this website could help a lot!
• adriangrigore 5y ago
  A little bit harsh "punishes". It's a cool search engine.
• horsh1 5y ago
  No cyrillic or hiragana suport :-(
• jerhewet 5y ago
  I use webcrawler.com, and IMO it's better than any other search engine for finding exactly what I'm looking for. Not what's "trending", or "popular", or what the sheeple are searching for. It finds the exact matching keywords that I'm looking for. No inference or other bullshit -- just the matches. Such a relief to not wade through oceans of worthless crap any more.
• oytis 5y ago
  Designed for serendipity indeed. Tried a few searches, results are quite fun, but none of them relevant.
• ilrwbwrkhv 5y ago
  This is soooo good. I'm finally finding sites I haven't heard of with good content. I didn't realize how much I missed this stuff. The popular web has become so bad nowadays.
• JohnJamesRambo 5y ago
  Saving this forever. Thank you for making it.
• aetherspawn 5y ago
  Beware, I got the impression straight away that some sites were censored from the results for no good reason. For example, if you search "jehovahs witnesses", all pages from jw.org are missing. Exactly the same thing happened when I searched "mormons" - the official website is missing and it only brings up sects/hate/conspiracies against mormons.
  • edrxty 5y ago
    jw.org appears to be the kind of modern web design this is trying to avoid. I seriously doubt it has anything to do with the cult.
  • marginalia_nu 5y ago
    If you want mormons in positive light, you should search for "latter-day saints", as that's how they typically brand themselves. jw.org is on the indexing list, just pretty far down, based on the fact that previous times it's been visited it's had ton of javascript. I don't have an axe to grind with fringe religious movements, I actually love them to bits. Try searching for Nag Hammadi or Hermes Trismegistos.
• kevinob11 5y ago
  I tested this with "Caribbean Vacation" and wow what a difference. Everything on Google is "TOP X LIST" and "BEST XYZ" which are just the worst when trying to find real interesting information about experiences you can have on vacation somewhere. I had used those as starting points then searched for long-form blogs of real experiences people have had. This surfaced those kinds of things immediately. I love it.
• macksd 5y ago
  This is a really cool idea. I tried a few technical queries I did on DDG today and didn't get amazing results - hence the warning in the About page about this engine giving you things you didn't know you were looking for, rather than specific facts. But the examples others have posted sound promising and refreshing. I would love to read about the algorithms behind this and how modern web design gets detected in order to punish it...
  • marginalia_nu 5y ago
    Yeah I think you typically have the most interesting results searching for older topics. I've posted a few comments outlining the tech, check my profile. It's also outlined some details in my blog if you dig around a bit, there's no real comprehensive overview, just bits and pieces though. https://memex.marginalia.nu/topic/astrolabe.gmi https://memex.marginalia.nu/topic/astrolabe.gmi
• grae_QED 5y ago
  Obligatory https://wiby.me/ https://wiby.me/ plug. If you're looking for a decent minimalist website search engine, this fits the bill pretty well.
• NmAmDa 5y ago
  Lets get the the internet great place foe knowledge again. I really loved the engine ans tried for different terms and very happy. Goos job
• lishzen 5y ago
  Love it! Searched for Shigeru Miyamoto and this was the third result: https://www.glitterberri.com/developer-interviews/miyamoto-horii-discussion/ https://www.glitterberri.com/developer-interviews/miyamoto-h...
• kag0 5y ago
  An interesting concept and awesome work! I searched for high pressure air (HPA) regulator trying to find a description of how one works. I didn't find that, but did find some interesting links on how they're used in scuba, and one guy's homemade gas laser.
• seoulmetro 5y ago
  This would be awesome if the search actually worked. Typed in 'runescape' and expected a few websites left over from the early 2000s. But I got nothing, just a lot of hits to other keywords.
  • marginalia_nu 5y ago
    Huh. That's an interesting case you've found. I think part of the problem is gold sellers were displacing all the good results. I blocked a few of them and got a few more relevant results, but it's not great. But still, the search engine only finds two great hits. I wonder why not. Maybe there just aren't that many runescape pages around still? Or it may just be that it hasn't found anything better yet. The index is pretty shallow right now, only 20M URLs, I aim for more than double. I honestly wasn't planning on this blowing up on HN at this stage.
• Valkhyr 5y ago
  As a quick test, I searched for the name of one of my favorite game series: "Baldur's Gate" (on its own, no qualifiers, properly spelled - I would usually spell it "baldurs gate" on Google, but I decided to give this one the best chance). I search for info around video games a lot, so that's quite representative of a good chunk of my web searches, and I pretty much know the top sites Google would give me for that query (on its own, without any further qualifiers). The results were all either barely relevant, outdated (sites that covered the game back in the 90s/2000s before it was re-released), at best tangentially relevant or complete garbage noise. Some of the most highly relevant pages (such as the Steam store listing, the fandom wiki, the publisher/developer's forums for the re-releases, the Baldur's Gate 3 website and the subreddit) were not included at all. Those are all fairly text heavy by any reasonable standard, so I assume they were "punished" because they use JS? Would make sense that nearly all of them are way out of date. Then I searched specifically for "Baldur's Gate Wiki" but still out of luck - some results, but nothing vaguely Wiki-like. Finally I searched for "Baldur's Gate Fandom Wiki". This is basically "search engine easy mode", by giving essentially the name of of the site I am looking for. I got ZERO results. At this point I gave up and decided that this thing is useless. Look, I'm all for unearthing good long-form content (in fact I would say that much of the content around this specific game would qualify), and I do get as annoyed at modern SPAs as the next grumpy neckbeard. I think considering both of those in a search engine is not a bad idea in and of itself. But I have to wonder what's the point of a search engine that weights some arbitrary aspect of web design higher than the relevancy of the subject matter (to the point of not returning any results at all)? In fact, considering that generally speaking more recent websites tend to include more scripting, you are intentionally skewing the results towards (very) old content, which is probably doing the user a disservice.
  • matesz 5y ago
    > But I have to wonder what's the point of a search engine that weights some arbitrary aspect of web design higher than the relevancy of the subject matter (to the point of not returning any results at all)? Because in some cases it returns arguably better/more to the point results, than other search engines – for example search for “Douglas Engelbart” or “Ted Nelson”. I thought that I’ve searched everything for those two yet marginalia gave results otherwise I would have never seen,.
• FriendWithMoon 5y ago
  As a sufferer of Tinnitus, and having spent near 100 hours researching it, I found a few sites I had never seen offering great data and tools. Thank you
  • marginalia_nu 5y ago
    Honestly it's probably not a great source for medical advice. At least take what you read with a healthy grain of salt.
• m1117 5y ago
  Love it! I can punish my employees by setting this as a default search engine on their work laptops.
• ahthat 5y ago
  Very cool idea! Room for lots of improvement, keep working on it, I like the direction this is going.
  • marginalia_nu 5y ago
    I just got it working reasonably well just this week. I've had it "working" for a few months, but the results were always extremely chaotic, bordering on random.
• spookthesunset 5y ago
  This is really cool! So retro! Here is the second result when you search for “cat food”. It takes you to some old dudes entire family tree with full history and biographies… it even uses sub domains and everything! Crazy! http://www.torrens.org/ http://www.torrens.org/
• necovek 5y ago
  Too bad the search index is currently restricted to ASCII-only (or at least Cyrillic and Latin-2 characters were rejected as "contains characters that are not currently supported"). I love the idea definitely, and I've long toyed around with building a similar thing that starts crawling off my own bookmarks (a personal small-deep-web if you wish). I also love the "Small Web" name: this is the first I hear of it, and it's what I've long complained about — the web today hides all of the cool gems search engines of old would have given you! I am also a bit split on the "www" prefix restriction (iiuc, domains which do not have "www" subdomain too are dropped from the index because many of them are spammy): it might for sure be a useful heuristic, but I've advocated for dropping "www" back in late 90s and early 2000s already (one reason being that for eg. Serbian, "w" is not in the alphabet, so you can't reasonably quote it as Serbian is otherwise a phonetic-language).
• talrand 5y ago
  Gave it a go with two different queries. The first I chose was “amazon vendor services” didn’t get a single result about the topic. The second query was a nation+city(in the nation). Got a lot of result that were in no way related to either. It seems to be biased towards IT topics (based uniquely on the two queries).
• bencollier49 5y ago
  This is brilliant - I can actually surf the web for fun again. This engine's actually a nice complement to another mainstream engine, as the regular one is good for searches during the working day, whilst "marginalia" (?) is great for recreational reading, and actual learning.
• raflemakt 5y ago
  I tried two searches in Norwegian ("norsk ordbok" [norwegian dictionary] and "stortinget" [the parliament]), and they both returned many extreme or "alternative" websites. It was especially striking that the neo-nazi group Vigrid's website was the top hit for both searches. Maybe these sites just have less modern web design?
  • marginalia_nu 5y ago
    Yeah this is actually a bit of a concern of mine. As very much a friend of Voltaire's, I don't think it's my place to police people's opinions no matter how disagreeable, but I also don't want my search engine to become branded as the search engine of choice for nazis because it's decent at cataloguing extremist sites.
• protontorpedo 5y ago
  I searched for "Starlink satellites" and found this Y2K-style Canadian UFO blog [1] explaining it isn't aliens. I might just waste my weekend with this search engine. [1] https://www.ufobc.ca/Reports/stringoflights.html https://www.ufobc.ca/Reports/stringoflights.html
• chx 5y ago
  Not sure what to do with this. https://search.marginalia.nu/search?query=gan+charger https://search.marginalia.nu/search?query=gan+charger aside from nexperia none of this looks even remotely relevant.
• tentacleuno 5y ago
  I'm all for nostalgia but IMO the information should be the most important thing. Presentation is a close second though, and I can kind of get behind this project when I see what the web is like without an ad-blocker.
• sireat 5y ago
  Fantastic idea and it works quite well for short phrases that I tried. As expected I am getting a lot of early 2000s sites which is something that I miss on regular Google. Hilariously searching for "array data structure" got me one of the top results this little tiny page: http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html
  • marginalia_nu 5y ago
    > We have designed Google to be scalable in the near term to a goal of 100 million web pages Funny, that's about where I see my search engine capping out as well.
• lazybreather 5y ago
  Amazing! How do I make this my search engine on browser? Not home page.
• jokoon 5y ago
  I don't even want to imagine how google and other search engines crawl websites that make heavy usage of react or other ajax stuff. I don't want to be that guy. I wonder if some browser engineers are trying to have some ideas on how to find a solution on this. Personally, I would just make a browser that breaks backward compatibility, remove old features, etc. I guess browsers would be much lighter, fast and simple if some hard choices were made. Mozilla already decided to break some websites with the strict cookie policy. I wish they would do the same for everything else that sucks on the modern web. I honestly don't think I have much respect for "web developers". In a way I want mobile apps to kill the modern web, just to prove a point.
• tejtm 5y ago
  There is an open standard way for an engine like this to provide a mechanism for your standards aware browser to add the site as a alternative search with a click. That way I would not have to remember or bookmark just use my search bar as normal and choose which engine for this query or set it as default. []https://developer.mozilla.org/en-US/docs/Web/OpenSearch https://developer.mozilla.org/en-US/docs/Web/OpenSearch
• AltruisticGapHN 5y ago
  I like the idea. However results take too much space vertically it's slow and cumbersome to scan through them. I think it would benefit from using a responsive layout, allow the text expand to a wide 1000+ px, make the font smaller, so the excerpt can fit one or two lines below the links. Google has problems but their search results layout is easy to scan. Otherwise I genuinely wish I would use it, because the Google search's "self referential reality bubble" is really annoying.
• hop34s3w 5y ago
  If the website is targeted towards international audience then its nice to have the first page links to content in english. All the four links in the main page https://www.marginalia.nu/ https://www.marginalia.nu/ have links to non-english content which is not useful. Disclaimer: I am not a native english speaker. English is my second language.
  • marginalia_nu 5y ago
    Yeah my main site is a bit of a disorderly mess. It started as a Swedish blog, but I've since added a few services aimed at a global audience. Haven't quite figured out how to unify it all just yet.
• hilbert42 5y ago
  It's excellent, I looked up some physics topics and got some excellent results - real meaty stuff full of text, eqations and applicable diagrams, etc. I've not only bookmarked it but also I've an icon linked to it on the taskbar. Will watch its progress with interest.
• davuinci 5y ago
  Congrats for the effort, I really like the idea and it works wonderfully for some searches. However, I searched "infiniband" and the results are far away from what I would expect or like to see. Most of the results that appear first are completely unrelated to the topic.
• zvxczvvzxzcxzm 5y ago
  This is great! I tried with "covid tyranny", and got some very interesting results I'd never get with any of the other search engines!
• Method5440 5y ago
  A big fan of your work! Just wanted to let you know what iOS devices provide quotes as “ rather than " - you may need to support the character “ or at least let people know that iOS is not supported etc… right now I get a generic character error.
  • marginalia_nu 5y ago
    That's very good to know. Thanks!
• megraf 5y ago
  Thank you so much. This is wonderful
• wizzwizz4 5y ago
  Add an OpenSearch file: https://developer.mozilla.org/en-US/docs/Web/OpenSearch https://developer.mozilla.org/en-US/docs/Web/OpenSearch Maybe something like: <OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" xmlns:moz="http://www.mozilla.org/2006/browser/search/"> <ShortName>Marginalia</ShortName> <Description>Marginalia Search - a search engine that favors text-heavy sites and punishes modern web design</Description> <InputEncoding>UTF-8</InputEncoding> <Image width="16" height="16" type="image/x-icon">https://search.marginalia.nu/favicon.ico</Image> <Url type="text/html" template="https://search.marginalia.nu/search?query={searchTerms}" /> <moz:SearchForm>https://search.marginalia.nu/</moz:SearchForm> </OpenSearchDescription> and then add: <link rel="search" type="application/opensearchdescription+xml" title="Marginalia" href="/opensearch.xml" /> to your page's <head>.
  • marginalia_nu 5y ago
    It's been added a few moments ago :-) Thanks everyone who suggested this.
• Lapsa 5y ago
  I like the design.
• vanattab 5y ago
  Cool idea but it needs to be able to handle special characters. Right now searching "Hello World c#" returns no results because the search term can't handle #. I also can't just delete the # because then I would be stuck writing C...
• NuNotNon 5y ago
  I have an interest in logic and cs curriculum and i like Geneses in general(last days i've read intro in math phylosophy from Russell and some acm report of cs curriculum. I search for cs curriculum and this is the first link https://www.cs.rice.edu/~vardi/sigcse/ https://www.cs.rice.edu/~vardi/sigcse/ Feels so good to recive good answers so easy. Thanks.
• AQXt 5y ago
  I searched for "giraffe evolution" (without quotes) and received the following links on the first page: - Evolutionist scientists say the theory is unscientific and worthless - Seven Mysteries of Evolution - OTHER EVIDENCE AGAINST EVOLUTION - Evolution Falsified Not a single result about the evolution of giraffes...
  • phendrenad2 5y ago
    Unfortunately, as you've discovered, giraffes are often used by crackpots to try to disprove evolution. Google seems to get around this by heavily boosting known authoritative sources like National Geographic and NIH. But, sadly, those are JS/image heavy sites.
• BlackLotus89 5y ago
  Are there any technical infos about the search engine? Found some information here https://memex.marginalia.nu/projects/edge/about.gmi https://memex.marginalia.nu/projects/edge/about.gmi Author said they threw it together on consumer hardware. How big is the index? (TB used or entries) how is it realised? I'm pretty much interested in this since I myself am crawling some pages for my own "search index". Oh and thx for making and posting. Added it as a keyword to firefox Edit: Just realized that my question is a bit shallow. What I'm particular interested in is the storage before the indexing. I'm trying to store the raw html so that I can reindex everything with better algorithms, but I'm hitting many limits. It takes a few minutes getting the size of a site-directory (every site has it's own dir) and I'm at a point where I can't reasonably manage the scrape-versioning over git and I cycled through a few filesystems only to find that the metadata management kind of sucks for most of them. It's rather interesting how we store such files and I'm thinking about storing a few sites in a simple sqlite format for easy access and search. I'm thinking about a a few low overhead solutions like facebooks project haystack (implemented open source in seaweedfs) or something similar... Hopefully this gives some context to the question of storage and sites that are indexed
  • deleted 5y ago
    [deleted]
  • marginalia_nu 5y ago
    The index is tiny, not even a terabyte. Right now it's a few hundred gigabytes for ~20 million URLs. But it's stored in an extremely dense binary format. Honestly you may just want to roll your own solution for storing a ton of files. If you don't need a general-purpose filesystem, but an append-only archive with extra metadata, then you can cut a lot of corners. Like if you have a file system that is fixed-size and append-only, you can build it in a way no off-the-shelf stuff can. This line of thinking is a large part of why my index is so small and fast. I have a lot of special built data-structures that are built for their exact use case. Like a fixed size append-only hash map that uses mapped memory and can in theory be larger than the system memory. Very good for a search engine, absolutely useless almost everywhere else.
• jccalhoun 5y ago
  It says it punishes modern web design but it has my most irritating feature of modern web design: a narrow strip of text on an otherwise blank page.
• SlapperKoala 5y ago
  I like the idea but could use some tweaking. I keep getting conservative christian websites for some reason. And foreign language sites
• smoyer 5y ago
  You can add this to Firefox as search engine option by right clicking on the URL and selecting "Add Marginalia". From there, setting it as your default search engine is done from the "Settings" panel as with other predefined search engines. I'm experimenting with using it as my default ...
• voidnullnil 5y ago
  How does this have 2.5K upvotes when every single HN related project needs JS and a quad core CPU (for the browser to open a blank page) to view a paragraph of text?
• billyharris 5y ago
  Search engines always like websites with more text and less graphics.
• xtiansimon 5y ago
  Does this also penalize pages with tons of ads and three paragraphs of text? Or anything from Medium?
• godshatter 5y ago
  Instead of looking for something specific, I decided to try a category of some sort to see what came up. Thinking about Jeopardy categories, I tried "potent potables" and found a lot of random pages that may or may not have made sense given that category but that I had a lot of fun reading. Definitely a win for me.
• AlexCoventry 5y ago
  I don't really care about website's design, as long as it gets out of the way of me reading it.
• SimplGy 5y ago
  Searched for “chocolate chip cookie recipe” First result had a recipe I could see both recipe and directions in a single page, no ads, no scrolling, no fake seo anecdotes about kids and grandmas. (Pls make the search query box fit small mobile devices) Great project idea!
• scopio918 5y ago
  All search engines favors more text and less graphics.
• JohnFen 5y ago
  Oh, this is brilliant! I think I'll make this my "first stop" search engine.
• amelius 5y ago
  Question: how do we benchmark search engines? Are there any groups attempting to provide (open) solutions in this space? (It seems to me that if you want to build a good search engine, this is the question you need to address first.)
  • X6S1x6Okd1st 5y ago
    The search term you might be looking for is "information retrieval" there are pretty standard measurements for whether you are getting good results, but they are generally conditioned on stuff like click through rate, comparing to expert ranking and other signals that the user gives you that it was a good or bad return of search results.
  • marginalia_nu 5y ago
    The benchmark clearly must depend on the purpose of the search engine, though. If the purpose is to find a thing you know is out there, but haven't been arsed to bookmark, then my search engine is hot garbage. If the purpose is to provide links you will find interesting when you click on them, my search engine really actually has its moments. I guess it's almost closer to old reddit, before it was overtaken by astroturfing and bots.
• carol_wang 5y ago
  wow super cool!
• stevenjgarner 5y ago
  How do you submit a site? I searched for "A search engine that favors text-heavy sites and punishes modern web design" but nothing was found.
• 1vuio0pswjnm7 5y ago
  I had an idea for an alternative search engine a few years ago thats a bit simpler to implement than this one. First we extract all the external links from Wikipedia dumps. Then we ingest only those sites into the index. The entire database comprises sites that have already been screened by Wikipedians. Gaming Wikipedia is generally more difficult than gaming Google or Bing. In theory, SERPs would be of a less commercial and more substantive nature than those we get from Google and Bing.
• th423i4234 5y ago
  This is amazing!
• more-coffee 5y ago
  Not sure how to contact you in case of possible bugs/problems, just going to drop a comment here. Was trying a bunch of "define:" queries and noticed something small; This works https://search.marginalia.nu/search?query=define%3Ahallucinating https://search.marginalia.nu/search?query=define%3Ahallucina... This doesn't, or at least it's empty https://search.marginalia.nu/search?query=define%3Ahallucinate https://search.marginalia.nu/search?query=define%3Ahallucina...
  • marginalia_nu 5y ago
    Yeah it's probably not the most complete dictionary. I'm using Wiktionary-data. There might be the occasional gaps and vandalism.
• Syzygies 5y ago
  After a few experiments, I'd compare this to panning for gold in a river others ignore because the yield isn't good. You will find nuggets you would have otherwise missed, but you'll work for it. This engine is likely best used to supplement other options. What set aside Google from its competitors was its use of eigenvalue weights. I don't sense a robust weighting system in use here.
• Igelau 5y ago
  This came back in a result set and I am thoroughly pleased. http://www.aliens-everything-you-want-to-know.com/ http://www.aliens-everything-you-want-to-know.com/ This is the Information Superhighway in all her glory.
• sureklix 5y ago
  So is this a filter on top of Google or is it search from scratch? Would love to understand more of the implementation.
  • marginalia_nu 5y ago
    It's built from scratch. I'm doing the crawling and indexing. Look through my comments and you'll find a few outlines of the stack and the index design. Here are my blog entries relating to this: https://memex.marginalia.nu/topic/astrolabe.gmi https://memex.marginalia.nu/topic/astrolabe.gmi
    • CreCre 5y ago
      This isn't meant to be pressurizing or to sound like a demand (if it seems that way), but have you thought about uploading the source code for your search engine? Something like this has the potential to be used in university courses to teach how to build a search engine and/or teach 'advanced' programming concepts and ideas. It's a real program showing what you need to do to optimize your database and software to work on consumer desktops (even if your specs are higher than what other people would have; 128 GiB for example is quite a lot of RAM for most consumers) and how to handle malicious data that you will come across (for example, link farms). In addition, I read all the posts on your site that were listed in the page you linked to, and to me those posts would actually seem more useful as an explanation of the code that people can view together side-by-side, rather than as the only way people can know how you implemented your algorithms and search engine. I guess what I'm saying is, having an explanation in words of the algorithms and code along with the actual code can be a very powerful combination for teaching and learning. Thus, again, would it be alright if you upload a copy of the source code for people (including myself) to look at? I personally don't care about if it's released under an open-source license (or not), or if you just add a zip file on your site vs making a repository on Github, or even if you never update the code you release. I (and most likely others) want to peek at at least one version of what you wrote to see how something like this works under-the-hood, which again, I'm asking if that's alright with you. Also, I'm not asking you to share the database(s) you have for this, especially since they're giant and would likely take up more traffic downloading from your site than anything the search engine can do.
• infogulch 5y ago
  Not to overemphasize meta commentary, but damn 3200 points, 650 comments in 2 days -- this is one of the highest rated posts I can remember. Seems HN readers are very interested in alternatives to the current search hegemony and the kind of low-quality junk articles that litter it.
  • marginalia_nu 5y ago
    I'm still a bit stunned at the reception this has gotten. I wasn't even planning to launch this, not like this, there's so much that needs to be fixed to get it working good, so much jank and so many weird limitatons. It's quietly been online for a few months. But it's not really worked all that well except in rare cases. Then I fixed a few issues and implemented some improvements, and it was really just last weekend that I was struck by the sense that it actually was coming together into something really viable, and then... this happened. I've gotten so many positive comments about this project, a large number of emails and I haven't had the time to get back to half the people who wrote, even people donating money to support it. I half thought I becoming the next TempleOS-guy, hacking away at some madcap scheme all by myself. I just had no idea this resonated with so many people. It's incredibly encouraging and motivating. Thanks everyone!
• mikkel125 5y ago
  I read from other comments that you're writing your crawler bot yourself. Instead of crawling "from scratch", have you considered using an existing DB like Commoncrawl? Or is there something else that you index not present in Commoncrawl?
• xhrpost 5y ago
  > This search engine isn't particularly well equipped to answering queries posed like questions, instead try to imagine some text that might appear in the website you are looking for, and search for that. Heh, I guess I'm getting old but I remember when this was the only way to search the web
• mlang23 5y ago
  This is absolutely great. Seeing this for the first time somehow reminds me when I learnt about Google. many years ago. The salespitch for google back then was "They have all the linux related docs and infos indexed". The promise of this engine seems even more promising. A search engine specialized in text, look 'ma! I hope this grows and gets the user attention it deserves. Google has becomes so ad-infested in the last 3 years, its time something replaces it.
• ohuf 5y ago
  I don't think this is a good idea: when I'm searching on the web I want to get results with a high relevance to my search query. I don't care a lot about the presentation.
• interleave 5y ago
  Wow, I'm floored by the quality of the search results. For example, I'm a huge fan of obscure Brian Eno stories and interviews and articles. Using the default blend for "Brian Eno", I found http://www.moredarkthanshark.org/eno_interviews.html http://www.moredarkthanshark.org/eno_interviews.html which is truly a labor of love and the most comprehensive list of articles (with links to them, too) I've ever seen. Not in a hundred years would I have found this using Google. Thank you for building this!