14 ms·
Marginalia: DIY search engine that focuses on non-commercial content
- themodelplumber 4y ago> The Random Mode has been overhauled, and is quite entertaining. I encourage you to give it a spin. Yep, this is a good example of warping the Feeling Lucky pattern into a really neat little discovery tool. IMO it would even be cool if the site was this part first, oh and hey it's also a search engine. (While I'm random-ing: The Arch Wiki is in there? Seriously? Just for that, I propose that it either be skinned to max vaporwave, or host a webcam pointed at a Manjaro machine, or both...I'll be waiting over here, downloading 4.1 GB of marginalia for my AUR build of PCManFM)
- BiteCode_dev 4y agoStumble upon used to do this. It was really cool.
- justusthane 4y agoStumbleUpon was amazing. I feel like that was really peak internet, at least for me. I found so many weird, awesome things.
- marginalia_nu 4y agoIt really could only work around the time it existed I feel. The internet was a lot weirder back then. One big difference then from now is that you basically need a PhD in the Canvas API (or WebGL or whatever) to accomplish something a 5 year old could do in Flash. Web design was a lot more accessible. You didn't have to worry about responsive designs and fluid layouts. You could just position:absolute everything and that was kinda fine.
- giantrobot 4y agoI think you might have some nostalgia goggles on at the moment. There's nothing holding people back from making "weird" web pages today, they can even make them nice and responsive. One of the better concepts around HTML and CSS was separation of data and layout. It's trivial to have a "weird" position:absolute design with a break for mobile that switches to a more fluid layout. Desktop users can have their "weird" layout but I can still read the page on mobile and you can readily crawl and index it. People moved away from design tools like DreamWeaver that helped make "weird" stuff and instead installed WordPress or some CSS/JavaScript framework that just bakes in all the "boring" fluid layouts. You're not necessarily wrong about Flash in terms of design or creation but your search engine wouldn't be terribly practical if everyone was still using Flash for everything. Flash allowed content packed inside SWFs but also allowed fetching external resources. You wouldn't be able to index any of that unless your crawler executed the Flash and/or inspected all the URL references for external resources. Flash created an inaccessible deep web just like today's JavaScript website-is-an-application "sites". Don't get me wrong, I love the old web with quirky table-based layouts, "unofficial" fansites, and personal homepages hosted on forgotten university servers in a closet. There was a vibrancy that's largely missing from today's web. I think a big change has been tools have become more geared for boring than the creative and people treat content on the web as a side hustle. Google et all haven't helped by favoring recency over other relevance factors.
- counttheforks 4y agoIs it trivial enough that a 5 year old could do in a point and click editor?
- giantrobot 4y agoIt could be. But modern tools don't bother. Then again, Flash's usability by a 5 year old is being a bit oversold here.
- themodelplumber 4y agoThere are quite a few interesting alternatives these days. Bored Button is one. There are some that are even more like it but I'm away from grep and my notes at the moment. Reddit even has some kinda-similar subs.
- marginalia_nu 4y agoI haven't got the time to curate this stuff. There's like 10,000 domains in the list. It's some one off SQL script I think that generated the sample based on parameters lost to time.
- muyuu 4y agoare domains whitelisted?
- marginalia_nu 4y agoThe domains you get from browse:random is from a small selection yeah. But if you start traversing with "similar" there is no such limitation, only limit is that they must have a screenshot. (There's also explore2.marginalia.nu which is not even limited to websites with a screenshot)
- selfhoster11 4y agoReminds me of StumbleUpon. I still miss that.
- arcanemachiner 4y agoThat period gave rise to the most educational web surfing I've ever experienced.
- melx 4y agoAll seem nice until I get to see the search results.. which I cannot "read". It's very difficult to read an output that goes 5x boxes horizontally, and each such line goes then vertically on "forever". It's like yellow pages book from the 1990s.
- marginalia_nu 4y agoYeah, the "magic: the gathering" layout some limitations. I want something that makes good use of a large screen though. I've got some ideas in the pipe, but haven't had the time to give them enough polish that I'm happy with them. This is an early draft: https://imgur.com/a/vMVO7CK https://imgur.com/a/vMVO7CK
- melx 4y agoOh I see! It displays good on mobile devices. The draft looks nice. The text colour is a bit hard to distinguish from the surrounding background, and I don't have any eye conditions.
- marginalia_nu 4y agoYeah, the contrast is one of the things I'm not entirely happy with. The positioning is also a bit off, especially if you resize the window a bit. As stated, needs polish. But I really like the idea of the search engine being a bit more transparent with how it works.
- ssd532 4y agoThat's much better. would love to use it.
- marginalia_nu 4y agoEncouraging to hear. I'm not a big front-end enthusiast so it's useful fuel for slogging through the design process.
- Jemm 4y agoFirst search for a commodity item returned pages full of conspiracy theories and then drifted in to anti-vax territory.
- marginalia_nu 4y agoInteresting, what did you search for?
- HeckFeck 4y agoThe unfiltered Internet in all its glory!
- shanebellone 4y agoI yearn for an exclusionary Internet. Most voices don't matter.
- manuelmoreale 4y agoWould you mind explain 1) why you want that and 2) who decides which voices do matter?
- shanebellone 4y ago"1) why you want that" Universal broadcast does not work (beneficially for society) in an industry built to monetize reach. Everyone is entitled to their opinions, but voices are not equal in utility or worth. "2) who decides which voices do matter?" This is always the problem, isn't it? I don't have an answer for you.
- manuelmoreale 4y agoIsn't 1 just the result of 2? It's because we don't have an answer and because there probably isn't an answer to the second question that we need universal broadcast as you called it. We could get away from it only if we figure out an answer to the second question but I suspect we'll never get to an answer.
- marginalia_nu 4y agoWatch my computer struggle here: https://www.marginalia.nu/stats/ https://www.marginalia.nu/stats/
- antman 4y agoWhat is the stack? Can it scale up?
- marginalia_nu 4y agoCustom index software built from scratch in Java. MariaDB link database. The entire search engine runs on a PC in my living room. You could pretty trivially shard the index by `hash(domain) % numShards`. There's no support for this because I literally only have this single server, but it wouldn't be much work.
- deleted 4y ago[deleted]
- toyg 4y agohttps://github.com/MarginaliaSearch/MarginaliaSearch https://github.com/MarginaliaSearch/MarginaliaSearch
- CalRobert 4y agoMarginalia comes up on HN every so often, and I always look at it, think "oh that's neat!", maybe add it to my bookmarks toolbar, and then forget about it. Are there a lot of people who find themselves using it daily?
- yuhong 4y ago[dead]
- Firmwarrior 4y agoI use it as my daily time-waster, it has a lot more interesting stuff than you'll find on sites like Reddit or Twitter https://search.marginalia.nu/explore/random https://search.marginalia.nu/explore/random
- newqer 4y agoWhat have you done? I tried to cut back on Reddit, but this seems like I could go done a rabbit hole for a few hours per day.
- marginalia_nu 4y agoI don't even use it daily. It's not a Google replacement, and it's not trying to be either. It's more of an on-ramp for the obscure web at this point. That said, it's gotten way better at finding stuff with the last few releases.
- GravitasFailure 4y agoIt's interesting, and I really appreciate that you aren't trying to out-Google Google. This seems like a useful tool in its own right in addition to what else exists.
- marginalia_nu 4y agoI don't think that would make sense. If Google is struggling with search, a one man Google clone isn't going to do it better. I also think that having "a google", one central search engine, is inherently a bad thing for the health of the Internet. It drives a lot of this search engine spam epidemic we're seeing. A broader and (IMO) more interesting problem is Internet discovery.
- nullandvoid 4y agoI get no results at all for this seemingly simple https://search.marginalia.nu/search?query=how+to+draw+a+3d+box&profile=default&js=default https://search.marginalia.nu/search?query=how+to+draw+a+3d+b..., presume it's been HN hugged to death?
- shp0ngle 4y agonah it just have a tiny index. search for just “3d box” or something like that.
- melx 4y agoNo, it's really means "404 nothing found" for your search query. I searched for my company name and got nothing as well. A bit surprising since it says "search the Internet" ;-)
- marginalia_nu 4y agoIt doesn't do semantic search or synonyms. Think keywords, not questions. Search for "draw 3d box" or "draw a cube" and it starts giving results.
- PaulHoule 4y agoWow! Relevance looks good for queries i tried and i like the square interface too.
- cs702 4y agoLike HN, Marginalia is a fresh of breath air in comparison to today's SEO-optimized, monolith-dominated web. Is there a way to donate money?
- postdb 4y agoit is on the front page of their page: https://memex.marginalia.nu/projects/edge/supporting.gmi https://memex.marginalia.nu/projects/edge/supporting.gmi
- hk__2 4y agoIs it limited to English? I made a few queries in French and Italian and got either no results either a couple of irrelevant ones.
- marginalia_nu 4y agoYes. It's in part a measure to limit the scope of the project (the entire thing runs off a single PC), but it's also hard to build a good language model for a language you don't speak, and I only speak English and Swedish. But if the project grows, gets more hardware, and contributors that speak other languages, then maybe this will change in the future.
- cloudyporpoise 4y agoI've always been curious about how search engines seed their scanning and index programs. Like how do you know what domains, ips, etc.. to start scanning and where is the origin?
- ddorian43 4y agoStart with Common Crawl and go from there.
- djoldman 4y agoThat may be a much easier question to answer than discovery. How do you discover relevant new domains?
- marginalia_nu 4y agoI've actually sort of solved this recently. Marginalia's ranking algorithm is a modified PageRank that instead of links uses website adjacencies[1]. It can rank websites even if they aren't indexed, based on who is linking to them. Vanilla PageRank can't do this very well. Domains that aren't indexed don't have (known) outgoing links, in the periphery of the rank. There's a some tricks to get these to not mess up the algorithm completely, but they basically all rank poorly. That's even without considering all the well known tricks for manipulating vanilla pagerank. The modified version seems very robust with regards to both problems. [1] https://memex.marginalia.nu/log/73-new-approach-to-ranking.gmi https://memex.marginalia.nu/log/73-new-approach-to-ranking.g...
- marginalia_nu 4y agoIt's basically seeded with my personal bookmark list. Like a few dozen links. Not exactly this, but close enough: https://memex.marginalia.nu/links/bookmarks.gmi https://memex.marginalia.nu/links/bookmarks.gmi I've changed the crawler design a couple of times, but the principle for growing the set of sites to be crawled is to look for sites that are (in some sense) adjacent to domains that were found to be good.
- 4y ago
- djoldman 4y agoFrom my goto search term, "chickens": https://theoutline.com/post/5608/bury-me-in-chicken-diapers https://theoutline.com/post/5608/bury-me-in-chicken-diapers
- marginalia_nu 4y agoI'll counter with the top result for "cats": http://diabellalovescats.com/catland.htm http://diabellalovescats.com/catland.htm
- Aeolun 4y agoThank you. I needed this in my life.
- pxc 4y agoAnecdotally, all the people I know who have recreational/pet/indoor chickens are lovely human beings, so I'm wholly in favor of this absurd industry and its success.
- marginalia_nu 4y ago"Recreational Chicken" is a two word poem if I ever saw one.
- bookofjoe 4y agohttps://www.amazon.com/Under-Henfluence-Inside-Backyard-Chickens/dp/1572843217/ref=sr_1_1?crid=RNT56S7AIO3Y&keywords=%E2%80%9CUnder+the+Henfluence%3A+Inside+the+World+of+Backyard+Chickens+and+the+People+Who+Love+Them.%E2%80%9D&qid=1681836653&s=instant-video&sprefix=under+the+henfluence+inside+the+world+of+backyard+chickens+and+the+people+who+love+them.+%2Cinstant-video%2C61&sr=1-1 https://www.amazon.com/Under-Henfluence-Inside-Backyard-Chic...
- barbs 4y agoHow does it compare to https://wiby.me/ https://wiby.me/ ?
- marginalia_nu 4y agoWiby is manually curated. Means their result align much better with the operator's vision. Marginalia has an orders of magnitude bigger index, but not all of it is as consistently good as Wiby.
- repeekad 4y agoI remember being so excited about the search engine Neeva because they seemed to be building a full fledge independent web index with top notch talent, I was really bought into the idea of a premium new search experience that I could pay for (no ads) and revenue share some kinds of content. But years later they focused on crypto and AI instead, and I always find myself just googling it because I have ad block and the results are more relevant, sigh
- whiplash451 4y agoPer their website, they still seem to focus on ad-less search. What am I missing?
- deleted 4y ago[deleted]
- ajmurmann 4y agoThere also is kagi.com which should fit all the requirements you described.
- slekker 4y agoTheir new pricing is quite unaffordable now :(
- ajmurmann 4y agoIt's really bad. For years I've been saying that I want someone to make a search engine where I am the customer and that I am happy to pay. Now kagi is here and I am to cheap to pay for it. I feel called out.
- mdaniel 4y agoFWIW, they are not the only one where you can pay to be a customer, they just happen to think that paying-per-search is the strategy they want to go with. I also just double-checked and they do actually offer an all-you-can-eat for $25/mo and $255 if you pay annually Now, the bad news is there is an associated discussion going on over in the "DDG integrates GPT" thread where the intersection of "I want to pay" and "I don't want to GPT anything" is damn near nil :-(
- gavinhoward 4y agoYay! Marginalia considers my site important and good enough to be indexed! Honestly, this makes me really happy. I would prefer that my traffic be driven by curated search engines, even if I get less traffic. Also, I use this. I think it's great.
- HarHarVeryFunny 4y agoI'm not sure how effective it is at present - just gave it a very quick test on searching for info on roman coins - but the concept is great. This is something that I've often wished existed. If I'm searching for roman coins I certainly don't want to find commercial sites selling them (I know what those are), or even the well-known online national collections or auction archives... I'd like to be able to find the specialist sites built by collectors (and maybe academics) that are non-commercial and way more interesting. In the early days of the internet some specialist content/pages were organized into "web rings" each linking to each other, but nowadays we're mostly relying on search to discover new pages, and it seems a lot of the hobbyist content is way harder to find, assuming it's even out there.
- marginalia_nu 4y agoWhat did you search? Try just 'roman coins' #1: http://www.romancoins.info/Content.html http://www.romancoins.info/Content.html #2-4: were not very good #5: https://www.forumancientcoins.com/dougsmith/voc1.html https://www.forumancientcoins.com/dougsmith/voc1.html #6: https://www.cngcoins.com/Greek+and+Roman+Coins.aspx https://www.cngcoins.com/Greek+and+Roman+Coins.aspx #7: https://www.crystalinks.com/romecoins.html https://www.crystalinks.com/romecoins.html If you search for specifically the 'as' it may be eaten as a stop word :-/
- HarHarVeryFunny 4y agoI was searching "imp constantinvs" which is part of the legend on many coins of constantine the great. Would expect to see these details listed on any hobbyist sites, as well as the commercial ones I'm not interested it. BTW #1, 5, 6 are all good sites, but those are very mainstream - those will be top links in Google as well. #6 is purely commercial - an auction house. #5 is a coin dealer's commercial site, but has good collector resources (discussion board, Wiki, collectors galleries) as well.
- marginalia_nu 4y agoDo you know of any hobbyist sites within this space? I want to check a thing, could be this corner of the internet isn't well indexed. I should be able to tell with explore2.
- overthemoon 4y agoThe "random" button sent me on an hour long rabbit hole and I learned about (among other things) the gopher protocol. A+, would lose that time again.
- gerdesj 4y agoMy first experience of the internet was telnet from Win 3 box to a X.25 PAD and then telnet to something JANET (UK) then something US based (NSF I think) and fire up Gopher or WAIS. Later my boss asked me to look at this web thing that he had heard about. I fired up telnet and eventually found an on ramp to CERN. To me it looked rather like everything else but I'm not exactly a rocket scientist! https://www.w3.org/History/1992/WWW/FAQ/WAISandGopher.html https://www.w3.org/History/1992/WWW/FAQ/WAISandGopher.html
- dizhn 4y agoFor all the talk of needing all the cloud infra to run even a simple website, Marginalia hits the frontpage of HN and we can't even bring a single PC sitting in some guy's living room to its knees.
- marginalia_nu 4y agohttps://www.marginalia.nu/junk/just-a-fleshwound.webp https://www.marginalia.nu/junk/just-a-fleshwound.webp If anything it's running faster now. All you've done is warm up the caches and given the JVM a chance to optimize the hottest code. (real talk the SSDs are running pretty near 100% utilization though)
- mvdwoord 4y agoVery well done! On mobile now but will check out the site once home. Do you happen to have a writeup somewhere of your tech stack?
- btown 4y agoNot OP but https://github.com/MarginaliaSearch/MarginaliaSearch/blob/master/code/readme.md https://github.com/MarginaliaSearch/MarginaliaSearch/blob/ma... has a diagram and component overview!
- marginalia_nu 4y agoI'm working on the documentation. It's getting there, but it's still kinda thin in many places.
- btown 4y agoIMO it's actually incredibly well-documented and thoughtfully organized for a one-person project! You should be proud of what you've put together here!
- 4y ago
- MichaelZuo 4y agoIt's quite hard to search for lists of any kind. For example: list of Italian generals list of CPU architectures list of positive rights etc... Return no relevant results. Perhaps it's not giving enough weight to the 'list' aspect?
- marginalia_nu 4y agoI think there's two reasons for this. The query processing is fairly crude. For better or worse, it doesn't do much special processing. Which means you basically need a website that repeatedly says "list of CPU architectures" to rank well. Most of the pages that contain such a title are also actual lists. The index de-prioritizes documents that are mostly lists or tabular data, as they're rarely very often false positives as they often contain repeated words.
- MichaelZuo 4y agoIt doesn't seem like an issue of ranking since there are no results with lists of any sort, even at the bottom. Even if the word 'list of ...' only appears once, it wouldn't be filtered out, right? Out of 100 million pages, it seems like there could easily be a few hundred thousand with lists.
- dylan808hewitt 4y ago[dead]
- deleted 4y ago[deleted]
- qwertox 4y agoWhat really drives me crazy with Google is that they think that it is ok to not label potentially paywalled articles as ads. I get tricked so often into clicking a news snippet offered by Google only to then land on a site which not only presents me a paywall, but also does want me to accept their cookie policy before they present me the paywall. It makes me angry every time anew.
- qwertox 4y agoHmm, "getting started with react" yields the following as the first match: "https://frontendmasters.com/courses/complete-react-v5/getting-started-with-pure-react/ https://frontendmasters.com/courses/complete-react-v5/gettin... Getting Started with Pure React - Complete Intro to React, v5 | Frontend Masters The "Getting Started with Pure React" Lesson is part of the full, Complete Intro to React, v5 course featured in this preview video."
- marginalia_nu 4y agoThanks for pointing it out. I blacklisted the domain. I don't mind commercial content, but if they're using SEO like that they're being a nuisance.
- dang 4y agoRelated. Others? Marginalia Search has received an NLNet grant - https://news.ycombinator.com/item?id=34945541 https://news.ycombinator.com/item?id=34945541 - Feb 2023 (17 comments) A Theoretical Justification (2021) - https://news.ycombinator.com/item?id=32586273 https://news.ycombinator.com/item?id=32586273 - Aug 2022 (22 comments) The Evolution of Marginalia's Crawling - https://news.ycombinator.com/item?id=32565052 https://news.ycombinator.com/item?id=32565052 - Aug 2022 (22 comments) Botspam apocalypse - https://news.ycombinator.com/item?id=32339314 https://news.ycombinator.com/item?id=32339314 - Aug 2022 (346 comments) Marginalia Goes Open Source - https://news.ycombinator.com/item?id=31536626 https://news.ycombinator.com/item?id=31536626 - May 2022 (72 comments) Uncertain Future for Marginalia Search - https://news.ycombinator.com/item?id=31200319 https://news.ycombinator.com/item?id=31200319 - April 2022 (37 comments) Marginalia Search: 1 Year - https://news.ycombinator.com/item?id=30823481 https://news.ycombinator.com/item?id=30823481 - March 2022 (29 comments) Show HN: Marginalia – Exploration Mode - https://news.ycombinator.com/item?id=30047455 https://news.ycombinator.com/item?id=30047455 - Jan 2022 (53 comments) A search engine that favors text-heavy sites and punishes modern web design - https://news.ycombinator.com/item?id=28550764 https://news.ycombinator.com/item?id=28550764 - Sept 2021 (717 comments) (just as a reminder, these lists are only to satiate curious readers - there's no reproach for reposting! Reposts are fine on HN after a year or so: https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html)
- yonrg 4y agoThanks for this! It absolutely touched me. www in a way it was back in the 90 :) those memories came back when getting results to neocities.
- SinePost 4y agoWhat advantages does this offer over something like typing "$QUERY -site:*.com" into a mainstream search engine? I think webmasters in general do a pretty good job at self-segregating their sites into commercial and non-commercial entities through the use of different top-level domains.
- 1123581321 4y agoThe advantage is a better set of sites since there are a ton of interesting little .com sites out there, and lots of unwanted sites on .org and country code domains. He also blocks some URL patterns that appear on spammy domains regardless of TLD. Try a few searches or the random results page to see the difference. It’s fun browsing.
- gerdesj 4y agoWhat a cracking resource! When you need to get away from the beige web, a few clicks on Random Mode is all you need.
- throwaway280382 4y ago@marginalia_nu, Few months ago, you said you would consider open sourcing this search engine. Are there any tasks that us github warriors can help with?
- Tiberium 4y agoIsn't it already open-source? https://github.com/MarginaliaSearch/MarginaliaSearch https://github.com/MarginaliaSearch/MarginaliaSearch
- throwaway280382 4y agoSorry, I was not aware of that. Thx
- Tiberium 4y agoVery cool project! How much hardware do I need to run this? E.g. can I run it on an old Xeon server with 12 threads and 48GB RAM with HDDs, or that'd be too slow?
- marginalia_nu 4y agoIt's mostly hungry for RAM, and needs fast SSDs. I've got 128 Gb RAM and more would be better. I run a test instance on 32 Gb though.