4 ms·
Marginalia: 3 Years
- marginalia_nu 3y agoThrowback that gives some indication of how both well and at the same time questionably it worked a mere 6 months in: https://news.ycombinator.com/item?id=28550764 https://news.ycombinator.com/item?id=28550764 Though I think now there's a bit too much reddit and stackexchange and wikipedia stuff in the default filter.
- RandomWorker 3y agoI’ve got this bookmarked and use it to find hyper niche materials on numerical modelling. The stuff it finds on solvers, mesh generation, and optimization methods is so much better than anything I could ever find on Google. Stuff from the 80s and 90s. I’ve found sites written by professionals that I would never find on Google. As someone that doesn’t just take the commercial package of the shelf getting to that knowledge maybe finding Fortran code examples is extremely valuable.
- perihelions 3y agoDo you have an example of a niche expert page you found useful, which is easier to find on Marginalia than Google?
- RandomWorker 3y agoSearch "numerical solver" in any of the major search engines what do you see? Now do that on Marginalia; you find https://www.scilab.org https://www.scilab.org within the first 10 results, which is an open-source numerical solver software, which get's me to code, which gets me to examples to use. To be nuanced, I could change my search to "open-source Range-Kutta numerical solver examples" or something better, but why? Give me the weird deeply technical stuff first. Maybe more a HN example; when I wanted to learn about load balancers, just search "load balancers." Google; lots of SEO crap (with soooo many ads), youtube videos?, and AWS commercials at the top. No idea, where I'm going. Marginalia; a linux wiki, nginx (official documentation), a couple blogs by professionals on the topic. yeah, there are some things in here that ain't great. But If I compare apples to apples the first ten results are just so much better I'd say 8/10 in Marginalia for this example are great to good, while the first 10 (things I could click on) are by companies that don't teach me anything or have articles full of ads.
- dewey 3y agoI don‘t think your examples are very good, these are very generic search terms and if the results are good or bad very much depends on the person searching and what they are looking for.
- marginalia_nu 3y agoUnderspecified queries ("generic search terms") is actually one of the tricky problems in search. The way Marginalia Search deals with them caters to a particular type of audience and set of usecases. I don't think that's wrong. Seems silly to try to cater to everyone. In that scenario you're more likely to not cater to anyone.
- subtra3t 3y agoAh, this helped clarify my doubts. Thanks.
- eviks 3y agoYour "domain" filters seems like a good solution to this?
- marginalia_nu 3y agoYeah. Some of them are a bit rough still, but that's the general idea. Instead of trying to guess what sort of content the user wants, it seems like it makes sense to just give them the option to express that. The recipe filter is approaching something I'd want to explore further, to be able to provide contextual information outside of the search query. https://search.marginalia.nu/search?query=cookie+recipe https://search.marginalia.nu/search?query=cookie+recipe
- ignoramous 3y ago> particular type of audience and set of usecases For signed in accounts (which is pretty much is ~3bn Android and/or Chrome users), Google can predict what the user might prefer and yet...
- unpopularopp 3y agoTried my last 3 Google searches india test cricket lowest total > None of the results are good or giving an answer raid calculator > The results are OK but you still have random noise like a Pokemon save/cheat editor page because it contains the word raid all quiet on the western front movie book differences > 0 results. Like straight up no hits, an empty page
- marginalia_nu 3y ago> india test cricket lowest total > None of the results are good or giving an answer, straight up wrong sites. The search engine has no ambitions to provide a knowledge graph at this point. It's for finding documents on the internet, rather than answering questions. Answering questions is a definitely something one might want, but it often comes at the expense of finding documents. > raid calculator > The results are OK but you still have random noise like a Pokemon save/cheat editor page? The pokemon result was discussing an application called "raidcalc". Seems like a good match, given the search engine does not profile you at all and has no clue about what your interests are. > all quiet on the western front movie book differences Hmm, I think there's an upper bound on the query length you hit. Could probably remove this, it's a pretty old, an artifact from when the query execution didn't deal with long queries well. --edit-- Hmm, I increased the limit but they're still kinda not very good. Although this is definitely squarely within the realm of what I'm working on next, which is query understanding and execution. Right now the search engine doesn't really know how group the terms. Like a human being can see that you'd want |all quiet on the western front| in a sequence, preferrably in the title or appearing a few times, and 'movie', 'book', and 'differences' should be important to the document, but not necessarily appear in that exact order. The search engine currently looks for either documents where they all appear in proximity, or all individual words have high tf-idf relevance markers. Not great for this query.
- klabb3 3y agoIs it possible to just quote the title of the book, old-school style, so it becomes a single phrase? It is arguably a better UI than handing a barrage of words and hoping the engine does the sense-making.
- doubloon 3y agoyup this is definitely alot like how it used to be. @unpopularop cant find "all quiet on the western front book movie differences". well you couldn't do that with AltaVista either in 1998. however if you just type "all quiet on the western front" you get a ton of niche obscure sites talking about it. literally someone's personal blog page. type in 'polytopes' you get a bunch of universities papers and code sites. "rust generics" - again, its a bunch of mailing list discussions, blogs, rust discussion groups, personal websites, obscure professional discussions. this IS how it was back in the day. my only question is how could this possibly be sustainable financially in the long run.
- marginalia_nu 3y ago> my only question is how could this possibly be sustainable financially in the long run. For now I'm funded by grants and donations, got a few years runway that way. The actual operational cost is like $100/month for colocation + personal expenses so what money comes in lasts a surprisingly long time. In the future, we'll see. There does seem to be a lot of people that want this type of thing to exist though, so the hope is if I polish it even more, further funding will become available from likeminded people, possibly selling API access to other search engines. Search is notoriously hard to make money from (outside of ads), though not having a lot of expenses seems like a reasonable path to go.
- 48864w6ui 3y agoIt sounds like you only need one person (not as deep pocketed as Andrew Carnegie but who has read "gospel of wealth" and agrees with it) to have support for decades if not perpetuity. Universities traditionally have done this sort of thing by playing golf and naming buildings, but I'm sure in the 21st century there are other models. (Fwiw $2k/yr is below a typical golf membership)
- gary_0 3y agoI think as long as you're not setting out to start a tech company with thousands of employees, or branch out into a sector with the word "cloud" in it, you'll be fine. Only unreasonably big ambitions cost billions. A project is usually on the road to success when it starts with a disclaimer like "just a hobby, won't be big and professional like gnu". I think a larger concern is how you'll address the Bus Factor going forward.
- ttt3ts 3y agoCool engine. Going to check out source soon but "ROME2D16-2T" returned relevant results from esoteric sources. Useful.
- NeutralForest 3y agoCongrats on the progress, I don't use marginalia as much as I should because I'm so used to rely on Google. It's a wonderful project though and I'll prob use it more since spammy SEO sites and AI generated answers seem to get more prevalent.
- marginalia_nu 3y agoProbably some ways away from daily driver material. Optimistically sometime this summer when I'm done with the query and execution stuff it'll start approaching that territory.
- deleted 3y ago[deleted]
- aqfamnzc 3y agoI've been impressed by the results I see there. And you've chosen a sick name for it.
- behnamoh 3y agoI just looked up "transformers intuition" and the results blew my mind. In comparison, Google's results led me to SEO'd websites (mostly Medium) and fancy-looking sites with inferior content. Awesome work Marginalia!
- 101008 3y agodid a search and first results were from "stack exchange sci fi", i was expecting something more nostalgic
- marginalia_nu 3y agoTry the vintage or tilde filter. https://search.marginalia.nu/search?query=anime&profile=vintage https://search.marginalia.nu/search?query=anime&profile=vint... https://search.marginalia.nu/search?query=anime&profile=tilde https://search.marginalia.nu/search?query=anime&profile=tild...
- renegat0x0 3y agoMost important lines for me. It’s proving a bit harder than anticipated, not because the software can’t handle it, but because the signal to noise ratio of the web isn’t very good; a huge reason why the search engine works relatively well is because of what it doesn’t index.
- InvOfSmallC 3y agoDo you offer API?
- marginalia_nu 3y agohttps://api.marginalia.nu/ https://api.marginalia.nu/ :-) Demo key is always under siege though.
- InvOfSmallC 3y agoThanks, I'm building a website focused on Metroidvanias. I liked the results so I was thinking I may use it to offer some interesting results on the various game pages.
- yuhong 3y ago[dead]
- dreamcompiler 3y agoViktor- I'm curious as to whether Common Crawl [0] would be useful to you. It's currently around 100TB and 3.35 billion pages, so it's going to be a long download unless you process it in place on S3. I have no idea what its signal/noise ratio is. [0] https://commoncrawl.org/overview https://commoncrawl.org/overview
- makmanalp 3y agoThis is neat - I found a random website [0] where someone binary patched C&C Tiberian Sun to have IPv6 support, just because. It feels so much like the old web I miss. For some reason all this makes me reminisce about Fravia's Searchlores [1] which always felt a bit like if Umberto Eco was interested in computers. And the site felt a bit like the library labyrinth from The Name of The Rose where you'd turn some random corner and find something incredible, only to lose it forever later on :D [0] http://ts.sesse.net/ http://ts.sesse.net/ [1] https://www.biostatisticien.eu/www.searchlores.org/indexo.htm https://www.biostatisticien.eu/www.searchlores.org/indexo.ht...
- random_question 3y agoWhat's up with the "random site" function? I would expect it to sample uniformly, but it seems to return certain sites over and over.
- marginalia_nu 3y agoThe problem is specifically that it's sampling uniformly, rather than doing a 'spotify shuffle' that actively eliminates repetition. That, and the set isn't very large, just a few thousand. If you randomly pick 25 items from a bag of ~3000 total a bunch of times, chances are relativley high that you're going to see repetition.
- silent_cal 3y agoI love Marginalia!