12 ms·
I don't think it's too late to fix this. We can build better search engines, or better yet, tools more suitable tools for discovery. It doesn't have to be big t
by marginalia_nu 5y ago
I don't think it's too late to fix this. We can build better search engines, or better yet, tools more suitable tools for discovery. It doesn't have to be big to be useful.
Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out.
Out of largely the same frustration permeating the link, I built search.marginalia.nu alone in less than a single year in my spare time without prior knowledge of search, paying for the hardware out of my own pocket.
I don't say that to promote how great I am. The point is specifically that I'm just some guy on the Internet. Just imagine what could be possible if actual resources were put toward finding solutions to this problem.
- kwerk 5y agoWhoa. Searching a few random terms you’re hitting your goal of “serendipity” results. Can you share more about the search basics? I’m learning search now
- marginalia_nu 5y agoDon't really know what to say, a lot of the algorithms are snore-inducingly standard. TF-IDF, an approximation of BM-25. Only twist is that I do Personalized PageRank biased toward a secret segment of the internet to promote the right kinds of search results.
- agumonkey 5y agoBetter search is not the issue, it distorts the spirit, google became the focus of attention, now you want to be listed and seen, to monetize too.. before that people spent less time formatting stuff to exist. The word of mouth levels is a different (maybe more fertile) soil.
- marginalia_nu 5y agoYeah I don't think search is actually the solution to the problem of discovery, it was just the first avenue I tried. Although the metadata from a search engine database does open up a lot of options.
- agumonkey 5y agoOh sorry btw, it was not an attack against you. And I actually enjoyed your engine and found a new lisp dialect I never heard of (even though I googled that a few hundred times).
- fallat 5y ago> We can build better search engines Yeah, they're called people. We can curate better results.
- dylan604 5y agoSheds a tear for Yahoo like search. What would the web be today if Sergey and Larry hadn't met? I really enjoyed the Halt And Catch Fire take on curated vs algo search.
- indymike 5y ago> I don't think it's too late to fix this. We can build better search engines, or better yet, tools more suitable tools for discovery. It doesn't have to be big to be useful. Over the next few years, things will change. When people don't like the product, it gives us an opportunity to make products people do like. Right now, a lot of the internet is just not very likable.
- marginalia_nu 5y agoI'm not sure. This is a problem that affects not just websites, but physical products as well. It's really hard to find what you actually want. Producers are rewarded more for paying for reviews and search results, for reddit astroturfing, for influencer recommendations; than they are for actually making something worthwhile. It's not that good things aren't being made, it's just that it's incredibly hard to find them among all crap. There doesn't seem to be a good mechanism that lets the actually good products float to the top.
- lanstin 5y agoI doubt this is a new phenomenon. I certainly risk buying things just by myself where pre-internet I would find someone that knew about the thing (cars, knitting, lamps, whatever) and tell them my issue and see what they said. now there is some expextation on my part that fifteen minutes of reading will somehow allow me to drop a lot of cash in some specialized area i have no edperience in and not be taken advantage of as a sucker in the market place? If you are spending money, possibly just having the worlds info at your fingertips doesn't make up for a lack of experience.
- marginalia_nu 5y agoBack in the brick and mortar days, you could at least go look at the thing. Not that there weren't lemons, but the sort of extreme low quality alibaba crap that you find everywhere nowdays wouldn't sell at all because it feels so obviously low quality. Even having it on the shelves would stain the reputation of the store.
- 5y ago
- _fat_santa 5y ago> Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out. It's a curation problem. We have billions of pieces of content being created and only a handful of curators to bring that info to us. All the big search engine players want to be "the one" but it's a fools errand IMO. No one search engine will be able to become "the one" because there is simply too much information. Google with all of their cash in the bank and talented engineers can't even crack the problem. The problem though is even though each search engine offers something different, they all think they offer the same thing and compete to muscle out any smaller players. So you get a situation where you want a metal roof for your house, metal roofer gives you a quote. Big roofer guy in town hears about this and runs the metal roofer guy out of town, then tells you he cant offer you a metal roof anyways.
- rtpg 5y agoWho even "gets content" through searching? I'm sure many people here are all basically using the internet through link aggregators (reddit, fb, HN etc). Has the web ever been much beyond that? Have people ever really been just hitting "give me a random web page about X"? Of course search engines are very useful, just that these conversations about content and "the web", I feel like there's not much focus on how so much traffic is guided through link aggregation.
- dgb23 5y agoI often search for articles/books about X, or reviews about Y. The top results for those things is a mix of SEO spam, Amazon links etc. which I all try to avoid. I use search for information, but I often have to dig quite deep to find something worth clicking on or reading. And that's true for Google and DDG (which I mostly use). When it's raw, basic knowledge, then I'm using !wiki or !mdn etc. in DDG to search those sites directly. Note: Wikipedia and Mozilla Developer Network are two of my favorite sites. But when it's something more subtle and specialized that I'm not very familiar with then often my only hope of finding something useful is to search via aggregators and forums like HN, Reddit etc. because then I can gauge the quality of the links via the discussion around it and the people who post them. Or in some cases I know about a specialist or journalist and can use their name to filter out spam. Discovering these kind of things via search is really hard, especially if their authors are not known. It's the spammers and low effort creators that get rewarded the most, partly because of search engines, partly because of visitors who are content with that kind of quality.
- deleted 5y ago[deleted]
- geuis 5y agoHey, edit your comment so your link is clickable.
- marginalia_nu 5y agoIt's past the event horizon and can't be edited anymore.
- silent_cal 5y agoYour search engine is awesome, I've been using it since I saw it elsewhere on HN a few months ago.
- alexashka 5y ago> Right now the big problem as I see it is that there is great content being created Is that so? Where? We'd have to define 'great'. If you mean Tik Tok dance videos and instagram photos/art, then yes. If you mean anything related to philosophy, sociology or explaining how the world works at large - no. You don't need any tools. You just need to pay people, who have a working brain they've accomplished something with. It's not an algorithm problem, it's a 'fuck you money' problem. Until people with a brain have fuck you money, they aren't going to waste their life fighting braindead Karens and corporate/political interests who don't want people with a brain to explain how the real world works. Look at the amount of shit Joe Rogan is getting for having accomplished scientists on to talk about their work. He has fuck you money. Do you know how many other people have fuck you money and don't say shit? A lot. You need to spread that fuck you money around thick, so that one in ten thousand will have the guts to put up with braindead Karens of the world.
- marginalia_nu 5y ago> If you mean anything related to philosophy, sociology or explaining how the world works at large - no. I'm the wrong person to ask for sociological links, but here's a whole flock of actively maintained black swans pertaining to philosophy and other topics: https://lileks.com/ https://lileks.com/ https://stpeter.im/ https://stpeter.im/ https://www.philosophizethis.org/ https://www.philosophizethis.org/ https://antiqueradio.org/ https://antiqueradio.org/ http://attalus.org/ http://attalus.org/
- bsedlm 5y agonone of the real issues breaking the internet (which is not quite broken yet) has technical solutions. No better tool nor "more technology" will solve the problems at hand. The problems are social and economical, they require "political" solutions, not more technology.
- myself248 5y agoGoogle has two problems. One is that they're actually Doubleclick in a trenchcoat, and serving you relevant information isn't their goal. Serving you ads is. The other is Goodhart's Law: When a measure (being highly-ranked on google) becomes a target, it ceases to be an accurate measure. Pages now optimize for search as measured by Google. If Marginalia because the #1 search engine in the world tomorrow, I suspect SEO spam would start trying to beat it a few days later, and the results would be crap inside a month.
- marginalia_nu 5y ago> If Marginalia because the #1 search engine in the world tomorrow, I suspect SEO spam would start trying to beat it a few days later, and the results would be crap inside a month. This is only actually a problem if there is one big search engine. Monocultures of all varieties are vulnerable to specialization, this is as true in a country that grows nothing but wheat to a world where every computer runs Windows XP. If there would be two or three big search engines with conflicting means of evaluating a site, it would decimate the effectiveness of SEO. A lot of what I'm doing to find good results is promoting the sort of sites that Google rejects. While there are search engine optimized sites that aren't SEO spam, there is no SEO spam that isn't search engine optimized.