5 ms·
> What I dream of is Google circa 2006. A time where a search engine searched what you asked for. A time before aggressive SEO. A time before social media, when
by marginalia_nu 4y ago
> What I dream of is Google circa 2006. A time where a search engine searched what you asked for. A time before aggressive SEO. A time before social media, when the only people on the internet had a reason to be there. A time before sticky headers and full-screen modal pop-ups asking you to subscribe to a newsletter before reading the article. A time before click-bait and subscription-only websites which tease you with a paragraph before blurring out the rest of the content.
Well, I built sort of exactly this: https://search.marginalia.nu/ https://search.marginalia.nu/
It's not great, but it sure has its moments.
Should be noted, regarding Koh Lanta, that travel is one of the most aggressively SEO-spammed topics, along with pharma and online casinos. It's extremely difficult to cut through the noise and reach any sort of signal.
- taftster 4y ago[Quoting your quote] - A time before aggressive SEO? No, just that Google was able to defeat it with Page Rank (one of their possibly few innovative ideas to the search arena). Mathematically, Page Rank did very well to weed out the SEO'd websites of the time. Point is, I don't think SEO at the time was any more or less "aggressive", per se. Just different. And clearly, the game has since escalated altogether.
- onlyrealcuzzo 4y ago> Well, I built sort of exactly this: https://search.marginalia.nu/ https://search.marginalia.nu/ What's the elevator pitch? What do you actually do that is special? From the about section: > This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed. > The software for this search engine is all custom-built, and all crawling and indexing is done in-house. The project is open source. Feel free to poke about in the source code or contribute to the development! But that doesn't really give me much info.
- tigerlily 4y agoI know it’s of low value to ask, but who really gives a shit?
- interroboink 4y agoI do! (also low value, perhaps :)
- tigerlily 4y agoSigh, I guess I should’ve clarified I meant “who cares whether or not there’s an elevator pitch, this is just something that someone thought we all needed and would be cool to build”. FWIW I think it is cool and much needed.
- marginalia_nu 4y agoThe point is to surface human websites and discriminate against content mills, click funnels and e-commerce. For one I have a sort of budget for how much javascript I will tolerate. Some is fine, like your standard wordpress config probably will fly, but not much more. I also do a personalized pagerank biased toward the blogosphere. The likelihood your website shows up in the results is directly determined by whether real humans link to the website. It's not a Google replacement by any measure, it's meant as a complement, but if you are looking for more in-depth content on a topic than you can find elsewhere, it's often a good starting point.
- spacemadness 4y agoI like the idea of a search engine that acts as a sort of filter for known dark patterns along with returning useful results. Limiting on javascript quantity seems like a sledgehammer approach, but I like the spirit of it all the same.
- droidist2 4y ago> For one I have a sort of budget for how much javascript I will tolerate. Some is fine, like your standard wordpress config probably will fly, but not much more. What about sites built with React, etc? Like would you be filtering out cool things like Ableton's "Learning Synths"[1] or do you mostly mean third party scripts? [1] https://learningsynths.ableton.com https://learningsynths.ableton.com
- marginalia_nu 4y agoYeah, those sited go out the window entirely. It's a search engine for documents, not applications.
- iakh 4y agoWhy should it matter if a website is marked up on the server side or client side to a search engine?
- 4y ago
- varispeed 4y agoSocial media existed way before 2006, it just wasn't so popular. But this was indeed a great time. Remember being on a social network ca 1999 and people used it as a diary where others could comment, give advice and meet others with similar interests. There was a timeline and posts from people you "followed". I think it didn't have likes or ratings though. Most people were super friendly and we were doing meet ups, so very much everyone knew each other. At peak time there was about 20k people and at that point owners couldn't cope with it. With that amount of people you'll find some nasty ones that will ruin it for everyone else and that's what happened. Owners couldn't afford to host it, so they were doing crowdfunding to cover the server costs and maintenance. Some bitter people didn't like that and started making unfounded accusations of theft or that they will report everything to the tax man and the police. So owners eventually closed it.
- Eratosthenes 4y agoBrilliant! This is fantastic. I'm in love with it already :)
- carschno 4y agoImpressive! I'm genuinely curious: Where do you host the index and the DB, and how do you finance it? Hosting an index of that size on AWS is not cheap, and maintaining the server infrastructure yourself seems quite some effort.
- marginalia_nu 4y agoIt's literally hosted off a beefy PC in my living room. I could definitely not afford to host this on AWS or rented servers. About ~5k USD total investment including a small UPS, operational costs are $40/mo. Doesn't actually require all that much maintenance, at least not compared to the work I've put into building a search engine (including crawler, index) from scratch.
- joshka 4y agoNeat, I wonder if applying something like GPT-3 to understand what each page gives might help with that a little. Kind of like reversing the SEO spam. I wonder if that's something that ML tools could detect given enough input?
- marginalia_nu 4y agoMy guess is doing exactly that is most the source of Google's problems right no, meanwhile the spammers do the exact same thing except trying to teach a ML model to successfully pose as authentic content, and what you get is the gloriosu mess we are in. (Even though it's just the last battle in a long war against spam)
- joshka 4y agoToo much / too little, both are problematic I guess. This ends up becoming a war of algorithmic generation vs understanding, and devolves to effectively answering the age old question "is this spam"? I was going to suggest that perhaps adding info about whether the site had ads, etc. might help the usability. Then I tried marginalia ... :) Perhaps we need to invest in similar detection for other antipatterns (listicles, blogspam etc.) - this is mostly what I was talking about. Less the understanding for indexing, and more the classification for user safety and authenticity of results. But again, this is very cool.
- marginalia_nu 4y agoYeah maybe. I don't know how useful these "hobo signs" actually are, but they did feel like a natural response to a lot of web sites justifying their use of trackers and cookies and affiliate links and whatnot with "everyone is doing this!"