4 ms·
Is there any real obstacle to building a search engine that primarily shows smaller independent websites? Seems pretty doable.
by marginalia_nu 4y ago
Is there any real obstacle to building a search engine that primarily shows smaller independent websites? Seems pretty doable.
- shagie 4y agoUnless this is a whitelist, it's a metric which will be gamed. If it is a white list, then why have a search engine rather than an old school curated Yahoo directory?
- marginalia_nu 4y agoSearch engine spam is actually a fairly solvable problem if you aren't in Google's questionable position of also selling the ads that make the spam economically viable. They can do everything except the one thing that would actually hurt the search engine spammers right in the coin purse: Penalize websites for having ads.
- jjoonathan 4y ago> The goals of the advertising business model do not always correspond to providing quality search to users. - Sergey Brin and Lawrence Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine
- marginalia_nu 4y agoYeah, that one aged well :P
- rightbyte 4y ago> Penalize websites for having ads. Oh ... this is such a good idea. I'm like tempted to try it and see what happens.
- chc 4y agoI don't know about that. If I recall correctly, there was a time when Google was trying to tamp down on pages with excessive advertising, so SEO spammers just switched to making pages that superficially looked like normal informational pages, but the content was all either ad copy or spun text, and all their links went to products they wanted to sell.
- warkdarrior 4y agoIs Wikipedia's prompt to donate (which they do quite often) an ad?
- CM30 4y agoI get where you're coming from, but my experience is that even many people making sites for fun stick ads on the site in the hopes of getting a bit of extra cash. Lots of fansites did so in the olden days, and many useful sites, wikis and blogs do so now, especially if they either use hosting that adds them or they get enough traffic they feel they need to.
- marginalia_nu 4y agoI'm not saying block ads, I'm saying prioritize sites that doesn't have ads above those that do. Prioritize those that have some over those that have a lot.
- hinkley 4y agoThe whole premise of Google at the beginning was that the web is a collaboration, and we can measure that. Then it was we can measure that and make money. Now it’s just we can make money. Whether that part is sinister or not, we know that we have a good number of bad actors, and from search engine results we can be sure that they have not developed a workable Byzantine fault tolerance mechanism to filter out the bad actors. Those who scream the loudest get put on a stage.
- r00fus 4y agoCompanies that acquire other companies and talent will always change. The goal of making money over everything is the one thing all Class-C corporations have in common.
- mayoi 4y agoIt cannot be gamed. You wrote your own search engine, you wrote your own filters, you know how it works and noone else. Whitelists that I wrote by hand also don't introduce new unexpected entries by the way :) This instead could be more like RSS where as your crawler gets new sites, you get updates on new things, and you could filter in your crawler or in RSS client directly, doesn't matter.
- 0xdeadbeefbabe 4y agoAn AI could make the list, and pretty soon the gamers of the system would opt for really terse recipe sites.
- jshen 4y agoThe biggest hurdle is that parsing web pages is really hard, and sites do a bad job at providing good meta data.
- jjk166 4y agoThat actually sounds like the solution. If you're getting something standard, you don't want it. If something is too non-standard to be identified, pass it through.
- marginalia_nu 4y agoBeautifulSoup and its clones does parsing pretty well. Just extracting the text out of HTML isn't incredibly hard, and metadata is too unreliable to ever be much use.
- jshen 4y agoThe hard part is understanding which parts are the content versus navigation or promotions of other content. I’ve written a couple search engines. Have you tried making one with beautiful soup?
- marginalia_nu 4y agoNo I use JSoup for my search engine. You can calculate anchor tag density across the DOM tree and prune branches that exceed a certain threshold to remove navigational elements with reasonable accuracy if that is a problem. It's not going to be perfect, but even Google messes this up every once in a while. I wouldn't consider it a major hurdle.
- peteradio 4y agoI don't presume the source is available... unbelievably cool project that I'm sure a lot of people have imagined themselves doing. edit: https://git.marginalia.nu/marginalia/marginalia.nu https://git.marginalia.nu/marginalia/marginalia.nu !!!
- alexb_ 4y agohttps://wiby.me https://wiby.me
- CloseChoice 4y agoThis is amazing and really made with love. Just found a website in the 90's style of guy who asked companies for free stuff by letter.
- disqard 4y agoNice! I discovered this: https://tilde.club/~fab1/ https://tilde.club/~fab1/ (might make your fans spin up)
- kyleee 4y agoI searched for “cactus” and found a fun history website: http://www.realhistoryww.com/world_history/ancient/Etruria_the_Etruscans_2.htm http://www.realhistoryww.com/world_history/ancient/Etruria_t...
- Blackthorn 4y agoYou can't find what's not there. Way too much of the primary content has moved behind those garden walls. My local farm has a website where they list the stuff they have available. Meanwhile their actual scheduling and detail updates are on their Instagram because of course it is.
- scyzoryk_xyz 4y agoAt some point it would probably become useful to teach “internet/tech literacy” to educate people on why this is a problem. But we’re a few decades from something like this.
- mrguyorama 4y agoWhen I was a kid we had "computer class" that taught how to type, how to use Microsoft office (and open office) applications for different use cases, and this was then mixed in with understanding different sources of information taught by the school library and english classes. As kids are now raised on smartphones instead of the family desktop, I think they need MORE of this, not less, for at least the very important skill of typing. I wonder how many 12 year olds in america can type using the "standard" method, instead of hunt and peck. I don't want computing to be something only known by the children of turbo nerds. I want young adults to be able to solve their own problems with computers, ie build some spreadsheets for home finance or even just be able to graph the data from one of their science classes.
- hammyhavoc 4y agoTrue technical literacy is at an all time low, IMO. Whilst more people than ever are "online", the barrier for admission is so low and very few people ever seem to learn more than they absolutely need to. As you can't develop software on phones and tablets, very few people are tinkering with software. The Pi and iOS app craze brought a momentary change, but it seems to have gone back to how it was—and worse. Kids of today are mostly out of their depth when put in front of a computer of any description if it is beyond basic website usage. Complex program? Forget it. Decent typing skills? Forget it. Networking know-how they'd have picked up from doing LAN gaming with consoles or PCs? No chance. Change a drive? LOL. For the handful of kids that game on PCs, they're generally not very clued up and they're just copying builds they've seen on YouTube to the word. It's a sad state of affairs. And yes—of course, there are the kids or us turbo nerds, because of course there are, but they are so few and far between.
- mayoi 4y ago