6 ms·

I want to make my own search engine, one day, with my own crawler. There is an SEO-proof way to determine what the ranking of a site should be - penalise it fo

I want to make my own search engine, one day, with my own crawler. There is an SEO-proof way to determine what the ranking of a site should be - penalise it for each advertisement, penalise further for delivering different content to the crawler[1], allow logged-in users to down-rank a site, etc. Basically, a site starts off with a perfect score, then gets penalised for each violation, for each dark-pattern, for each anti-user decision they took. This sort of ranking cannot be gamed by SEO spammers, because ... why get their site to the top of the rankings if they can't put any advertisements on it? Even if they do manage it, their time in the sun will be brief. [1] Random spot-checks with different user agents with different fingerprints should work.

amelius 2y ago
There may be a place for a green (climate-friendly) search engine, where every ad for physical products is penalized.
nicce 2y ago
> allow logged-in users to down-rank a site, etc. And then you need a huge anti-bot mechanism if the search engine will get any popularity. There is money-based motivation to affect the rankings.
- nine_k 2y ago
  Karma, occasional moderation rights, and occasional meta-moderation rights worked pretty well at Slashdot, for instance. It depends heavily on user involvement though.
  - nicce 2y ago
    Yeah, there is a dilemma. To get a wider audience, you need very good search results. But if you search quality depends on the amount of users, how do you get in there? People like power, that keeps them moderating (looking at Facebook groups). What kind of reward system works here?
    - chongli 2y ago
      Well the baseline search quality should not rely on users. It should be great out of the box until spammers begin targeting it directly. Relying on users to moderate should happen organically as the number of users grow, which ideally would be timed to offset the growth in spam targeted at the engine.
      - nicce 2y ago
        The issue is that it needs the really really good on baseline before people start switching from other services. But if you are able provide better baseline than others, then you might not even need the users to be "better. With users, you would be then exceptional.
        nine_k 2y ago
        I suppose services like Gmail benefit a lot from users who mark the occasional spam letter in their inboxes. While each such marking is inconsequential, correlated over many users, they allow to detect new and unusual waves of spam, and to start filtering them for everyone. A similar mechanism could work for a search engine. It would be harder to game if it required high karma (many other users matching your judgment before it starts to count), and a paid account, like with Kagi.
- Aachen 2y ago
  I keep thinking of a hierarchical invite model for these sorts of problems. I'd be very curious if someone could give a second opinion on this idea. The mechanism: Everyone has to be invited by someone, so it all traces back to the creator. The creator knows they themselves are legit, but let's say someone online asked for an invite and bad inputs keep coming from somewhere down that branch of the invite tree. Either the person the creator invited is a spammer, or someone they've invited in turn is the spammer. All accounts leading up to them can be progressively killed (and their inputs nullified), starting with the ones actually causing trouble → if it keeps happening in that branch then kill one layer up, and so on. Incentives: People risk losing their own account when inviting someone who they don't trust to be good netizens. Maybe there needs to be an incentive why you care about your account in the first place, or maybe (looking at Wikipedia or OpenStreetMap, or HN with its voting system and the homepage meaning a lot of attention for your page) a majority of people are simply honest and happy to contribute and that suffices Problem it solves: Wouldn't such a hierarchical invite system work around the online identification problem? If you would DNA check everyone and ban people who abused the system in the last decade, you'd also not have any spam online, but that's way too invasive (besides not being legal and prohibitively expensive, it's also not ethical). However, a pseudonymous (all that is known about you is a random user ID) invite tree seems to me like it would have similar properties. It requires banning the same person perhaps a hundred times until they run out of people that will give them invites, but wouldn't it eventually distill the honest people from the population? (Which is probably almost everyone if there is no gain from systematic cheating and there's social pressure to not ask for multiple invites because account holders know that means you were either messing with the system they enjoy using or invited someone onto it that did that.) (Implementation details: One bad input isn't an instant ban: people misclick or misunderstand interfaces, but eventually it gets to a point where, if they can't click the right buttons, there's also no point having them be moderators of the search engine (or whatever this is used for) and so their account is removed. If multiple removals happen in a tree that's deep and recent, remove more than one layer at a time to get rid of malicious sockpuppet layers. The tree's maximum depth can be limited to something on the order of 50: it doesn't take many steps to find a chain of relationships that links two random persons on the planet, so a fairly low depth is enough for the whole world. People should be told on the invitation page how many bad apples were removed in each layer below them, so if they're 1 bad apple removed from having their own account pruned then they know to only invite people they're very sure about. One problem I see with the system is that it reveals social graphs which not everyone is happy about. If that means being able to kill virtually all spam, content farms, etc., maybe it's worth it, but or course more research is needed beyond an initial proposal.)
- bdw5204 2y ago
  There's also the issue of users trying to downrank any polarizing site because they disagree with it. To pick an example, think of the search term "vaccine" and how strongly people on both sides feel about it. Eventually, you'll probably end up with results for "vaccine" that have nothing whatsoever to do with vaccines because everything vaccine-related will end up downranked.
flir 2y ago
It's your ranking system, so it's correct by definition (for you), but I've got to point out you're not considering the quality of the text. In your model, an empty page is perfect. Cynically, maybe you're on to something...
- lelanthran 2y ago
  > It's your ranking system, so it's correct by definition (for you), but I've got to point out you're not considering the quality of the text. Yes. This is by design![1] > In your model, an empty page is perfect. Yes. What's the problem with that? An empty page won't match any search terms, will it? [1] It's impossible to quickly and cheaply determine the quality of the text, in an age where it's cheaper for a blog-spammer to use AI to write 3k articles than for the crawler to use AI to check 3m articles.
  - Folcon 2y ago
    You might be onto an interesting new view of the web here, if you've ever seen https://www.builtwith.com https://www.builtwith.com, something similar that also processes script tags and downweights based on what the script is would also add to this effect. At the very least I'd be interested in looking at such a site and seeing how it differs from https://www.millionshort.com https://www.millionshort.com =)...
  - verisimi 2y ago
    I think this is a great idea. I really like the principle. But it does miss the point that you need a huge amount of resources. Plus 20 billion or so to get Apple to use your search.
    - Aachen 2y ago
      If it is actually as much better as it sounds, even apple users will find a way to set their search engine to what works once word of mouth gets around. (Until Apple turns that option off and waits for for the EU to pass another gatekeeper law anyway.) If it is actually as much better as it sounds, one wonders why Google or Duckduckgo doesn't do this already. For Google, killing the advertisement incentive is probably counterproductive in the long run, but why doesn't DDG?
corn13read2 2y ago
Let's collaborate
- lelanthran 2y ago
  If you've got capital, I'd love to :-) On a more serious note, I'm actually very busy trying to make money, so I don't really explore things that can't pay off unless they are small. Something like a crawler + engine is very capital intensive.
6510 2y ago
Each char of css scores 1 bad, a char of js is 100 bad, one small advertisement is one million bad, a cookie banner is 5 million bad, news letter modals 123 456 bad, links and embeds pointing at big tech 100 trillion bad. The pro version comes in 3 kinds: 50 cent, 10 usd and 100 usd per search. For this money you get one or more employees modifying the search results in real time. Pressing the "work harder" button increases the fee by .50,10,100 respectively. You can press it as often as you like.
acatton 2y ago
I wish you luck and I hope you succeed, but you make it sound much much easier than what it would be. First of all, you're going to drown in hardware costs, if you run your own hardware. If you run on AWS, you will be the largest AWS customer. 2 years ago, when Google was still displaying result counts, I got 1.3 billions results for "sushi"[1]. This means that if you use a reverse index to lookup your results, the "sushi" entry will be ~19GiB large, assuming you use UUIDs. If you think 90% of this is spam, and you only index the non spam (detecting spam/seo is far from trivial, but let's say you figure it out), you still need ~2GiB just for mapping "sushi". With 755,865 words in the English dictionary, according to wikipedia[2], you'll need ~1.5 PiB (yes, pebi/peta, 1,536 TiB) just to store relationships for English pages. This is assuming you don't support other languages, you discard 90% of pages, and you don't cache the content of pages for re-indexing. In addition to this, you also need to store the meta-data for each pages (vote counts from your voting system, whether it's serving different content, etc...). The order of magnitude has to be in the O(100TiB) from my conservative gut feeling. (still assuming you discard 90% of the web, and I'll assume you aggregate the metadata on the domain, not on the individual pages) The second challenge is your ranking. Now that you've become the dominant search engine with your awesome ranking system, you will become the main target for swaths of motivated click-farms which are exploiting workers from low income countries. They will be trying to register accounts, vote and game your ranking. You can most likely detect this behaviour, but their behaviour will be very similar to a significant portion of your real users. So you'll be fishing in a pond with a rocket launcher, and some of your legitimate users will be collateral victims. Otherwise, you'll spend most of your time playing a cat-and-mouse game with the SEO spammers instead of improving your search engine and fixing bugs. I'm also falling in the trap "i could rewrite that in a weekend" sometimes, but for a search engine, I would love to see decent competition, but it's near impossible. [1] https://news.ycombinator.com/item?id=30925402 https://news.ycombinator.com/item?id=30925402 [2] https://en.wikipedia.org/w/index.php?title=List_of_dictionaries_by_number_of_words&oldid=1223696888 https://en.wikipedia.org/w/index.php?title=List_of_dictionar...
- adileo 2y ago
  Absolutely spot on. Additionally, it's worth mentioning that a lot of content is now locked behind a few major platforms (eg. Facebook, LinkedIn, Medium, YouTube, etc.) or CDNs like Cloudflare, which often block crawling from non-Google IPs or well-known search engines. While the other costs mentioned here can be optimized with current hardware prices and a good database, anti-crawling measures necessitate thousands of IPs/proxies, making the process even more challenging and costly.
Ferret7446 2y ago
> There is an SEO-proof way to determine what the ranking of a site should be This comment deserves a "my sweet summer child". Everything can be gamed; if you don't see how, then you really shouldn't be suggesting ideas. How do you define an advertisement? How do you prevent downvote spam (if captchas worked, there wouldn't be bots/astroturfers on the Internet)? How will you fund your search engine?
- codetrotter 2y ago
  > How do you define an advertisement? Headless browser with uBlock Origin. Check programmatically how many ads were blocked on the page.
  - fl0id 2y ago
    But many ads and crap today are actually not traditional ads. Don’t they count? If not, then gpt generated text ads or whatever will become more prevalent
- lelanthran 2y ago
  > This comment deserves a "my sweet summer child". I think being condescending really removes any impact from any point you want to make. After all, I posed this as an idle musing, not as a all-in venture, didn't I? > Everything can be gamed; if you don't see how, then you really shouldn't be suggesting ideas. If people are gaming the engine by avoiding all the penalties, then that's the exact sort of gaming I would like to see happen. I mean, you have read through the rest of the thread, here, right?
chongli 2y ago
There’s quite a lot of dark patterns to watch out for. One is including text on a page meant to be read by your crawler but which is hidden from users by various CSS tricks. To get around this you’d either need to specifically blacklist certain CSS / DOM structures or do something very radical: render the page and then attempt to OCR it back into text to match against the original text.
- theturtletalks 2y ago
  Remember when Google said they would penalize websites that use dark patterns like Quora, Pinterest, Twitter, etc? Google can easily penalize those websites for their dark patterns so websites and blogs by actual people rise to the top but they don’t.
  - chongli 2y ago
    I think they actually do that but then specifically white-list those sites. It’s really annoying!
    - theturtletalks 2y ago
      That’s even more damning. So the established sites are playing by different SEO rules altogether?
      - chongli 2y ago
        I don’t have any proof. I’m just taking it as a given that this is what they’re doing because those sites are still engaging in cloaking despite Google making it clear they have a policy against cloaking.
        theturtletalks 2y ago
        Is their cloaking policy still active? Felt like they enforced it at the beginning, but when big sites started using the same dark patterns, they stopped? If it’s an official policy still, would love to hear why they selectively enforce it.
hooby 2y ago
"I invented a ranking system that cannot be gamed!" By my experience, ALL ranking systems can - and will be gamed, no matter what. I'm still willing to give you the benefit of the doubt - maybe you are the one who can finally truly come up with an un-game-able system. But I'll consider that an extraordinary claim - and will require to see extraordinary proof for that.
VikingCoder 2y ago
Wikipedia shows an ad asking for money. So a hundred times a day, I open a new Wikipedia clone with no ads. I get perfect scores. Once I detect your bot has crawled it, I put ads all over it until the traffic finally goes away. No system is fool-proof.
- NayamAmarshe 2y ago
  But there's a difference between an embedded ad and a native ad. You can allow websites to just add an image or a section of an ad that does not hit a third-party endpoint. Native ads are not as intrusive and if they become intrusive, like a popup, you can penalize that. It sounds like a sound system.
  - VikingCoder 2y ago
    This is a predator/prey relationship. OP suggested they had a fool-proof way to get around that. They're wrong. Might there be some merit in their idea? Sure. Is it so fool-proof that it would actually be worth the effort to build and maintain their own crawler and index? I highly doubt it.
mark_l_watson 2y ago
I used to play with this, a long time ago. I set up Apache Nutch on my home Linux Desktop system to crawl about 20 of my favorite and most useful sites. That was fun, and I was playing with some knowledge management ideas. If I try that again, I would probably use a Hetzner server and not run it at home. I would probably still stick with spidering only about 20 web sites.
tambourine_man 2y ago
If you allow logged-in users to down-rank a site, you already did the hardest part, which is to differentiate bots and spammers from real users. From there, you can easily add positive feedback as well, IMO. But the first part is really hard. Maybe it needs to piggyback on some centralized government emitted number, which is sad, but maybe the only solution. I'd love some biometric hash, or similar, but those decentralized IDs have been a promisse for years and have yet to catch on, for whatever reason.
tambourine_man 2y ago
I've long felt that if we separated the problem we could open the flood gates for a lot of innovation. Crawling should be its own thing. That is much easily distributed and decentralized. We could then freely experiment with algorithms and strategies on that open database.
p3rls 2y ago
Counterpoint: I'm spending thousands on stuff like APIs to graph out data for my users and would like to be compensated. The existing system could work with tweaks. I like metrics like time-on-page, load times, and "domain authority". With the right weights on the algorithm, real practical guidelines and a robust system of manual actions (hopefully as transparent as possible) I believe something like Google can last for as long as the internet does.
1vuio0pswjnm7 2y ago
"This sort of ranking cannot be gamed by SEO spammers, because ... why get their site to the top of the rankings if they can't put any advertisements on it?" Because it links to pages with advertisements on them? There are easier and more objective methods for ordering search results. Alphabetical, chronological, relevance, etc. People who grew up in a world before Google used these extensively. They were everywhere. I used these in libraries for decades. "Developers" want to pretend these methods were inferior. When I think about people trying to game alphabetical business listings by naming their business "A-1" it seems delightfully quaint compared to the sociopathatic "dark patterns" of software developers. "Rankings", a subjective measure, powered by a secret, ever-changing, hand-tweaked algorithm, are superior because they can be used strategically for commercial gain by intermediaries like Google. When the intermediary becomes wealthy and influential enough, acting as gatekeeper to vast amounts of information, then the other, objective methods might seem inferior. Google accelerates the death of objectivity. It's an advertising company, not an academic search think tank.
cj 2y ago
> why get their site to the top of the rankings if they can't put any advertisements on it? You would need to penalize any website that accepts money from users. There are a lot of SaaS companies out there writing blog spam to promote subscription products.
- iamacyborg 2y ago
  Some of that blog spam is genuinely useful, so you need to differentiate between content quality with brands writing content to promote their saas product.
- lelanthran 2y ago
  > You would need to penalize any website that accepts money from users. There are a lot of SaaS companies out there writing blog spam to promote subscription products. I don't see that as a problem, because blog-spam without advertisements is going to get down-ranked until it is exactly where it needs to be. If a single site dominates too much, then a hard cap on how how often a specific site can appear in the results is feasible, or weighing each page with a percentage of all the pages on a site (thereby penalising sites that have millions of pages). Specific exceptions can be made for wikipedia, or similar. For example, lets look at digital-ocean: Is that constant blogging something that users find helpful? Well, if a page is relevant to the search term then there's no problem. If it's not then the numbers of 'irrelevant to this search' feedback from logged-in users will gradually down-vote it until it is correctly ranked. If it's relevant but is one of 30k pages on the site that matches relevancy with that search term then it gets down-ranked. Basically, the motivation to do nothing but spam is completely eliminated. People spamming "How to do $TRIVIAL-THING" to drive engagement with their SaaS product is, I feel, a completely valid result if the search term matches relevancy with the specific blog post, and the site isn't doing a barrage of postings. Humans respond to incentives. Taking away the incentive to generate endless drivel and replacing it with the incentive to provide good content is the intention behind my proposal.
  - chongli 2y ago
    Specific exceptions can be made for wikipedia, or similar. Once you go down the dark path of white-listing sites, you’ve admitted defeat. How are you going to find that upstart Reddit competitor or federated Wikipedia competitor?
underwater 2y ago
But that web doesn’t exist anymore. The dominance of Google means all the creators and authors went elsewhere. Filter out the crap and you’ll realise that there’s very little substance left.
- kristopolous 2y ago
  Maybe it does. I certainly write clean content. And even if it doesn't (which I think it does), disincentiving bad behavior has to start somewhere
- lelanthran 2y ago
  > But that web doesn’t exist anymore. The dominance of Google means all the creators and authors went elsewhere. Filter out the crap and you’ll realise that there’s very little substance left. Yes, and that signal/noise ratio will drop even further until some filtering gets done. Filtering out the crap might slow the trend.
Fethbita 2y ago
Then maybe you might be interested in Marginalia Search https://search.marginalia.nu/ https://search.marginalia.nu/
- lelanthran 2y ago
  > Then maybe you might be interested in Marginalia Search https://search.marginalia.nu/ https://search.marginalia.nu/ I have tried it before, but it appears to me that it dismisses all commercial content altogether, which is not what I want. That is throwing out the baby with the bathwater. For example, I searched for "The Vietnam of computer science" (Without the quotes) and it returned zero results.
  - thfuran 2y ago
    >but it appears to me that it dismisses all commercial content altogether, which is not what I want Isn't it? You want to downrank for ads and downrank for paywalls. How is commercial content supposed to be funded?
    - lelanthran 2y ago
      > >but it appears to me that it dismisses all commercial content altogether, which is not what I want > Isn't it? Of course not. Appearing lower in the results than non-monetised content is very different from not appearing in the results at all. > How is commercial content supposed to be funded? They'll find a way. After all, if more people turn to ChatGPT for queries than to search engines, those sites are under the same sink-or-swim pressure that they would be under if demonetised search engines were dominant.
      - thfuran 2y ago
        >Appearing lower in the results than non-monetised content is very different from not appearing in the results at all. I'd say it's only technically different unless there are only a handful of results. It probably might as well not appear in results if it's past the twentieth position. On Google, click through rate is already down to ~1% by the tenth result.
dexen 2y ago
Love this. Aligns with some of my recent musings. Even better, it would be largely immune to certain recently discovered problems with AI, including ingesting AI-generated content.
raincole 2y ago
If this search engine ever gets popular, it will just push advertisers from ads to product placements. That's it. Good luck writing your own algorithm that detects product placements from genuine human experience. (It would need an AI that is smarter than average human.)
- lelanthran 2y ago
  > If this search engine ever gets popular, it will just push advertisers from ads to product placements. For me, that would be an improvement over the current 'cover content with advertisements', 'interstitials', 'user-tracking', 'privacy-abusing', 'displace legitimate search results' situation we have now. > That's it. Good luck writing your own algorithm that detects product placements from genuine human experience. Pretty trivial to detect if the content-author (whether ChatGPT or human) is going overboard. A single page exhorting the benefits of 20 different products? Unless that page is performing a comparison or review, pretty easy to pick it up just from the links alone. And if it is performing a review or comparison, it's going to get excluded from search results for people not looking for reviews or comparisons. IOW, "working as intended", for me.
  - nicce 2y ago
    > For me, that would be an improvement over the current 'cover content with advertisements', 'interstitials', 'user-tracking', 'privacy-abusing', 'displace legitimate search results' situation we have now. Now, we have possibility to separate ads and real content. With product placements, there isn't any way and you are forced to absorb all the content to see what is actually ad and real content, wasting your time endlessly.
- oneeyedpigeon 2y ago
  "Product placement" sounds vague/subjective - are the posts on Hacker News 'product placement'? I think, on average, most people would prefer a form of product placement to privacy-abusing, experience-debilitating, irrelevant ads.
  - 2y ago

amelius 2y ago

There may be a place for a green (climate-friendly) search engine, where every ad for physical products is penalized.

nicce 2y ago

> allow logged-in users to down-rank a site, etc. And then you need a huge anti-bot mechanism if the search engine will get any popularity. There is money-based motivation to affect the rankings.

nine_k 2y ago
Karma, occasional moderation rights, and occasional meta-moderation rights worked pretty well at Slashdot, for instance. It depends heavily on user involvement though.
- nicce 2y ago
  Yeah, there is a dilemma. To get a wider audience, you need very good search results. But if you search quality depends on the amount of users, how do you get in there? People like power, that keeps them moderating (looking at Facebook groups). What kind of reward system works here?
  - chongli 2y ago
    Well the baseline search quality should not rely on users. It should be great out of the box until spammers begin targeting it directly. Relying on users to moderate should happen organically as the number of users grow, which ideally would be timed to offset the growth in spam targeted at the engine.
    - nicce 2y ago
      The issue is that it needs the really really good on baseline before people start switching from other services. But if you are able provide better baseline than others, then you might not even need the users to be "better. With users, you would be then exceptional.
      - nine_k 2y ago
        I suppose services like Gmail benefit a lot from users who mark the occasional spam letter in their inboxes. While each such marking is inconsequential, correlated over many users, they allow to detect new and unusual waves of spam, and to start filtering them for everyone. A similar mechanism could work for a search engine. It would be harder to game if it required high karma (many other users matching your judgment before it starts to count), and a paid account, like with Kagi.
Aachen 2y ago
I keep thinking of a hierarchical invite model for these sorts of problems. I'd be very curious if someone could give a second opinion on this idea. The mechanism: Everyone has to be invited by someone, so it all traces back to the creator. The creator knows they themselves are legit, but let's say someone online asked for an invite and bad inputs keep coming from somewhere down that branch of the invite tree. Either the person the creator invited is a spammer, or someone they've invited in turn is the spammer. All accounts leading up to them can be progressively killed (and their inputs nullified), starting with the ones actually causing trouble → if it keeps happening in that branch then kill one layer up, and so on. Incentives: People risk losing their own account when inviting someone who they don't trust to be good netizens. Maybe there needs to be an incentive why you care about your account in the first place, or maybe (looking at Wikipedia or OpenStreetMap, or HN with its voting system and the homepage meaning a lot of attention for your page) a majority of people are simply honest and happy to contribute and that suffices Problem it solves: Wouldn't such a hierarchical invite system work around the online identification problem? If you would DNA check everyone and ban people who abused the system in the last decade, you'd also not have any spam online, but that's way too invasive (besides not being legal and prohibitively expensive, it's also not ethical). However, a pseudonymous (all that is known about you is a random user ID) invite tree seems to me like it would have similar properties. It requires banning the same person perhaps a hundred times until they run out of people that will give them invites, but wouldn't it eventually distill the honest people from the population? (Which is probably almost everyone if there is no gain from systematic cheating and there's social pressure to not ask for multiple invites because account holders know that means you were either messing with the system they enjoy using or invited someone onto it that did that.) (Implementation details: One bad input isn't an instant ban: people misclick or misunderstand interfaces, but eventually it gets to a point where, if they can't click the right buttons, there's also no point having them be moderators of the search engine (or whatever this is used for) and so their account is removed. If multiple removals happen in a tree that's deep and recent, remove more than one layer at a time to get rid of malicious sockpuppet layers. The tree's maximum depth can be limited to something on the order of 50: it doesn't take many steps to find a chain of relationships that links two random persons on the planet, so a fairly low depth is enough for the whole world. People should be told on the invitation page how many bad apples were removed in each layer below them, so if they're 1 bad apple removed from having their own account pruned then they know to only invite people they're very sure about. One problem I see with the system is that it reveals social graphs which not everyone is happy about. If that means being able to kill virtually all spam, content farms, etc., maybe it's worth it, but or course more research is needed beyond an initial proposal.)
bdw5204 2y ago
There's also the issue of users trying to downrank any polarizing site because they disagree with it. To pick an example, think of the search term "vaccine" and how strongly people on both sides feel about it. Eventually, you'll probably end up with results for "vaccine" that have nothing whatsoever to do with vaccines because everything vaccine-related will end up downranked.

flir 2y ago

It's your ranking system, so it's correct by definition (for you), but I've got to point out you're not considering the quality of the text. In your model, an empty page is perfect. Cynically, maybe you're on to something...

lelanthran 2y ago
> It's your ranking system, so it's correct by definition (for you), but I've got to point out you're not considering the quality of the text. Yes. This is by design![1] > In your model, an empty page is perfect. Yes. What's the problem with that? An empty page won't match any search terms, will it? [1] It's impossible to quickly and cheaply determine the quality of the text, in an age where it's cheaper for a blog-spammer to use AI to write 3k articles than for the crawler to use AI to check 3m articles.
- Folcon 2y ago
  You might be onto an interesting new view of the web here, if you've ever seen https://www.builtwith.com https://www.builtwith.com, something similar that also processes script tags and downweights based on what the script is would also add to this effect. At the very least I'd be interested in looking at such a site and seeing how it differs from https://www.millionshort.com https://www.millionshort.com =)...
- verisimi 2y ago
  I think this is a great idea. I really like the principle. But it does miss the point that you need a huge amount of resources. Plus 20 billion or so to get Apple to use your search.
  - Aachen 2y ago
    If it is actually as much better as it sounds, even apple users will find a way to set their search engine to what works once word of mouth gets around. (Until Apple turns that option off and waits for for the EU to pass another gatekeeper law anyway.) If it is actually as much better as it sounds, one wonders why Google or Duckduckgo doesn't do this already. For Google, killing the advertisement incentive is probably counterproductive in the long run, but why doesn't DDG?

corn13read2 2y ago

Let's collaborate

lelanthran 2y ago
If you've got capital, I'd love to :-) On a more serious note, I'm actually very busy trying to make money, so I don't really explore things that can't pay off unless they are small. Something like a crawler + engine is very capital intensive.

6510 2y ago

Each char of css scores 1 bad, a char of js is 100 bad, one small advertisement is one million bad, a cookie banner is 5 million bad, news letter modals 123 456 bad, links and embeds pointing at big tech 100 trillion bad. The pro version comes in 3 kinds: 50 cent, 10 usd and 100 usd per search. For this money you get one or more employees modifying the search results in real time. Pressing the "work harder" button increases the fee by .50,10,100 respectively. You can press it as often as you like.

acatton 2y ago

I wish you luck and I hope you succeed, but you make it sound much much easier than what it would be. First of all, you're going to drown in hardware costs, if you run your own hardware. If you run on AWS, you will be the largest AWS customer. 2 years ago, when Google was still displaying result counts, I got 1.3 billions results for "sushi"[1]. This means that if you use a reverse index to lookup your results, the "sushi" entry will be ~19GiB large, assuming you use UUIDs. If you think 90% of this is spam, and you only index the non spam (detecting spam/seo is far from trivial, but let's say you figure it out), you still need ~2GiB just for mapping "sushi". With 755,865 words in the English dictionary, according to wikipedia[2], you'll need ~1.5 PiB (yes, pebi/peta, 1,536 TiB) just to store relationships for English pages. This is assuming you don't support other languages, you discard 90% of pages, and you don't cache the content of pages for re-indexing. In addition to this, you also need to store the meta-data for each pages (vote counts from your voting system, whether it's serving different content, etc...). The order of magnitude has to be in the O(100TiB) from my conservative gut feeling. (still assuming you discard 90% of the web, and I'll assume you aggregate the metadata on the domain, not on the individual pages) The second challenge is your ranking. Now that you've become the dominant search engine with your awesome ranking system, you will become the main target for swaths of motivated click-farms which are exploiting workers from low income countries. They will be trying to register accounts, vote and game your ranking. You can most likely detect this behaviour, but their behaviour will be very similar to a significant portion of your real users. So you'll be fishing in a pond with a rocket launcher, and some of your legitimate users will be collateral victims. Otherwise, you'll spend most of your time playing a cat-and-mouse game with the SEO spammers instead of improving your search engine and fixing bugs. I'm also falling in the trap "i could rewrite that in a weekend" sometimes, but for a search engine, I would love to see decent competition, but it's near impossible. [1] https://news.ycombinator.com/item?id=30925402 https://news.ycombinator.com/item?id=30925402 [2] https://en.wikipedia.org/w/index.php?title=List_of_dictionaries_by_number_of_words&oldid=1223696888 https://en.wikipedia.org/w/index.php?title=List_of_dictionar...

adileo 2y ago
Absolutely spot on. Additionally, it's worth mentioning that a lot of content is now locked behind a few major platforms (eg. Facebook, LinkedIn, Medium, YouTube, etc.) or CDNs like Cloudflare, which often block crawling from non-Google IPs or well-known search engines. While the other costs mentioned here can be optimized with current hardware prices and a good database, anti-crawling measures necessitate thousands of IPs/proxies, making the process even more challenging and costly.

Ferret7446 2y ago

> There is an SEO-proof way to determine what the ranking of a site should be This comment deserves a "my sweet summer child". Everything can be gamed; if you don't see how, then you really shouldn't be suggesting ideas. How do you define an advertisement? How do you prevent downvote spam (if captchas worked, there wouldn't be bots/astroturfers on the Internet)? How will you fund your search engine?

codetrotter 2y ago
> How do you define an advertisement? Headless browser with uBlock Origin. Check programmatically how many ads were blocked on the page.
- fl0id 2y ago
  But many ads and crap today are actually not traditional ads. Don’t they count? If not, then gpt generated text ads or whatever will become more prevalent
lelanthran 2y ago
> This comment deserves a "my sweet summer child". I think being condescending really removes any impact from any point you want to make. After all, I posed this as an idle musing, not as a all-in venture, didn't I? > Everything can be gamed; if you don't see how, then you really shouldn't be suggesting ideas. If people are gaming the engine by avoiding all the penalties, then that's the exact sort of gaming I would like to see happen. I mean, you have read through the rest of the thread, here, right?

chongli 2y ago

There’s quite a lot of dark patterns to watch out for. One is including text on a page meant to be read by your crawler but which is hidden from users by various CSS tricks. To get around this you’d either need to specifically blacklist certain CSS / DOM structures or do something very radical: render the page and then attempt to OCR it back into text to match against the original text.

theturtletalks 2y ago
Remember when Google said they would penalize websites that use dark patterns like Quora, Pinterest, Twitter, etc? Google can easily penalize those websites for their dark patterns so websites and blogs by actual people rise to the top but they don’t.
- chongli 2y ago
  I think they actually do that but then specifically white-list those sites. It’s really annoying!
  - theturtletalks 2y ago
    That’s even more damning. So the established sites are playing by different SEO rules altogether?
    - chongli 2y ago
      I don’t have any proof. I’m just taking it as a given that this is what they’re doing because those sites are still engaging in cloaking despite Google making it clear they have a policy against cloaking.
      - theturtletalks 2y ago
        Is their cloaking policy still active? Felt like they enforced it at the beginning, but when big sites started using the same dark patterns, they stopped? If it’s an official policy still, would love to hear why they selectively enforce it.

hooby 2y ago

"I invented a ranking system that cannot be gamed!" By my experience, ALL ranking systems can - and will be gamed, no matter what. I'm still willing to give you the benefit of the doubt - maybe you are the one who can finally truly come up with an un-game-able system. But I'll consider that an extraordinary claim - and will require to see extraordinary proof for that.

VikingCoder 2y ago

Wikipedia shows an ad asking for money. So a hundred times a day, I open a new Wikipedia clone with no ads. I get perfect scores. Once I detect your bot has crawled it, I put ads all over it until the traffic finally goes away. No system is fool-proof.

NayamAmarshe 2y ago
But there's a difference between an embedded ad and a native ad. You can allow websites to just add an image or a section of an ad that does not hit a third-party endpoint. Native ads are not as intrusive and if they become intrusive, like a popup, you can penalize that. It sounds like a sound system.
- VikingCoder 2y ago
  This is a predator/prey relationship. OP suggested they had a fool-proof way to get around that. They're wrong. Might there be some merit in their idea? Sure. Is it so fool-proof that it would actually be worth the effort to build and maintain their own crawler and index? I highly doubt it.

mark_l_watson 2y ago

I used to play with this, a long time ago. I set up Apache Nutch on my home Linux Desktop system to crawl about 20 of my favorite and most useful sites. That was fun, and I was playing with some knowledge management ideas. If I try that again, I would probably use a Hetzner server and not run it at home. I would probably still stick with spidering only about 20 web sites.

tambourine_man 2y ago

If you allow logged-in users to down-rank a site, you already did the hardest part, which is to differentiate bots and spammers from real users. From there, you can easily add positive feedback as well, IMO. But the first part is really hard. Maybe it needs to piggyback on some centralized government emitted number, which is sad, but maybe the only solution. I'd love some biometric hash, or similar, but those decentralized IDs have been a promisse for years and have yet to catch on, for whatever reason.

tambourine_man 2y ago

I've long felt that if we separated the problem we could open the flood gates for a lot of innovation. Crawling should be its own thing. That is much easily distributed and decentralized. We could then freely experiment with algorithms and strategies on that open database.

p3rls 2y ago

Counterpoint: I'm spending thousands on stuff like APIs to graph out data for my users and would like to be compensated. The existing system could work with tweaks. I like metrics like time-on-page, load times, and "domain authority". With the right weights on the algorithm, real practical guidelines and a robust system of manual actions (hopefully as transparent as possible) I believe something like Google can last for as long as the internet does.

1vuio0pswjnm7 2y ago

"This sort of ranking cannot be gamed by SEO spammers, because ... why get their site to the top of the rankings if they can't put any advertisements on it?" Because it links to pages with advertisements on them? There are easier and more objective methods for ordering search results. Alphabetical, chronological, relevance, etc. People who grew up in a world before Google used these extensively. They were everywhere. I used these in libraries for decades. "Developers" want to pretend these methods were inferior. When I think about people trying to game alphabetical business listings by naming their business "A-1" it seems delightfully quaint compared to the sociopathatic "dark patterns" of software developers. "Rankings", a subjective measure, powered by a secret, ever-changing, hand-tweaked algorithm, are superior because they can be used strategically for commercial gain by intermediaries like Google. When the intermediary becomes wealthy and influential enough, acting as gatekeeper to vast amounts of information, then the other, objective methods might seem inferior. Google accelerates the death of objectivity. It's an advertising company, not an academic search think tank.

cj 2y ago

> why get their site to the top of the rankings if they can't put any advertisements on it? You would need to penalize any website that accepts money from users. There are a lot of SaaS companies out there writing blog spam to promote subscription products.

iamacyborg 2y ago
Some of that blog spam is genuinely useful, so you need to differentiate between content quality with brands writing content to promote their saas product.
lelanthran 2y ago
> You would need to penalize any website that accepts money from users. There are a lot of SaaS companies out there writing blog spam to promote subscription products. I don't see that as a problem, because blog-spam without advertisements is going to get down-ranked until it is exactly where it needs to be. If a single site dominates too much, then a hard cap on how how often a specific site can appear in the results is feasible, or weighing each page with a percentage of all the pages on a site (thereby penalising sites that have millions of pages). Specific exceptions can be made for wikipedia, or similar. For example, lets look at digital-ocean: Is that constant blogging something that users find helpful? Well, if a page is relevant to the search term then there's no problem. If it's not then the numbers of 'irrelevant to this search' feedback from logged-in users will gradually down-vote it until it is correctly ranked. If it's relevant but is one of 30k pages on the site that matches relevancy with that search term then it gets down-ranked. Basically, the motivation to do nothing but spam is completely eliminated. People spamming "How to do $TRIVIAL-THING" to drive engagement with their SaaS product is, I feel, a completely valid result if the search term matches relevancy with the specific blog post, and the site isn't doing a barrage of postings. Humans respond to incentives. Taking away the incentive to generate endless drivel and replacing it with the incentive to provide good content is the intention behind my proposal.
- chongli 2y ago
  Specific exceptions can be made for wikipedia, or similar. Once you go down the dark path of white-listing sites, you’ve admitted defeat. How are you going to find that upstart Reddit competitor or federated Wikipedia competitor?

underwater 2y ago

But that web doesn’t exist anymore. The dominance of Google means all the creators and authors went elsewhere. Filter out the crap and you’ll realise that there’s very little substance left.

kristopolous 2y ago
Maybe it does. I certainly write clean content. And even if it doesn't (which I think it does), disincentiving bad behavior has to start somewhere
lelanthran 2y ago
> But that web doesn’t exist anymore. The dominance of Google means all the creators and authors went elsewhere. Filter out the crap and you’ll realise that there’s very little substance left. Yes, and that signal/noise ratio will drop even further until some filtering gets done. Filtering out the crap might slow the trend.

Fethbita 2y ago

Then maybe you might be interested in Marginalia Search https://search.marginalia.nu/ https://search.marginalia.nu/

lelanthran 2y ago
> Then maybe you might be interested in Marginalia Search https://search.marginalia.nu/ https://search.marginalia.nu/ I have tried it before, but it appears to me that it dismisses all commercial content altogether, which is not what I want. That is throwing out the baby with the bathwater. For example, I searched for "The Vietnam of computer science" (Without the quotes) and it returned zero results.
- thfuran 2y ago
  >but it appears to me that it dismisses all commercial content altogether, which is not what I want Isn't it? You want to downrank for ads and downrank for paywalls. How is commercial content supposed to be funded?
  - lelanthran 2y ago
    > >but it appears to me that it dismisses all commercial content altogether, which is not what I want > Isn't it? Of course not. Appearing lower in the results than non-monetised content is very different from not appearing in the results at all. > How is commercial content supposed to be funded? They'll find a way. After all, if more people turn to ChatGPT for queries than to search engines, those sites are under the same sink-or-swim pressure that they would be under if demonetised search engines were dominant.
    - thfuran 2y ago
      >Appearing lower in the results than non-monetised content is very different from not appearing in the results at all. I'd say it's only technically different unless there are only a handful of results. It probably might as well not appear in results if it's past the twentieth position. On Google, click through rate is already down to ~1% by the tenth result.

dexen 2y ago

Love this. Aligns with some of my recent musings. Even better, it would be largely immune to certain recently discovered problems with AI, including ingesting AI-generated content.

raincole 2y ago

If this search engine ever gets popular, it will just push advertisers from ads to product placements. That's it. Good luck writing your own algorithm that detects product placements from genuine human experience. (It would need an AI that is smarter than average human.)

lelanthran 2y ago
> If this search engine ever gets popular, it will just push advertisers from ads to product placements. For me, that would be an improvement over the current 'cover content with advertisements', 'interstitials', 'user-tracking', 'privacy-abusing', 'displace legitimate search results' situation we have now. > That's it. Good luck writing your own algorithm that detects product placements from genuine human experience. Pretty trivial to detect if the content-author (whether ChatGPT or human) is going overboard. A single page exhorting the benefits of 20 different products? Unless that page is performing a comparison or review, pretty easy to pick it up just from the links alone. And if it is performing a review or comparison, it's going to get excluded from search results for people not looking for reviews or comparisons. IOW, "working as intended", for me.
- nicce 2y ago
  > For me, that would be an improvement over the current 'cover content with advertisements', 'interstitials', 'user-tracking', 'privacy-abusing', 'displace legitimate search results' situation we have now. Now, we have possibility to separate ads and real content. With product placements, there isn't any way and you are forced to absorb all the content to see what is actually ad and real content, wasting your time endlessly.
oneeyedpigeon 2y ago
"Product placement" sounds vague/subjective - are the posts on Hacker News 'product placement'? I think, on average, most people would prefer a form of product placement to privacy-abusing, experience-debilitating, irrelevant ads.
- 2y ago