4 ms·

Three thoughts: First, effective writing will almost always restate the main idea somehow; you'll want your algorithm to be more sophisticated than "demote con

Three thoughts: First, effective writing will almost always restate the main idea somehow; you'll want your algorithm to be more sophisticated than "demote content that seems to repeat itself." Second, your novel algorithm will be expensive to compute, so you'll need some other signals to filter out very low value pages before you pay to analyze others. Finally, your technique will be valuable as a ranking signal until you (or someone else) begins to use your algorithm to help steer non trivial purchasing decisions. At that point, someone will dress up their low-value (or negative value) content to appear "high value" to your algorithm and your output will be filled with spam. You can delay this effect by keeping your user-base small, so consider selling private access to institutions who want a unique view of the internet for research or intelligence purposes. Good luck!

Nextgrid 2y ago
> someone will dress up their low-value content to appear "high value" to your algorithm You are falling into the trap of thinking that Google and its competitors are victims of this. They are not. They are knowingly letting themselves get "exploited" because it turns out that those spam sites contain ads (that may be Google's) or analytics (that may be Google's), or make the search result page ads (from Google) look good in comparison and more likely to be clicked. The heuristic to detect and block spam in general is very simple. Spam is there to get you to either buy something (the signal is the presence of a credit card form, "buy" call to action, or links that ultimately lead to the former) or view ads (signal there is presence of ads). This would be very hard to game because those signals only work when they are visible (and obnoxious), which also makes them trivially detectable purely at the visual level by a crude classifier (trained on website screenshots with ads vs the same page with an ad blocker). The resulting penalty score can be used as a ranking signal, so that all other signals being equal, a result without ads would rank higher than the one with ads for a given query. This is not rocket science. The problem is that the mainstream search engines have the same business model as the spammers (and profit off each other), so there is no reason to suddenly slaughter the cash cow. The myth that search engines are "victims" to spammers/SEO provides very convenient plausible deniability so they have no reason to disprove that either.
- Mindwipe 2y ago
  This scoring is awful. Literally what you have done here is incentivise sites from well funded bad actors, or sites that push more elaborate scams on elderly people. A plausible and sustainable funding model is a good sign for sites.
  - Nextgrid 2y ago
    It's about incentivizing sites to do what the user wants them to do. As a user, 99% of the time I am not out to buy something, so all sites that do try to sell me something are just wasting both my time and their server resources. This isn't a complete blanket ban on ads or the commercial web. If there isn't a profitable way to run a website that provides the content matching the search query, the ad-infested website will still come up first. But now, a hobbyist, non-profit or even commercial enterprise that doesn't directly sell to you has a chance to outrank the ad-infested garbage.
mystified5016 2y ago
> someone will dress up their low-value (or negative value) content to appear "high value" to your algorithm and your output will be filled with spam That's where we are right now. I'm not sure that this is a solvable problem.
reaperducer 2y ago
First, effective writing will almost always restate the main idea somehow Effective corporate writing. This goes back to the era 50 years ago when public speakers were taught "Tell me what you're going to tell me, then tell me what you're telling me, then tell me what you just told me." It works fine for certain long-form speeches, but as the length of the document/talk decreases, so does the need for repetition. For the length of 90% of what's on the internet, repetition becomes redundancy and nuisance filler.
- thwarted 2y ago
  Not just length, but interactivity too. That suggestion makes sense for a long monologue that is ephemeral, but in the modern age where the content consumer is in the driver's seat and very little is actually "live", they can scroll/seek back, rereading or relistening to earlier portion, and review the summary and comments to regain context.
- sebastiennight 2y ago
  Your message above contains a quote, and four sentences. Sentence 1: hints that the quote is wrong, because only corporate writing is repetitive. Sentence 2: expands on this: corporate speech has been repetitive on purpose since the 70s. Sentence 3: contrasts corporate writing, which is repetitive, with shorter content, which does not need to be repetitive. Sentence 4: concludes that the quote above is wrong, because most Internet content (which is not corporate writing) does not need to be repetitive.
  - robwwilliams 2y ago
    Initially snarky/amusing but the post you critique is a well written paragraph in which there is a clear progression and elaboration on one major idea.
  - watwut 2y ago
    I do not think this is fair evaluation. Sentence 1: specified that quote is about one specific kind of speech. Sentence 2: explains origins of the quote - history. It is the history part that makes this sentence valuable and adds something to the reader (me). Sentence 3: states that this is not effective to the kind of content we talk about - internet articles. Sentence 4: concludes that repetition is irrelevant and wrong. ----------------------- The way you tl;dr it, the poster would need to completely change the topic to "not repeat themselves". What they did was saying something new with each sentence, just keeping the "repetition" topic on.
watwut 2y ago
> First, effective writing will almost always restate the main idea somehow; you'll want your algorithm to be more sophisticated than "demote content that seems to repeat itself." You very much not want it repeated the way it repeats now. Effective writing does not involve writing three similar paragraphs, it makes readers skip those paragraphs.
- signalToNose 2y ago
  I’ve read some books where there where some major developments/ twists that were not clearly stated and not repeated. I had to go back and reread paragraphs to understand. If you read the Bible as a fantasy novel you will come across this several times
  - watwut 2y ago
    I would argue that the "were not clearly stated" is the root cause. There is also difference between repeating some fact a reader could have forgotten and repeating whole paragraphs of fluff. The internet articles we complain about are in the latter category.

Nextgrid 2y ago

> someone will dress up their low-value content to appear "high value" to your algorithm You are falling into the trap of thinking that Google and its competitors are victims of this. They are not. They are knowingly letting themselves get "exploited" because it turns out that those spam sites contain ads (that may be Google's) or analytics (that may be Google's), or make the search result page ads (from Google) look good in comparison and more likely to be clicked. The heuristic to detect and block spam in general is very simple. Spam is there to get you to either buy something (the signal is the presence of a credit card form, "buy" call to action, or links that ultimately lead to the former) or view ads (signal there is presence of ads). This would be very hard to game because those signals only work when they are visible (and obnoxious), which also makes them trivially detectable purely at the visual level by a crude classifier (trained on website screenshots with ads vs the same page with an ad blocker). The resulting penalty score can be used as a ranking signal, so that all other signals being equal, a result without ads would rank higher than the one with ads for a given query. This is not rocket science. The problem is that the mainstream search engines have the same business model as the spammers (and profit off each other), so there is no reason to suddenly slaughter the cash cow. The myth that search engines are "victims" to spammers/SEO provides very convenient plausible deniability so they have no reason to disprove that either.

Mindwipe 2y ago
This scoring is awful. Literally what you have done here is incentivise sites from well funded bad actors, or sites that push more elaborate scams on elderly people. A plausible and sustainable funding model is a good sign for sites.
- Nextgrid 2y ago
  It's about incentivizing sites to do what the user wants them to do. As a user, 99% of the time I am not out to buy something, so all sites that do try to sell me something are just wasting both my time and their server resources. This isn't a complete blanket ban on ads or the commercial web. If there isn't a profitable way to run a website that provides the content matching the search query, the ad-infested website will still come up first. But now, a hobbyist, non-profit or even commercial enterprise that doesn't directly sell to you has a chance to outrank the ad-infested garbage.

mystified5016 2y ago

> someone will dress up their low-value (or negative value) content to appear "high value" to your algorithm and your output will be filled with spam That's where we are right now. I'm not sure that this is a solvable problem.

reaperducer 2y ago

First, effective writing will almost always restate the main idea somehow Effective corporate writing. This goes back to the era 50 years ago when public speakers were taught "Tell me what you're going to tell me, then tell me what you're telling me, then tell me what you just told me." It works fine for certain long-form speeches, but as the length of the document/talk decreases, so does the need for repetition. For the length of 90% of what's on the internet, repetition becomes redundancy and nuisance filler.

thwarted 2y ago
Not just length, but interactivity too. That suggestion makes sense for a long monologue that is ephemeral, but in the modern age where the content consumer is in the driver's seat and very little is actually "live", they can scroll/seek back, rereading or relistening to earlier portion, and review the summary and comments to regain context.
sebastiennight 2y ago
Your message above contains a quote, and four sentences. Sentence 1: hints that the quote is wrong, because only corporate writing is repetitive. Sentence 2: expands on this: corporate speech has been repetitive on purpose since the 70s. Sentence 3: contrasts corporate writing, which is repetitive, with shorter content, which does not need to be repetitive. Sentence 4: concludes that the quote above is wrong, because most Internet content (which is not corporate writing) does not need to be repetitive.
- robwwilliams 2y ago
  Initially snarky/amusing but the post you critique is a well written paragraph in which there is a clear progression and elaboration on one major idea.
- watwut 2y ago
  I do not think this is fair evaluation. Sentence 1: specified that quote is about one specific kind of speech. Sentence 2: explains origins of the quote - history. It is the history part that makes this sentence valuable and adds something to the reader (me). Sentence 3: states that this is not effective to the kind of content we talk about - internet articles. Sentence 4: concludes that repetition is irrelevant and wrong. ----------------------- The way you tl;dr it, the poster would need to completely change the topic to "not repeat themselves". What they did was saying something new with each sentence, just keeping the "repetition" topic on.

watwut 2y ago

> First, effective writing will almost always restate the main idea somehow; you'll want your algorithm to be more sophisticated than "demote content that seems to repeat itself." You very much not want it repeated the way it repeats now. Effective writing does not involve writing three similar paragraphs, it makes readers skip those paragraphs.

signalToNose 2y ago
I’ve read some books where there where some major developments/ twists that were not clearly stated and not repeated. I had to go back and reread paragraphs to understand. If you read the Bible as a fantasy novel you will come across this several times
- watwut 2y ago
  I would argue that the "were not clearly stated" is the root cause. There is also difference between repeating some fact a reader could have forgotten and repeating whole paragraphs of fluff. The internet articles we complain about are in the latter category.