6 ms·

there's a paper that basically shows these LLMs are pretty much tapped out in terms of quality data to train on, not sure if there are any legit architecture im

there's a paper that basically shows these LLMs are pretty much tapped out in terms of quality data to train on, not sure if there are any legit architecture improvements in the works to improve results quality but it seems brute force training with more data may be about over - https://arxiv.org/pdf/2211.04325.pdf https://arxiv.org/pdf/2211.04325.pdf As for SEO, Google is going to have to stop being lazy and relying so much on backlinks and domain authority. They might have to consider allowing searchers to essentially vote on what they consider good. I don't see any other way low quality AI content doesn't dominate SERPs, established players will spin up entire sites and spam backlinks to them all. There are also people working on using AI to identify obvious AI content by finding patterns in the output which is probably our only hope. For social media and forums they will have to rely even harder on captcha or they will be taken over by bots talking to other bots

kneebonian 4y ago
An interesting idea but really then aren't you just recreating reddit to an extent. The things people like or that are popular are upvoted because the voter agrees whereas what is unpopular gets down voted regardless of veracity because people like the way it sounds. Beyond that a once popular website or producer says or does something that offends the sensibilities of the internet crowd and then everyone goes and down votes their results because it violates the current ziegiest
- ren_engineer 4y ago
  >aren't you just recreating reddit to an extent there are tons of ways to prevent that by weighting votes and only taking a random sample. SEO is basically a war where SEOs find a way to exploit Google's algorithm and Google tries to respond. Backlinks were Google's original solution to keyword spam that made other search engines suck, they didn't expect people would create entire sites or pay other websites to send links to them so they could rank higher. Backlinks were themselves considered to be "votes" for quality similar to citations in academia. The problem is the barrier to entry to make a site and link to it is high, so SEOs themselves are often the only ones to do it. Adding some sort of voting mechanism would allow actual readers to give an opinion on quality, although you'd probably see SEOs pay people to vote on their link via a VPN switching IPs, so only people with established Google accounts could be allowed to vote. Obviously its a huge challenge, which is why Google is worth hundreds of billions for even partially solving it
  - debesyla 4y ago
    I guess a return of "don't show this page in my results" could work better: it removes the worst results and it's hard to fake, no influencer is going to pay to downvote 200 pages and if they do, it would be easy to see that same user downvoted everything everything at random, without even clicking trough to read. Helps for both search users and the platform. Like a free moderation service. Why have an employee too check if these pages should be removed if a crowd of tracked users can do it. I wonder where is my view wrong because I wonder why isn't there such system yet...
    - guiambros 4y ago
      You'd still have to deal with adversarial spam. Competitor A pays a botnet to visit and downvote Competitor B's links at will, thus tanking its ranking. In order to detect malicious activity you'd have to use a bunch of other signals, and also to use the downvotes with some suspicion, so you're back at looking content quality, reputation, page speed, and many other signals. Which sounds very similar to status quo. Personally I'd like to see some sort of explicit input. If not used collectively, at least to customize my own results, like it happens on YT/Netflix/Instagram/etc - e.g. "never show results again from this site", or "I dislike this type of content".
- pixl97 4y ago
  >The things people like or that are popular are upvoted because You forgot to add "Because clever posters use bots to give an early upvote boost to their posts, and downvote competing posts" And the more eyeballs on the site, the more things will be manipulated this way by bad actors.
  - giantrobot 4y ago
    Yeah, it's nice to see the best parts of Digg survived! /s
- DoughnutHole 4y ago
  People are already appending "reddit" to their search queries to filter out the SEO blogspam clogging up the results of a regular search. Reddit is already superior to base search results for anything that's not plain uncontroversial facts because at least you're getting an actual human's opinion, which you can treat and evaluate as such. For all its flaws maybe becoming like reddit it what search needs. AI generated spam will likely always be better at gaming a search algorithm than a human. Human moderation has its flaws but at least we're very familiar with them - they might be worth dealing with if it keeps the internet vaguely usable.
  - lamontcg 4y ago
    Lots of reddit is already pretty useless, drowning in people who are desperate to sound clever by regurgitating one of the same 50 meme jokes with some minor twist to it. And the astroturfing on the political reddits are likely to start hitting the niche ones with products getting pushed by manufacturers. Just buy accounts that have been built by bots on front-page subreddits, then have them make AI-driven positive comments on the niche subreddits and now you've got organic-looking advertisement for your product. Moderators likely won't help since they are there mostly to keep the trolls out (or they're more focused on whatever political axe they're grinding).
    - duskwuff 4y ago
      Reddit is also having some pretty serious problems with bots reposting highly upvoted content, both in posts and comments. Sometimes it gets spotted -- especially when the bots pick content which was specific to some point in time, or when they repost a comment out of context -- but a lot of it goes unnoticed, especially in comment threads.
  - spoils19 4y ago
    Myself, I append hackernews to my search queries as the discourse quality is much higher as well as the communities maturity. Vox populi, Vox de!
CuriouslyC 4y ago
While there is certainly a lot of room for progress in architecture to improve training time, model size and inference speed, high quality data and curated data sets are the ultimate bottleneck in terms of what AI can actually do. Low quality data can be useful to a point, but once you have enough data all it's doing is teaching the model to produce low quality output. The next wave of major progress in AI will come when we create image/text data sets that have been heavily annotated with aesthetic judgment metadata, and train models to make aesthetic judgments, then condition generative models on aesthetic judgment scores during training.
- eternalban 4y ago
  "The key roadblock to achieving human-level language understanding in AI systems is the lack of a comprehensive model of human language and cognition. While large amounts of data and powerful algorithms can help improve the performance of language models, ultimately a better understanding of how human language works and how it is processed by the brain is necessary to build truly intelligent language systems. In addition, the ability to integrate multiple modalities such as vision, speech, and gesture, and to handle the complexity and variability of real-world language use are also important challenges to be addressed." -- you know who
nradov 4y ago
My understanding is that Google does essentially allow searchers to "vote" on what they consider good by looking for bounces. If the searcher follows on a search results link and then quickly returns to the search page then it's considered a failure.
- ren_engineer 4y ago
  true, but that's part of the reason for the life stories being told on recipe sites that the parent comment was talking about. Bounce rate is another metric that SEOs have gamed by throwing a bunch of fluff onto pages where people have to dig to find what they want
  - TeMPOraL 4y ago
    So that's why they're doing it. I've been wondering for a while now just why so many sites make you work hard to fish out the information you're after. I assumed ad exposure, but I had this feeling that it's not the whole story. Delaying people to fool Google bounce metrics is the idea I missed.
    - thefreeman 4y ago
      that’s not actually why they do it. it’s because recipes are not copyrightable. so they throw in a “story” to make it something they can copyright and try to prevent people from duplicating.
      - TeMPOraL 4y ago
        That's specific to recipe. Here I meant in general - e.g. blogs and news articles that bury the core point / answer somewhere around the final 1/3 of text, but also don't seem to exploit the artificially elongated text to expose you to more ads.
      - at-w 4y ago
        How would having additional content above the recipe change the copyright status of the recipe itself, divorced from the additional content? It prevents directly copying the entire article, but not the actual recipe (and Google already heavily penalizes exact duplicate content). What would make more sense is that they are actually taking the recipes from elsewhere and adding the stories to avoid directly duplicating another page, and the Google ranking penalty associated with exact or close duplicates. My moderate experience with SEO suggests it's a combination of factors. First, Google generally rewards longer content (https://www.sweor.com/seocontentlength https://www.sweor.com/seocontentlength), or at least content longer than the few hundred words a simple recipe would likely be limited to. Secondly, there is (mixed) evidence that Google rewards sites on which people spend longer before clicking back. Thirdly, the longer format allows them to insert more keywords/longer tail keywords vs. a short format where inserting lots of keywords would be seen as keyword stuffing. I'd also guess that as much as some people dislike superfluous stories when looking for a recipe, it is a positive for a subset of users that are more likely to subscribe to/return to that site directly in the future.
fnordpiglet 4y ago
Google doesn’t actually have to do anything at all other than slurp personal data up and use it to improve ad margins. Once your the default search engine, the default home page, and the default browser, you’ve locked 99% of the world into your portal. You control what people see and do, and they have to accept it. That’s what triggered the browser wars of the 1990’s - the realization that that trifecta is the keys to world domination. Saying “yeah but Kagi” is talking about you and your tribe only. The world isn’t going to pay for Kagi or take the N steps required to switch. Microsoft realized they could kill Netscape by simply not requiring the user to download anything, then failed to consolidate. Google managed to be better for just long enough to consolidate so pervasively that any further investment in improvement is a literal waste of energy and money. So, don’t hold your breath from an improvement in SEO garbage. It’ll just get worse. Definitely use Kagi and hopefully we will have great niche subscription based tools that have an incentive to be better. But don’t expect that Google perceives ANY problems with the status quo or that they have to do anything about anything other than increasing ad margins.
janalsncm 4y ago
I’m not sure it’s Google’s fault at this point. Maybe it’s just that every site thinks they need to be SEO spam in order to be noticed. And so the internet has become a cesspool of stories about how this is the same recipe passed down in your family since Julius Caesar. There have been a ton of articles about how Google sucks lately. Well, maybe Google has gotten worse, but maybe the internet has just gotten worse. If I search for “best gaming laptop” or “taco recipe” the results are pretty much uniformly spammy across Google, Bing, and Neeva, the latter being a paid search engine. All of them are just farms for ads, popups, and paid links sandwiched between AI generated copy. Is that the search engine’s fault? Is there some oasis of quality results I don’t know about? I would like to know.
- freediver 4y ago
  > If I search for “best gaming laptop” or “taco recipe” the results are pretty much uniformly spammy across Google, Bing, and Neeva, the latter being a paid search engine. I believe you will find results on Kagi less spammy https://kagi.com/search?q=best+gaming+laptop&r=us&sh=HJrdZzDIZPpoU3NiGVTVNA https://kagi.com/search?q=best+gaming+laptop&r=us&sh=HJrdZzD... (Kagi is also a paid search engine that requires an accunt, the above is publicly shared search) Removing, downranking and classyfing spam/ads is part of things Kagi is focusing on and does decently well (IMO, Kagi founder here).
  - techdragon 4y ago
    I do like to hear a founder of a still developing startup describe their current results as “decently well” … it speaks volumes about what their aim is and how they assess their current position trying to reach those goals. If you thought it wasn’t possible you wouldn’t be trying but you know you aren’t there yet. Thanks for being honest like this. I really should get around to trying Kagi … I think I’m just still in the current habit inertia between DDG and Google
  - janalsncm 4y ago
    Great to hear from someone fighting the good fight! In your opinion, is the internet just spammy now, or is something else going on?
garfieldnate 4y ago
They had upvotes working before! Just bring back google+! I found it incredibly useful to land on a page and see a high upvote in the little plugin widget I had in Chrome.

kneebonian 4y ago

An interesting idea but really then aren't you just recreating reddit to an extent. The things people like or that are popular are upvoted because the voter agrees whereas what is unpopular gets down voted regardless of veracity because people like the way it sounds. Beyond that a once popular website or producer says or does something that offends the sensibilities of the internet crowd and then everyone goes and down votes their results because it violates the current ziegiest

ren_engineer 4y ago
>aren't you just recreating reddit to an extent there are tons of ways to prevent that by weighting votes and only taking a random sample. SEO is basically a war where SEOs find a way to exploit Google's algorithm and Google tries to respond. Backlinks were Google's original solution to keyword spam that made other search engines suck, they didn't expect people would create entire sites or pay other websites to send links to them so they could rank higher. Backlinks were themselves considered to be "votes" for quality similar to citations in academia. The problem is the barrier to entry to make a site and link to it is high, so SEOs themselves are often the only ones to do it. Adding some sort of voting mechanism would allow actual readers to give an opinion on quality, although you'd probably see SEOs pay people to vote on their link via a VPN switching IPs, so only people with established Google accounts could be allowed to vote. Obviously its a huge challenge, which is why Google is worth hundreds of billions for even partially solving it
- debesyla 4y ago
  I guess a return of "don't show this page in my results" could work better: it removes the worst results and it's hard to fake, no influencer is going to pay to downvote 200 pages and if they do, it would be easy to see that same user downvoted everything everything at random, without even clicking trough to read. Helps for both search users and the platform. Like a free moderation service. Why have an employee too check if these pages should be removed if a crowd of tracked users can do it. I wonder where is my view wrong because I wonder why isn't there such system yet...
  - guiambros 4y ago
    You'd still have to deal with adversarial spam. Competitor A pays a botnet to visit and downvote Competitor B's links at will, thus tanking its ranking. In order to detect malicious activity you'd have to use a bunch of other signals, and also to use the downvotes with some suspicion, so you're back at looking content quality, reputation, page speed, and many other signals. Which sounds very similar to status quo. Personally I'd like to see some sort of explicit input. If not used collectively, at least to customize my own results, like it happens on YT/Netflix/Instagram/etc - e.g. "never show results again from this site", or "I dislike this type of content".
pixl97 4y ago
>The things people like or that are popular are upvoted because You forgot to add "Because clever posters use bots to give an early upvote boost to their posts, and downvote competing posts" And the more eyeballs on the site, the more things will be manipulated this way by bad actors.
- giantrobot 4y ago
  Yeah, it's nice to see the best parts of Digg survived! /s
DoughnutHole 4y ago
People are already appending "reddit" to their search queries to filter out the SEO blogspam clogging up the results of a regular search. Reddit is already superior to base search results for anything that's not plain uncontroversial facts because at least you're getting an actual human's opinion, which you can treat and evaluate as such. For all its flaws maybe becoming like reddit it what search needs. AI generated spam will likely always be better at gaming a search algorithm than a human. Human moderation has its flaws but at least we're very familiar with them - they might be worth dealing with if it keeps the internet vaguely usable.
- lamontcg 4y ago
  Lots of reddit is already pretty useless, drowning in people who are desperate to sound clever by regurgitating one of the same 50 meme jokes with some minor twist to it. And the astroturfing on the political reddits are likely to start hitting the niche ones with products getting pushed by manufacturers. Just buy accounts that have been built by bots on front-page subreddits, then have them make AI-driven positive comments on the niche subreddits and now you've got organic-looking advertisement for your product. Moderators likely won't help since they are there mostly to keep the trolls out (or they're more focused on whatever political axe they're grinding).
  - duskwuff 4y ago
    Reddit is also having some pretty serious problems with bots reposting highly upvoted content, both in posts and comments. Sometimes it gets spotted -- especially when the bots pick content which was specific to some point in time, or when they repost a comment out of context -- but a lot of it goes unnoticed, especially in comment threads.
- spoils19 4y ago
  Myself, I append hackernews to my search queries as the discourse quality is much higher as well as the communities maturity. Vox populi, Vox de!

CuriouslyC 4y ago

While there is certainly a lot of room for progress in architecture to improve training time, model size and inference speed, high quality data and curated data sets are the ultimate bottleneck in terms of what AI can actually do. Low quality data can be useful to a point, but once you have enough data all it's doing is teaching the model to produce low quality output. The next wave of major progress in AI will come when we create image/text data sets that have been heavily annotated with aesthetic judgment metadata, and train models to make aesthetic judgments, then condition generative models on aesthetic judgment scores during training.

eternalban 4y ago
"The key roadblock to achieving human-level language understanding in AI systems is the lack of a comprehensive model of human language and cognition. While large amounts of data and powerful algorithms can help improve the performance of language models, ultimately a better understanding of how human language works and how it is processed by the brain is necessary to build truly intelligent language systems. In addition, the ability to integrate multiple modalities such as vision, speech, and gesture, and to handle the complexity and variability of real-world language use are also important challenges to be addressed." -- you know who

nradov 4y ago

My understanding is that Google does essentially allow searchers to "vote" on what they consider good by looking for bounces. If the searcher follows on a search results link and then quickly returns to the search page then it's considered a failure.

ren_engineer 4y ago
true, but that's part of the reason for the life stories being told on recipe sites that the parent comment was talking about. Bounce rate is another metric that SEOs have gamed by throwing a bunch of fluff onto pages where people have to dig to find what they want
- TeMPOraL 4y ago
  So that's why they're doing it. I've been wondering for a while now just why so many sites make you work hard to fish out the information you're after. I assumed ad exposure, but I had this feeling that it's not the whole story. Delaying people to fool Google bounce metrics is the idea I missed.
  - thefreeman 4y ago
    that’s not actually why they do it. it’s because recipes are not copyrightable. so they throw in a “story” to make it something they can copyright and try to prevent people from duplicating.
    - TeMPOraL 4y ago
      That's specific to recipe. Here I meant in general - e.g. blogs and news articles that bury the core point / answer somewhere around the final 1/3 of text, but also don't seem to exploit the artificially elongated text to expose you to more ads.
    - at-w 4y ago
      How would having additional content above the recipe change the copyright status of the recipe itself, divorced from the additional content? It prevents directly copying the entire article, but not the actual recipe (and Google already heavily penalizes exact duplicate content). What would make more sense is that they are actually taking the recipes from elsewhere and adding the stories to avoid directly duplicating another page, and the Google ranking penalty associated with exact or close duplicates. My moderate experience with SEO suggests it's a combination of factors. First, Google generally rewards longer content (https://www.sweor.com/seocontentlength https://www.sweor.com/seocontentlength), or at least content longer than the few hundred words a simple recipe would likely be limited to. Secondly, there is (mixed) evidence that Google rewards sites on which people spend longer before clicking back. Thirdly, the longer format allows them to insert more keywords/longer tail keywords vs. a short format where inserting lots of keywords would be seen as keyword stuffing. I'd also guess that as much as some people dislike superfluous stories when looking for a recipe, it is a positive for a subset of users that are more likely to subscribe to/return to that site directly in the future.

fnordpiglet 4y ago

Google doesn’t actually have to do anything at all other than slurp personal data up and use it to improve ad margins. Once your the default search engine, the default home page, and the default browser, you’ve locked 99% of the world into your portal. You control what people see and do, and they have to accept it. That’s what triggered the browser wars of the 1990’s - the realization that that trifecta is the keys to world domination. Saying “yeah but Kagi” is talking about you and your tribe only. The world isn’t going to pay for Kagi or take the N steps required to switch. Microsoft realized they could kill Netscape by simply not requiring the user to download anything, then failed to consolidate. Google managed to be better for just long enough to consolidate so pervasively that any further investment in improvement is a literal waste of energy and money. So, don’t hold your breath from an improvement in SEO garbage. It’ll just get worse. Definitely use Kagi and hopefully we will have great niche subscription based tools that have an incentive to be better. But don’t expect that Google perceives ANY problems with the status quo or that they have to do anything about anything other than increasing ad margins.

janalsncm 4y ago

I’m not sure it’s Google’s fault at this point. Maybe it’s just that every site thinks they need to be SEO spam in order to be noticed. And so the internet has become a cesspool of stories about how this is the same recipe passed down in your family since Julius Caesar. There have been a ton of articles about how Google sucks lately. Well, maybe Google has gotten worse, but maybe the internet has just gotten worse. If I search for “best gaming laptop” or “taco recipe” the results are pretty much uniformly spammy across Google, Bing, and Neeva, the latter being a paid search engine. All of them are just farms for ads, popups, and paid links sandwiched between AI generated copy. Is that the search engine’s fault? Is there some oasis of quality results I don’t know about? I would like to know.

freediver 4y ago
> If I search for “best gaming laptop” or “taco recipe” the results are pretty much uniformly spammy across Google, Bing, and Neeva, the latter being a paid search engine. I believe you will find results on Kagi less spammy https://kagi.com/search?q=best+gaming+laptop&r=us&sh=HJrdZzDIZPpoU3NiGVTVNA https://kagi.com/search?q=best+gaming+laptop&r=us&sh=HJrdZzD... (Kagi is also a paid search engine that requires an accunt, the above is publicly shared search) Removing, downranking and classyfing spam/ads is part of things Kagi is focusing on and does decently well (IMO, Kagi founder here).
- techdragon 4y ago
  I do like to hear a founder of a still developing startup describe their current results as “decently well” … it speaks volumes about what their aim is and how they assess their current position trying to reach those goals. If you thought it wasn’t possible you wouldn’t be trying but you know you aren’t there yet. Thanks for being honest like this. I really should get around to trying Kagi … I think I’m just still in the current habit inertia between DDG and Google
- janalsncm 4y ago
  Great to hear from someone fighting the good fight! In your opinion, is the internet just spammy now, or is something else going on?

garfieldnate 4y ago

They had upvotes working before! Just bring back google+! I found it incredibly useful to land on a page and see a high upvote in the little plugin widget I had in Chrome.