8 ms·

> Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal i

> Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out. It's a curation problem. We have billions of pieces of content being created and only a handful of curators to bring that info to us. All the big search engine players want to be "the one" but it's a fools errand IMO. No one search engine will be able to become "the one" because there is simply too much information. Google with all of their cash in the bank and talented engineers can't even crack the problem. The problem though is even though each search engine offers something different, they all think they offer the same thing and compete to muscle out any smaller players. So you get a situation where you want a metal roof for your house, metal roofer gives you a quote. Big roofer guy in town hears about this and runs the metal roofer guy out of town, then tells you he cant offer you a metal roof anyways.

rtpg 5y ago
Who even "gets content" through searching? I'm sure many people here are all basically using the internet through link aggregators (reddit, fb, HN etc). Has the web ever been much beyond that? Have people ever really been just hitting "give me a random web page about X"? Of course search engines are very useful, just that these conversations about content and "the web", I feel like there's not much focus on how so much traffic is guided through link aggregation.
- dgb23 5y ago
  I often search for articles/books about X, or reviews about Y. The top results for those things is a mix of SEO spam, Amazon links etc. which I all try to avoid. I use search for information, but I often have to dig quite deep to find something worth clicking on or reading. And that's true for Google and DDG (which I mostly use). When it's raw, basic knowledge, then I'm using !wiki or !mdn etc. in DDG to search those sites directly. Note: Wikipedia and Mozilla Developer Network are two of my favorite sites. But when it's something more subtle and specialized that I'm not very familiar with then often my only hope of finding something useful is to search via aggregators and forums like HN, Reddit etc. because then I can gauge the quality of the links via the discussion around it and the people who post them. Or in some cases I know about a specialist or journalist and can use their name to filter out spam. Discovering these kind of things via search is really hard, especially if their authors are not known. It's the spammers and low effort creators that get rewarded the most, partly because of search engines, partly because of visitors who are content with that kind of quality.
  - godshatter 5y ago
    Whenever I'm searching for something (using DDG, too) I usually scroll down the results page until I see a website that I don't recognize or one I remember as being small and targeted. The big company websites usually have shortest blurbs on the topic and the worst "NoScript minigames". Smaller websites usually have more information and less reliance on massive ad networks. This is especially true if you are looking for reviews on games.
    - dgb23 5y ago
      I don’t get to play video games anymore as often as a couple of years ago. Then I was naturally informed about interesting titles (typically indie). Now when I have a couple of hours to kill and would like to play something new/original I usually give up after 20min of searching and grab a book. I wish I knew some decent curators or magazines, but most of the stuff I come across is just too far from what I‘m looking for. Search engines don’t help much here either.
      - marginalia_nu 5y ago
        If you find yourself looking for a game in the future, just get "Vampire Survivors". Trust me. It's the pinnacle of casual gaming. $3 and as simple as it is plain fun. Can easily spend 10 hours on it if you want to, or 10 minutes if you don't have any more time.
        dgb23 5y ago
        Thank you, will check it out! Some of these raw looking indie games have turned out to be gems. Thinking of Faster than Light, Loop Hero etc. It's hard to gauge the game-play from a quick look. Not even "similar to" searches help too much. It's not really about the money spent, but rather about the time both finding and then playing. I wish I had a list of curators that I really dig. So far, hit and miss.
      - godshatter 5y ago
        I keep an eye on r/gamingsuggestions on reddit for game discovery. Follow that for long enough and you'll hear some more interesting games talked about upon occasion enough times that you'll seek them out. If you can stand reddit, that is.
- marginalia_nu 5y ago
  I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you. Link aggregators are also very much part of the problem. Besides the astroturfing, they tend to favor the lowest common denominator in content. Popular vote just doesn't seem to be a good way of surfacing quality content, especially not with the social incentive of winning internet acclaim for submitting a good result. HN does surface the odd gem, but why can't I pick one of those and say "Hey, this topic interests me, show me more like this"? I'm currently working on build something to that effect, and so far it seems at least reasonably possible. RSS was good in one way, but it also sort of turns everything it comes into contact with into a blog.
  - fishtacos 5y ago
    >> I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you. Why not start it? One of the beauties of reddit subgroups is that not only do you find random very specific ones, but if you don't, you can start your own.
    - marginalia_nu 5y ago
      I've started subreddits for niche topics. Only visitors I got were spambots.
      - dylan604 5y ago
        What search engine were people using to find your niche topic's thread. Oh, wait
Shacklz 5y ago
> Google with all of their cash in the bank and talented engineers can't even crack the problem. "Can't they" or "won't they" because it's simply not in their business interest?
- DannyBee 5y ago
  Ah yes, the amazing hn cynicism - it's not that the problem is really hard, and there is a really complex system here with all sorts of incentives, variables, and issues on all sides. Instead, everything is easy, and the reason the "right" thing (as defined by the poster) doesn't happen can be reduced to some single cartoon-villain level bullet point like "not in business interest". Small problem: There are plenty of folks, foundations, companies who are not resource limited in attacking something like this, yet it still doesn't happen. So maybe it's not as simple as you think?
  - marginalia_nu 5y ago
    I don't think this is particularly cynical. Ads are how Google makes money. Show results with many ads, make money. Show results with no ads, make no money. I don't think it's an example of mustache twisting villainy, but rather their business model. I can't really blame them, either.
  - dylan604 5y ago
    Please, the cynicism pointed in the Googs direction is well deserved. It is clear and obvious by their behaviour and changes to company ethos that serving ads is priority #1. Any thing that diminishes that ability will not be tolerated. Just look at the recent (as of yesterday) stories of the changes that Googs is pushing to re-enable tracking that has been previously manually/intentionally disabled by the users. Look at how Maps is forcing users to enable wifi in order to provide routing directions. This one happened to me personally as well.
    - DannyBee 5y ago
      So you don't at all address whether the problem is hard, and instead deflect completely. The wifi thing is just a silly example. The wifi is required to do fast start GPS. 99.99999999% of their users probably don't want slow start GPS. If you do, sure, it's no longer the app for you. Yes, they are optimizing for the customers they want and giving them a good experience . I'm not sure exactly how this is something abnormal. Like, you are basically complaining they won't add knobs for you. That's pretty standard in good products - but also means if the knobs they aren't what you want, it's not a good fit for you. I got the email in the latter, and that is a super weird take. They had one control for literally everything, which was owned by admins. They almost certainly got complaints that turning it off it made search suggestions not work in sheets/docs/etc. So they introduced a new separate control for that, and made it controllable by the user. Because it's controllable by the user, it also means the admin can't see the doc search history of the user, only the user can. Nor can they delete your history for you - only you can control it. Your auto-delete preferences, etc apply to it. None of this seems bad or horrible to me. The only thing even worth picking a bone about is whether the new setting should take it's default from the old setting or not. There are issues in there of course, because as mentioned, before admins could turn it on/off for everyone - and this led to lots of complaints. do you take the admin setting? Do you take the setting the user had if it hadn't been overridden by the admin? etc The claim, in that story, i that it is forced on by default even if you had it off. This claim is wrong, afaict. It was not worth wading into that clear firestorm of a thread. The support doc clearly says: "If you disabled Web & App Activity in the Admin console, it is turned off for Google Workspace users. Users can turn it on if they want." IE if the admin had it disabled, it will be turned off. It also says if you had it disabled yourself, it will default to off. It goes through this quite clearly if you click "what if i had it disabled" and "what happens if i don't do anything". Overall presenting this as "re-enable tracking that was manually/intentionally disabled by users" is a real stretch.
new_guy 5y ago
> Google with all of their cash in the bank and talented engineers can't even crack the problem. Search is a solved problem. But you're starting from the false premise that Google and your expectations are aligned, they're not. Google shows you what they want you to see, typically spam sites carrying their ads. That's not what you're looking for but from their perspective they've 'solved search'.
- dylan604 5y ago
  I see nothing wrong with this comment, and find my understanding of Google to be very same.
eitland 5y ago
> Google with all of their cash in the bank and talented engineers can't even crack the problem. Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away. Ingo Molnar From a message to the Linux Kernel mailing list in 1999.
- shepherdjerred 5y ago
  Can you explain? Was there a competitor to Linux that never surfaced?
  - quotehelp9128 5y ago
    Here's the mailing list entry referenced: http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html
  - fuzzer37 5y ago
    I assume they're talking about all the OS research done at places like XEROX and Bell Labs that never really came to fruition, but formed the basis for modern OS's.
  - pdonis 5y ago
    I wonder if it was a veiled reference to the debate between Linus and Andrew Tanenbaum, who wrote MINIX based on OS research that, according to him, showed that microkernels are better than monolithic kernels. https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_debate https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
    - monetus 5y ago
      what is the ratio between dynamic and static content amongst say the top100 web sites on the web? Can we say 0% static pages? [yes, i accept Dean Gaudet's point that webpages can be split up into dynamic and static parts, probably that will be the future.] This is delightful
    - 5y ago
hinkley 5y ago
PageRank worked when humans were creating content across a huge number of sites. Once you start consolidating the humans (Medium), then the organic content and the machine generated content start to blur together. And the machine sites can collude to make their content look more organic than organic. Today there's 10x as much of it so if only 20% of the organic content is high quality, that's practically in the noise floor for the sorts of graphs I stare at in any given week. Perhaps what's missing here is a garbage collection algorithm. Whatever they are using as 'roots' is not working, and they can no longer infer trust by association. They need to curate and review sites. Or maybe they already do, but the humans' focus is directed by heuristics and those heuristics have been gamed. Does anyone know if Google still has a functioning QA department? Perhaps this is the same disease we see everywhere of developers thinking they understand testing when really we know just enough to be dangerous.
- onion2k 5y ago
  PageRank worked when humans were creating content across a huge number of sites. PageRank worked when there was very little tangible value besides vanity in being number one for a particular search term, and when there was little competition between sites. Once SEO became a key strategy to growing a business PageRank had to shift from finding the most relevant results to removing the things that were pretending to be relevant results. That challenge is much harder. I contend that practically every update to Google Search over the past 15 years has been to try to defeat SEO spam. There doesn't appear to be anything else. And, unfortunately for all of us, it's a battle Google are not winning.
  - hinkley 5y ago
    I can’t fault any of your reasoning. Perhaps it might be accurate to say that as email has declined in popularity, “spam” has simply moved to the Web.
    - xp84 5y ago
      Indeed - and a glance at the ads on even mainstream sites sure back that up. They're mostly shilling casino mobile games, scam tech (like 'portable air conditioners' that are just a fan that blows over a wet sponge for $75) and the big one, unregulated dietary supplements sold with disgusting imagery and "One weird trick" ad copy. It is almost entirely just utter bottom-feeding trash. And this is what's paying the bills for 'reputable' sites like CNN, Bloomberg, CNBC, whatever.
ColinHayhurst 5y ago
A fools errand indeed. But we currently aren't far from "the one" in three world(view)s: China - Baidu and Sogou; Russia - Yandex; Elsewhere - Google and Bing/syndicates. Korea (Naver) and Czechia/Slovakia (Seznam) are not international but local exceptions. We choose not to offer the same thing based on our values; no-tracking and infrastructural independence, meaning we aren't captured like other search options: https://blog.mojeek.com/2021/03/to-track-or-not-to-track.html https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm...
megous 5y ago
For me it would be enough if RSS feeds could include <recommended-feed href="https://someone-elses/rss.xml https://someone-elses/rss.xml" reason="Friend's travel blog" /> and RSS reader allowed to browse through the web of recomendations and add allowed to add selected ones to my feedreader list. A potentially standard way to find other related feeds, instead of just consuming the content of one author/website. Curated by real humans that produce useful content (that's why I have them subscribed). Simple thing that could have been very powerful for discovery if this was thought about early enough.
- johnny22 5y ago
  webrings for rss?
  - megous 5y ago
    yep

rtpg 5y ago

Who even "gets content" through searching? I'm sure many people here are all basically using the internet through link aggregators (reddit, fb, HN etc). Has the web ever been much beyond that? Have people ever really been just hitting "give me a random web page about X"? Of course search engines are very useful, just that these conversations about content and "the web", I feel like there's not much focus on how so much traffic is guided through link aggregation.

dgb23 5y ago
I often search for articles/books about X, or reviews about Y. The top results for those things is a mix of SEO spam, Amazon links etc. which I all try to avoid. I use search for information, but I often have to dig quite deep to find something worth clicking on or reading. And that's true for Google and DDG (which I mostly use). When it's raw, basic knowledge, then I'm using !wiki or !mdn etc. in DDG to search those sites directly. Note: Wikipedia and Mozilla Developer Network are two of my favorite sites. But when it's something more subtle and specialized that I'm not very familiar with then often my only hope of finding something useful is to search via aggregators and forums like HN, Reddit etc. because then I can gauge the quality of the links via the discussion around it and the people who post them. Or in some cases I know about a specialist or journalist and can use their name to filter out spam. Discovering these kind of things via search is really hard, especially if their authors are not known. It's the spammers and low effort creators that get rewarded the most, partly because of search engines, partly because of visitors who are content with that kind of quality.
- godshatter 5y ago
  Whenever I'm searching for something (using DDG, too) I usually scroll down the results page until I see a website that I don't recognize or one I remember as being small and targeted. The big company websites usually have shortest blurbs on the topic and the worst "NoScript minigames". Smaller websites usually have more information and less reliance on massive ad networks. This is especially true if you are looking for reviews on games.
  - dgb23 5y ago
    I don’t get to play video games anymore as often as a couple of years ago. Then I was naturally informed about interesting titles (typically indie). Now when I have a couple of hours to kill and would like to play something new/original I usually give up after 20min of searching and grab a book. I wish I knew some decent curators or magazines, but most of the stuff I come across is just too far from what I‘m looking for. Search engines don’t help much here either.
    - marginalia_nu 5y ago
      If you find yourself looking for a game in the future, just get "Vampire Survivors". Trust me. It's the pinnacle of casual gaming. $3 and as simple as it is plain fun. Can easily spend 10 hours on it if you want to, or 10 minutes if you don't have any more time.
      - dgb23 5y ago
        Thank you, will check it out! Some of these raw looking indie games have turned out to be gems. Thinking of Faster than Light, Loop Hero etc. It's hard to gauge the game-play from a quick look. Not even "similar to" searches help too much. It's not really about the money spent, but rather about the time both finding and then playing. I wish I had a list of curators that I really dig. So far, hit and miss.
    - godshatter 5y ago
      I keep an eye on r/gamingsuggestions on reddit for game discovery. Follow that for long enough and you'll hear some more interesting games talked about upon occasion enough times that you'll seek them out. If you can stand reddit, that is.
marginalia_nu 5y ago
I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you. Link aggregators are also very much part of the problem. Besides the astroturfing, they tend to favor the lowest common denominator in content. Popular vote just doesn't seem to be a good way of surfacing quality content, especially not with the social incentive of winning internet acclaim for submitting a good result. HN does surface the odd gem, but why can't I pick one of those and say "Hey, this topic interests me, show me more like this"? I'm currently working on build something to that effect, and so far it seems at least reasonably possible. RSS was good in one way, but it also sort of turns everything it comes into contact with into a blog.
- fishtacos 5y ago
  >> I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you. Why not start it? One of the beauties of reddit subgroups is that not only do you find random very specific ones, but if you don't, you can start your own.
  - marginalia_nu 5y ago
    I've started subreddits for niche topics. Only visitors I got were spambots.
    - dylan604 5y ago
      What search engine were people using to find your niche topic's thread. Oh, wait

Shacklz 5y ago

> Google with all of their cash in the bank and talented engineers can't even crack the problem. "Can't they" or "won't they" because it's simply not in their business interest?

DannyBee 5y ago
Ah yes, the amazing hn cynicism - it's not that the problem is really hard, and there is a really complex system here with all sorts of incentives, variables, and issues on all sides. Instead, everything is easy, and the reason the "right" thing (as defined by the poster) doesn't happen can be reduced to some single cartoon-villain level bullet point like "not in business interest". Small problem: There are plenty of folks, foundations, companies who are not resource limited in attacking something like this, yet it still doesn't happen. So maybe it's not as simple as you think?
- marginalia_nu 5y ago
  I don't think this is particularly cynical. Ads are how Google makes money. Show results with many ads, make money. Show results with no ads, make no money. I don't think it's an example of mustache twisting villainy, but rather their business model. I can't really blame them, either.
- dylan604 5y ago
  Please, the cynicism pointed in the Googs direction is well deserved. It is clear and obvious by their behaviour and changes to company ethos that serving ads is priority #1. Any thing that diminishes that ability will not be tolerated. Just look at the recent (as of yesterday) stories of the changes that Googs is pushing to re-enable tracking that has been previously manually/intentionally disabled by the users. Look at how Maps is forcing users to enable wifi in order to provide routing directions. This one happened to me personally as well.
  - DannyBee 5y ago
    So you don't at all address whether the problem is hard, and instead deflect completely. The wifi thing is just a silly example. The wifi is required to do fast start GPS. 99.99999999% of their users probably don't want slow start GPS. If you do, sure, it's no longer the app for you. Yes, they are optimizing for the customers they want and giving them a good experience . I'm not sure exactly how this is something abnormal. Like, you are basically complaining they won't add knobs for you. That's pretty standard in good products - but also means if the knobs they aren't what you want, it's not a good fit for you. I got the email in the latter, and that is a super weird take. They had one control for literally everything, which was owned by admins. They almost certainly got complaints that turning it off it made search suggestions not work in sheets/docs/etc. So they introduced a new separate control for that, and made it controllable by the user. Because it's controllable by the user, it also means the admin can't see the doc search history of the user, only the user can. Nor can they delete your history for you - only you can control it. Your auto-delete preferences, etc apply to it. None of this seems bad or horrible to me. The only thing even worth picking a bone about is whether the new setting should take it's default from the old setting or not. There are issues in there of course, because as mentioned, before admins could turn it on/off for everyone - and this led to lots of complaints. do you take the admin setting? Do you take the setting the user had if it hadn't been overridden by the admin? etc The claim, in that story, i that it is forced on by default even if you had it off. This claim is wrong, afaict. It was not worth wading into that clear firestorm of a thread. The support doc clearly says: "If you disabled Web & App Activity in the Admin console, it is turned off for Google Workspace users. Users can turn it on if they want." IE if the admin had it disabled, it will be turned off. It also says if you had it disabled yourself, it will default to off. It goes through this quite clearly if you click "what if i had it disabled" and "what happens if i don't do anything". Overall presenting this as "re-enable tracking that was manually/intentionally disabled by users" is a real stretch.

new_guy 5y ago

> Google with all of their cash in the bank and talented engineers can't even crack the problem. Search is a solved problem. But you're starting from the false premise that Google and your expectations are aligned, they're not. Google shows you what they want you to see, typically spam sites carrying their ads. That's not what you're looking for but from their perspective they've 'solved search'.

dylan604 5y ago
I see nothing wrong with this comment, and find my understanding of Google to be very same.

eitland 5y ago

> Google with all of their cash in the bank and talented engineers can't even crack the problem. Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away. Ingo Molnar From a message to the Linux Kernel mailing list in 1999.

shepherdjerred 5y ago
Can you explain? Was there a competitor to Linux that never surfaced?
- quotehelp9128 5y ago
  Here's the mailing list entry referenced: http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html
- fuzzer37 5y ago
  I assume they're talking about all the OS research done at places like XEROX and Bell Labs that never really came to fruition, but formed the basis for modern OS's.
- pdonis 5y ago
  I wonder if it was a veiled reference to the debate between Linus and Andrew Tanenbaum, who wrote MINIX based on OS research that, according to him, showed that microkernels are better than monolithic kernels. https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_debate https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
  - monetus 5y ago
    what is the ratio between dynamic and static content amongst say the top100 web sites on the web? Can we say 0% static pages? [yes, i accept Dean Gaudet's point that webpages can be split up into dynamic and static parts, probably that will be the future.] This is delightful
  - 5y ago

hinkley 5y ago

PageRank worked when humans were creating content across a huge number of sites. Once you start consolidating the humans (Medium), then the organic content and the machine generated content start to blur together. And the machine sites can collude to make their content look more organic than organic. Today there's 10x as much of it so if only 20% of the organic content is high quality, that's practically in the noise floor for the sorts of graphs I stare at in any given week. Perhaps what's missing here is a garbage collection algorithm. Whatever they are using as 'roots' is not working, and they can no longer infer trust by association. They need to curate and review sites. Or maybe they already do, but the humans' focus is directed by heuristics and those heuristics have been gamed. Does anyone know if Google still has a functioning QA department? Perhaps this is the same disease we see everywhere of developers thinking they understand testing when really we know just enough to be dangerous.

onion2k 5y ago
PageRank worked when humans were creating content across a huge number of sites. PageRank worked when there was very little tangible value besides vanity in being number one for a particular search term, and when there was little competition between sites. Once SEO became a key strategy to growing a business PageRank had to shift from finding the most relevant results to removing the things that were pretending to be relevant results. That challenge is much harder. I contend that practically every update to Google Search over the past 15 years has been to try to defeat SEO spam. There doesn't appear to be anything else. And, unfortunately for all of us, it's a battle Google are not winning.
- hinkley 5y ago
  I can’t fault any of your reasoning. Perhaps it might be accurate to say that as email has declined in popularity, “spam” has simply moved to the Web.
  - xp84 5y ago
    Indeed - and a glance at the ads on even mainstream sites sure back that up. They're mostly shilling casino mobile games, scam tech (like 'portable air conditioners' that are just a fan that blows over a wet sponge for $75) and the big one, unregulated dietary supplements sold with disgusting imagery and "One weird trick" ad copy. It is almost entirely just utter bottom-feeding trash. And this is what's paying the bills for 'reputable' sites like CNN, Bloomberg, CNBC, whatever.

ColinHayhurst 5y ago

A fools errand indeed. But we currently aren't far from "the one" in three world(view)s: China - Baidu and Sogou; Russia - Yandex; Elsewhere - Google and Bing/syndicates. Korea (Naver) and Czechia/Slovakia (Seznam) are not international but local exceptions. We choose not to offer the same thing based on our values; no-tracking and infrastructural independence, meaning we aren't captured like other search options: https://blog.mojeek.com/2021/03/to-track-or-not-to-track.html https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm...

megous 5y ago

For me it would be enough if RSS feeds could include <recommended-feed href="https://someone-elses/rss.xml https://someone-elses/rss.xml" reason="Friend's travel blog" /> and RSS reader allowed to browse through the web of recomendations and add allowed to add selected ones to my feedreader list. A potentially standard way to find other related feeds, instead of just consuming the content of one author/website. Curated by real humans that produce useful content (that's why I have them subscribed). Simple thing that could have been very powerful for discovery if this was thought about early enough.

johnny22 5y ago
webrings for rss?
- megous 5y ago
  yep