5 ms·

Hardly. Scratch most topics past the surface level and if you're lucky the Web can find you the book you need. Usually though you'll need to find the best book

Hardly. Scratch most topics past the surface level and if you're lucky the Web can find you the book you need. Usually though you'll need to find the best book the Web can find you, then use that book and/or correspondence with experts to find the ones you actually need. The problem seems to be a mix of tons of stuff just not being on the Web yet, even decades in (some of it's in ebooks, though, yes, but a whole lot isn't) and Google having given up on "organizing the world's knowledge" or whatever their supposed mission was (now it's plainly "organize the world's advertising dollars into our bank account") so it may be there but good luck finding it.

pharke 6y ago
Due to the changing nature of the web I'd even say that the sum total of useful information is actually decreasing year over year. I believe we hit an inflection point a decade or more ago when the number of people with specialized knowledge and skills who were publishing information about their domain online was surpassed by the new "content creators" whose primary goal was to optimize for clicks and ad revenue. This is unevenly applied across different areas of knowledge, the amount of information about computer and media related topics has increased but other areas seem to have realized a sharp decrease. I don't have any hard numbers to support this, only the feeling of it becoming increasingly difficult to find expert information in many areas. More and more I have to resort to public domain books as the only substantive source. There's just a general sense of something deeply wrong with the current state of the web.
- nbardy 6y ago
  Even if the content creators overwhelm the rest of the crowd now, you can still find experts and their content continues to pile up. For example, iq of internet graphics is still adding incredible content to his site 10 years later. https://iquilezles.org/ https://iquilezles.org/. He's not the only example of this. There may be a higher noise to quality ratio, but the quality content is still growing, you just have to get better at tuning out the noise.
  - asdff 6y ago
    Discoverability is still hard when noise increases by the day, and search engines have become useless to find anything meaningful that isn't playing the SEO game.
SubiculumCode 6y ago
This might be field dependent. All my research is based on scientific journal articles and reviews, which are electronically available, and books are more vanity affairs
- youareostriches 6y ago
  Surely you started with textbooks.
  - SubiculumCode 6y ago
    College textbooks in cognitive sciences are the epitome of "scratching the surface"
castis 6y ago
Google, for all its current problems, _has_ made organization of the worlds knowledge vastly better than it was before they showed up. I could be wrong but just surrounding the field of software development, there's probably already more relevant information on the internet than I could possibly consume in my life.
rfdearborn 6y ago
Approximately once per week I observe a comment on HN saying approximately "search doesn't work anymore." Why is this? Why do I only observe such complaints here?
- karatestomp 6y ago
  Lots of people here who were and remember being "good at search" and able to make exactly the thing they're looking for shoot to the top 3 spots on the results screen and being able to craft a series of searches such that, if they all failed, one could be pretty damn sure the information wasn't online or at least was hidden from the Google bot, neither of which are really possible anymore. The 100th time one attempts to find something on Google that one knows is there but just cannot get to show up no matter how many rare keywords one uses, one concludes Google has lost some pretty significant utility it used to have and starts to wonder what one is not finding when one doesn't know exactly the page one is trying to find. (of course it may be better for lots of other things, but for a fairly large set of "finding things on the Web" tasks it's way worse than it used to be, to the point of being nearly useless—in part I think this is because sometime around 2008-2010 they stopped trying to fight webspammers, choosing instead to embrace some set of them provided they play by Google's rules, and downrank anything that wasn't "well-behaved" webspam or well-known sites hard)
  - bityard 6y ago
    I definitely remember a time when Google was much, much better at returning relevant results for technical content. My feeling is that there are (at least) two factors at play: 1) Google and other search engines revised their algorithms more than a decade ago. Instead of showing you results for what you searched for, they now show you what they _think you meant_ based on your search history and the search histories of millions of other users. This means searching for uncommon topics and phrases get you useless results. And you can't disable this functionality because it's baked into the core of how Google indexes and categories the content that it slurps up from the web. 2) Blogspam authors and e-commerce sites have gotten SEO down to a science, to the point where searching for nearly _anything_ not super specific no longer gets you _information_ about that particular thing, it gets you blogspam articles filled with fluff and affiliate links, or half-broken e-commerce sites trying to sell you something vaguely related to what you searched for. This is not technically Google's fault but there is a lot they could do to curb this, but all that ad revenue on those sites is how they earn that sweet sweet lucre.
joefourier 6y ago
That's an awfully pessimistic view of things. I've studied some fairly obscure corners of computer science and the Internet has allowed me to get instant access to mountains of scientific papers, a huge variety of books (many of which are out of circulation), incredibly high-quality courses on maths, DSP, algorithms and the like. Previously I would have had no way of accessing any of that information unless I happened to be at the right university with the right department, or in some high-level research institute.
- karatestomp 6y ago
  Certain fields are much better off than others. One might expect that CS would be among the best-represented on the Web.
eyko 6y ago
It's not for us to judge what sort of information constitutes knowledge.
VBprogrammer 6y ago
I'm not sure about that. The topics I want to dig into I usually find it's harder to get the level of depth I want on a topic in book form than I do than to find videos on YouTube. Some random examples: - I'm interested in turbine engines, AgentJayZ has a while host of videos taking real turbine engines apart, talking about obscure features like compressor variable valve vanes etc. - Plumbing, almost everything I've ever needed to know about plumbing has been available from PlumberParts / dereton33. - Woodworking, I basically learned everything I know from watching people like Matthias Wandel. - Recently I've been digging to electronics, I have an excellent book "Practical Electronics for Inventors" but every now and again I need to hear a different take on the same things and invariably there is a good video or article which clears it up for me. What's more, to get the level of detail these guys show for free in book form would mean investing in some seriously expensive books, most of which hide the interesting parts under a lot of uninteresting maths.
- karatestomp 6y ago
  Oh yeah, for the subset of content that is "watch an expert do a thing" the Web has become extremely good. Nothing before touches it. In part I think it's remained so because almost all the content is on one site—there's no web search involved in finding it at all, you just search Youtube (which is owned by the main company that might help you find videos on other sites). If Google'd found a way to reward putting academic & deep cultural-knowledge material on a platform they owned such that they had a near-monopoly, perhaps that site would be good. The web at-large might remain basically useless for it (or, probably, even worse, as everyone went to that single platform) such that it depends on one's perspective whether it's the Web delivering value or just Google's platform that could be served just about any damn way at this point (if they decided to transition over to a new non-Web protocol just for Youtube and put it in Chrome they might well succeed in forcing everyone else to adopt it)
- gibolt 6y ago
  What a great point. If I want to know how to do something physical, or learn how a piece of hardware works, YouTube is the place. Since it is Google, how awesome would it be if videos also had related web searches attached to them, instead of just related videos. How-tos could have links to the original manual, the most detailed maintenance manuals, or patents and blogs detailing the physics behind how that thing works. Probably not going to happen and would only be for power users, but it could be so powerful.
akiselev 6y ago
Thirty years ago you couldn't scratch past the surface of more than a few dozen topics based on your local library's availability, if you were lucky to have a decent one to begin with. Between libgen, scihub, Google Books, Project Gutenberg, torrent sites, and the rest of the internet we've got access to almost all of the world's knowledge for free with plenty of avenues to interact with experts using a variety of different channels for knowledge that hasn't been put to paper yet.
- karatestomp 6y ago
  Making piracy easy has been super-helpful. Libgen doesn't have (anywhere near) everything but it's great for surveying a field and picking up some portion of the must-have resources. > access to almost all of the world's knowledge for free I'd guess we've got access to ~10% of it (but it's a good 10%!) for free, ~30% paid (but usually piratable!), and the remainder unavailable as a paid electronic resource but maybe (often not) as a paid paper one and maybe (often not) available pirated—maybe you can at least locate or learn of its existence with the Web, but possibly it's totally unknown to the Web outside of maybe a reference in some books that happen to be digitized (this does actually happen, though you do usually have to get a little obscure before it does, but sometimes not as obscure as one might think). [EDIT] to be clear I think the Web is an excellent research tool, however an awful lot of its value in that role is from piracy (saving me, say, having to inter-library-loan or buy a book just to find the books that book name-drops in its preface, or to read one relevant chapter, or to see the book's index so I can find out whether I need it in the first place). I think its containing anything like "all the world's knowledge", even for liberal values of "all", is far from true, and there's a risk of thinking if something exists it's probably available as a digital resource delivered over the Web (far from true), and if it doesn't then surely it's at least possible to find out about its existence on the Web (also not true), and if neither of those are true it must be something of no value whatsoever to anyone or wildly obscure (not true). [EDIT EDIT] then even if it is on the web, it can be really hard to find something that's not on one of a few major sites using search. More so than it used to be. It can take so damn long that pirating and reading a book on the topic on Libgen can be faster than finding the same info on the open web, even if it's there, which is pretty damning of the state of web indexing. As I wrote in another comment on this thread, it was once possible to be fairly sure when one had reached the edge of Google's knowledge and the thing you're looking for was not in its index—such certainty is now almost impossible, mostly because of how Google search (and seemingly all other web search tools) has changed, not because of increased total content.
dplavery92 6y ago
I've definitely felt and experienced the situation you're describing, but I think you might be painting with too broad a brush. Sources like ArXiv, Google Scholar, and Semantic Scholar have made it really easy for me to access tons of deep, academic/professional level knowledge in my field instantaneously and for free. I know that Physics, Math, and Computer Science are relatively uniquely privileged when it comes to ArXiv, but even paywalled journals are more accessible in the internet age than they were before. I'm even able to find public-released technical info from the DoD through DTIC that I don't think I would ever know what to request without having it come up through search engines. Likewise, tons of undergrad classes through MIT and the Ivies are available for free on Youtube; more still are available with exercises and some form of certification through services like Coursera, EdX, and Udacity. Some graduate and professional-level courses are starting to come online through these institutions as well; the Institute for Advanced Study has started releasing its lectures on Youtube over the past few years. I can find manuals for my car, my appliances, and all my electronics online. I have access to all sorts of do-it-yourself how-tos on Youtube when I want to do work around the house. I have StackOverflow for when I have challenges with programming languages and tools that aren't clear from the docs, and the rest of the StackOverflow network when I have questions that are more specific to my problem domain. There's more throughout all sorts of fields: instant translations with Google translate and langauges lessons through DuoLingo, Rosetta Stone, et al; music lessons on Youtube, etc. Again, I've been through the experience where some older materials just aren't as available as I'd like, and where the normal search tools only bring you part of the way. But I think you're taking a lot for granted here.
marcosdumay 6y ago
"A screen" is not "the web". That book will probably be in a screen too, since it's way too hard to carry it around on paper. Even "correspondence with experts" will be on a screen.

pharke 6y ago

Due to the changing nature of the web I'd even say that the sum total of useful information is actually decreasing year over year. I believe we hit an inflection point a decade or more ago when the number of people with specialized knowledge and skills who were publishing information about their domain online was surpassed by the new "content creators" whose primary goal was to optimize for clicks and ad revenue. This is unevenly applied across different areas of knowledge, the amount of information about computer and media related topics has increased but other areas seem to have realized a sharp decrease. I don't have any hard numbers to support this, only the feeling of it becoming increasingly difficult to find expert information in many areas. More and more I have to resort to public domain books as the only substantive source. There's just a general sense of something deeply wrong with the current state of the web.

nbardy 6y ago
Even if the content creators overwhelm the rest of the crowd now, you can still find experts and their content continues to pile up. For example, iq of internet graphics is still adding incredible content to his site 10 years later. https://iquilezles.org/ https://iquilezles.org/. He's not the only example of this. There may be a higher noise to quality ratio, but the quality content is still growing, you just have to get better at tuning out the noise.
- asdff 6y ago
  Discoverability is still hard when noise increases by the day, and search engines have become useless to find anything meaningful that isn't playing the SEO game.

SubiculumCode 6y ago

This might be field dependent. All my research is based on scientific journal articles and reviews, which are electronically available, and books are more vanity affairs

youareostriches 6y ago
Surely you started with textbooks.
- SubiculumCode 6y ago
  College textbooks in cognitive sciences are the epitome of "scratching the surface"

castis 6y ago

Google, for all its current problems, _has_ made organization of the worlds knowledge vastly better than it was before they showed up. I could be wrong but just surrounding the field of software development, there's probably already more relevant information on the internet than I could possibly consume in my life.

rfdearborn 6y ago

Approximately once per week I observe a comment on HN saying approximately "search doesn't work anymore." Why is this? Why do I only observe such complaints here?

karatestomp 6y ago
Lots of people here who were and remember being "good at search" and able to make exactly the thing they're looking for shoot to the top 3 spots on the results screen and being able to craft a series of searches such that, if they all failed, one could be pretty damn sure the information wasn't online or at least was hidden from the Google bot, neither of which are really possible anymore. The 100th time one attempts to find something on Google that one knows is there but just cannot get to show up no matter how many rare keywords one uses, one concludes Google has lost some pretty significant utility it used to have and starts to wonder what one is not finding when one doesn't know exactly the page one is trying to find. (of course it may be better for lots of other things, but for a fairly large set of "finding things on the Web" tasks it's way worse than it used to be, to the point of being nearly useless—in part I think this is because sometime around 2008-2010 they stopped trying to fight webspammers, choosing instead to embrace some set of them provided they play by Google's rules, and downrank anything that wasn't "well-behaved" webspam or well-known sites hard)
- bityard 6y ago
  I definitely remember a time when Google was much, much better at returning relevant results for technical content. My feeling is that there are (at least) two factors at play: 1) Google and other search engines revised their algorithms more than a decade ago. Instead of showing you results for what you searched for, they now show you what they _think you meant_ based on your search history and the search histories of millions of other users. This means searching for uncommon topics and phrases get you useless results. And you can't disable this functionality because it's baked into the core of how Google indexes and categories the content that it slurps up from the web. 2) Blogspam authors and e-commerce sites have gotten SEO down to a science, to the point where searching for nearly _anything_ not super specific no longer gets you _information_ about that particular thing, it gets you blogspam articles filled with fluff and affiliate links, or half-broken e-commerce sites trying to sell you something vaguely related to what you searched for. This is not technically Google's fault but there is a lot they could do to curb this, but all that ad revenue on those sites is how they earn that sweet sweet lucre.

joefourier 6y ago

That's an awfully pessimistic view of things. I've studied some fairly obscure corners of computer science and the Internet has allowed me to get instant access to mountains of scientific papers, a huge variety of books (many of which are out of circulation), incredibly high-quality courses on maths, DSP, algorithms and the like. Previously I would have had no way of accessing any of that information unless I happened to be at the right university with the right department, or in some high-level research institute.

karatestomp 6y ago
Certain fields are much better off than others. One might expect that CS would be among the best-represented on the Web.

eyko 6y ago

It's not for us to judge what sort of information constitutes knowledge.

VBprogrammer 6y ago

I'm not sure about that. The topics I want to dig into I usually find it's harder to get the level of depth I want on a topic in book form than I do than to find videos on YouTube. Some random examples: - I'm interested in turbine engines, AgentJayZ has a while host of videos taking real turbine engines apart, talking about obscure features like compressor variable valve vanes etc. - Plumbing, almost everything I've ever needed to know about plumbing has been available from PlumberParts / dereton33. - Woodworking, I basically learned everything I know from watching people like Matthias Wandel. - Recently I've been digging to electronics, I have an excellent book "Practical Electronics for Inventors" but every now and again I need to hear a different take on the same things and invariably there is a good video or article which clears it up for me. What's more, to get the level of detail these guys show for free in book form would mean investing in some seriously expensive books, most of which hide the interesting parts under a lot of uninteresting maths.

karatestomp 6y ago
Oh yeah, for the subset of content that is "watch an expert do a thing" the Web has become extremely good. Nothing before touches it. In part I think it's remained so because almost all the content is on one site—there's no web search involved in finding it at all, you just search Youtube (which is owned by the main company that might help you find videos on other sites). If Google'd found a way to reward putting academic & deep cultural-knowledge material on a platform they owned such that they had a near-monopoly, perhaps that site would be good. The web at-large might remain basically useless for it (or, probably, even worse, as everyone went to that single platform) such that it depends on one's perspective whether it's the Web delivering value or just Google's platform that could be served just about any damn way at this point (if they decided to transition over to a new non-Web protocol just for Youtube and put it in Chrome they might well succeed in forcing everyone else to adopt it)
gibolt 6y ago
What a great point. If I want to know how to do something physical, or learn how a piece of hardware works, YouTube is the place. Since it is Google, how awesome would it be if videos also had related web searches attached to them, instead of just related videos. How-tos could have links to the original manual, the most detailed maintenance manuals, or patents and blogs detailing the physics behind how that thing works. Probably not going to happen and would only be for power users, but it could be so powerful.

akiselev 6y ago

Thirty years ago you couldn't scratch past the surface of more than a few dozen topics based on your local library's availability, if you were lucky to have a decent one to begin with. Between libgen, scihub, Google Books, Project Gutenberg, torrent sites, and the rest of the internet we've got access to almost all of the world's knowledge for free with plenty of avenues to interact with experts using a variety of different channels for knowledge that hasn't been put to paper yet.

karatestomp 6y ago
Making piracy easy has been super-helpful. Libgen doesn't have (anywhere near) everything but it's great for surveying a field and picking up some portion of the must-have resources. > access to almost all of the world's knowledge for free I'd guess we've got access to ~10% of it (but it's a good 10%!) for free, ~30% paid (but usually piratable!), and the remainder unavailable as a paid electronic resource but maybe (often not) as a paid paper one and maybe (often not) available pirated—maybe you can at least locate or learn of its existence with the Web, but possibly it's totally unknown to the Web outside of maybe a reference in some books that happen to be digitized (this does actually happen, though you do usually have to get a little obscure before it does, but sometimes not as obscure as one might think). [EDIT] to be clear I think the Web is an excellent research tool, however an awful lot of its value in that role is from piracy (saving me, say, having to inter-library-loan or buy a book just to find the books that book name-drops in its preface, or to read one relevant chapter, or to see the book's index so I can find out whether I need it in the first place). I think its containing anything like "all the world's knowledge", even for liberal values of "all", is far from true, and there's a risk of thinking if something exists it's probably available as a digital resource delivered over the Web (far from true), and if it doesn't then surely it's at least possible to find out about its existence on the Web (also not true), and if neither of those are true it must be something of no value whatsoever to anyone or wildly obscure (not true). [EDIT EDIT] then even if it is on the web, it can be really hard to find something that's not on one of a few major sites using search. More so than it used to be. It can take so damn long that pirating and reading a book on the topic on Libgen can be faster than finding the same info on the open web, even if it's there, which is pretty damning of the state of web indexing. As I wrote in another comment on this thread, it was once possible to be fairly sure when one had reached the edge of Google's knowledge and the thing you're looking for was not in its index—such certainty is now almost impossible, mostly because of how Google search (and seemingly all other web search tools) has changed, not because of increased total content.

dplavery92 6y ago

I've definitely felt and experienced the situation you're describing, but I think you might be painting with too broad a brush. Sources like ArXiv, Google Scholar, and Semantic Scholar have made it really easy for me to access tons of deep, academic/professional level knowledge in my field instantaneously and for free. I know that Physics, Math, and Computer Science are relatively uniquely privileged when it comes to ArXiv, but even paywalled journals are more accessible in the internet age than they were before. I'm even able to find public-released technical info from the DoD through DTIC that I don't think I would ever know what to request without having it come up through search engines. Likewise, tons of undergrad classes through MIT and the Ivies are available for free on Youtube; more still are available with exercises and some form of certification through services like Coursera, EdX, and Udacity. Some graduate and professional-level courses are starting to come online through these institutions as well; the Institute for Advanced Study has started releasing its lectures on Youtube over the past few years. I can find manuals for my car, my appliances, and all my electronics online. I have access to all sorts of do-it-yourself how-tos on Youtube when I want to do work around the house. I have StackOverflow for when I have challenges with programming languages and tools that aren't clear from the docs, and the rest of the StackOverflow network when I have questions that are more specific to my problem domain. There's more throughout all sorts of fields: instant translations with Google translate and langauges lessons through DuoLingo, Rosetta Stone, et al; music lessons on Youtube, etc. Again, I've been through the experience where some older materials just aren't as available as I'd like, and where the normal search tools only bring you part of the way. But I think you're taking a lot for granted here.

marcosdumay 6y ago

"A screen" is not "the web". That book will probably be in a screen too, since it's way too hard to carry it around on paper. Even "correspondence with experts" will be on a screen.