8 ms·
Against the burden of knowledge
- quonn 2y agoRegarding the number of researchers, of all of my friends who got a PhD only a single one did it because he liked research. For most it was a stepping stone for a totally unrelated career in industry and for some just the easiest option due to lack of an alternative. Given that, is is surprising that progress does not scale proportionally? Maybe, maybe not. For sure my comment is not backed by anything but anecdotes.
- gumby 2y ago> Regarding the number of researchers, of all of my friends who got a PhD only a single one did it because he liked research. This is sad; luckily for me the percentage is higher among my friends. I suspect your experience is closer to the statistical mean (perhaps because there's a good chance your friends are a few decades younger than me and many of my friends). Also: I bailed out because I didn't like research, or rather because I very like the practice of research but only ever worked on super obscure things, and I wanted to make a difference I could see. Also I didn't want to have to deal with the culture of research funding and related bureaucracy. Not that there's anything wrong with those who do like that stuff! It just seemed not worth getting a doctorate; it definitely would have made some things easier but in retrospect I made the right call (for me!). There is a lot of basic research done that does make a difference but where that difference is unknown to the original researcher -- perhaps separated too far in time. I have found value in decades-old PhD theses in the life sciences, mainly physiology.
- CrazyStat 2y ago> There is a lot of basic research done that does make a difference but where that difference is unknown to the original researcher -- perhaps separated too far in time. I have found value in decades-old PhD theses in the life sciences, mainly physiology. Have you ever tried tracking down the author to express your appreciation? It might make their day (or month!).
- Calavar 2y agoThat's actually the system working as intended. There's not nearly enough faculty positions or even non-faculty research staff positions to accommodate all the PhDs who graduate. The presumption is that most people who graduate will go into something other than academia. If you want to see the system failing, look to humanities, where most people who get a PhD are legitimately interested in academics and yet they will, in the best case, be stuck as terminal adjuncts
- advael 2y agoIt's crazy how much institutional decay there is basically everywhere in this era. Like I grew up hearing stories about institutions being crazy effective compared to the ones I actually encountered. Sure, humans have always had folly, but the society we currently occupy is obsessed with often nonsensical metrics, rewards con artists disproportionately for gaming those metrics, and creates significant barriers for anyone trying to navigate the world through other means. There is no faculty in any academic institution untouched by the Reagan-era push to make colleges into degree factories, the administrative bloat caused by incursions of the demands of an increasingly muscular and obtuse finance sector (that also affects most other institutions), or the pervasive instinct to automate decisions based on dubious metrics, and most adults understand this at an intuitive level We are in a period of widespread outright institutional collapse. Institutional decay should be the null hypothesis when considering the causal factors behind any widespread problem
- deleted 2y ago[deleted]
- devwastaken 2y agoThis is a natural outcome when governments create rules that in effect solidify a market practice. It becomes irrelevant over time and fails to change because the rules will not allow for it, and even if they did, committees are not known for being industry leaders. In education the solution is simple and effective - cut all federal student aid. The educational institutions that have strong industry ties and therefore the best outcomes will persist. Those that were warming chairs to hand out fake degrees will go under. Let it fail for the admins, because it's already failing the students.
- rexpop 2y agoEducation has to be broader than industry, or we'll all end up slaves.
- devwastaken 2y agoYou cannot live in a capital society without capital. There is not nor has there ever been an education system disconnected from industry until government subsidy and "support" did it. This is because once there is free money there is no longer a natural market effect and organizational failure is acceptable and funded.
- frogeyedpeas 2y agoSometimes low barriers result in better results. If the rent in a city is too high you are not going to get the MOST interesting restaurants, bars, and clubs. You are going to get only the businesses that will DEFINITELY convince an investor to write a check to dump on such high rents; regardless of whether that is a good idea or not. The PhD cohorts for R1 Universities hasn't really gotten any bigger than 50 years ago. The number of academic jobs hasn't really gotten any bigger either. The only people having success in the system are the types of people that seem low-risk to the system. So of course we should expect a decline in innovative ideas as time rolls on. The only way to reverse this is to literally create more tenured jobs (or perhaps temporary tenure ex: 10 years you're guaranteed employment) and increase the size of PhD student cohorts so that they are large enough that iconoclasts can fit in again.
- advael 2y agoYou make a really good point. Defining risk in terms of financial ROI and trying to assess it algorithmically and default-gating more and more endeavors behind such risk assessment is a surprisingly coherent model for a lot of widespread modern rot
- vinnyvichy 2y agoThanks for being one of the few commenters questioning the microconomic angle itself. In my view, the problem is that no rational economic agent will ever embrace the idea that research should be expensive. Well some very few of us do, we realize that we are competing against Nature and the Universe (not other economic entities), so one of the sensible avenues left to us is to try and recuperate all the losses that have already gone into past research. (maybe Kepler was trying desperately to salvage Hellenic heliocentrism? He also had a thing for platonic solids, after all)
- salomonk_mur 2y agoThere is another way. Reinvent the way academic careers work. No need to even have the concept of tenure. No need to get a PhD degree to be recognized as someone researching something. Anyone should be able to publish research without being on such restrictive terms. The article's main takeaway is that the institution of academia has perverse incentives that hold back innovation. I believe the way jobs and degrees are structured are part of that.
- paulpauper 2y agoBut then, Copernicus came along with the heliocentric model which, in its simplest initial form, made worse prediction than the tuned-up Ptolemaic model. But the burden of knowledge was dissolved in an instant. Improving the Copernican model meant shifting orbital paths from perfect circles to ellipses. It had nothing to do with the epicycles and perihelions of the Ptolemaic model and none of that burdensome knowledge was necessary to expand the frontier anew. disagree here. it does not make it easier. to show why a new model is right or superior means having to understand the old models well enough to show it's wrong or suboptimal. hence knowledge.
- bawolff 2y agoI think people have recognized that the current research grant system tends to reward safe, status quo ideas, and discourage potentially revolutionary risky long shot ideas. Its been talked about for a long time now. Maybe even decades at this point. The hard part is what to do about it.
- FrustratedMonky 2y agoSabine Hossenfelder had video on this exact subject. https://www.youtube.com/watch?v=LKiBlGDfRU8 https://www.youtube.com/watch?v=LKiBlGDfRU8
- immibis 2y agoNote that Sabine Hossenfelder is a rather unreliable source for anything outside her field of expertise.
- salomonk_mur 2y agoHence, more savvy researchers would rather get private funding instead of jumping through grant hoops.
- knowaveragejoe 2y agoThere isn't an infinite amount of grant money to go around, and of course handing it out to any crackpot that comes along is going to be a nonstarter. It seems to me that the solution is somewhere in between the two extremes.
- mncharity 2y agoI've repeatedly encountered research I'd have liked to use, but which was inaccessibly buried by the pipeline from academic research to patents, to optional failed startup, to bigco portfolio entombment. And wished for an industrial policy dialed more towards a progression from research, to open source and community exploration, to cottage commercial, to small-scale industrial. Or consider a slider from non-competes, to California I'm-working-at-a-competitor-tomorrow, to China and-I've-brought-stuff - I'd like to see more California and China than not. Our optimization for pharma and VC, often seems terminally ill fitted for progress along many vectors.
- Ygg2 2y agoMy observation is that both of those could be linked. We might be reaching the limits of technology. It means that to push boundaries we need more money to buy/construct new tests. In other words, you need exponentially more money. Which leads to the kind of blockbusterization of science. You don't get hundred blockbusters each year, instead you get two, by team with track record. Which sounds a lot like institutional decay.
- dartos 2y agoI think you’re putting the cart before the horse. Needing exponentially more money is the decay. The blockbusterization of science is a symptom.
- Ygg2 2y agoExponentially more money isn't just sign of institutional decay. Resources/energy required to discover X ray is a grain of sand on a beach compared to energies required to discover Higg's boson. That's just energy. Compare cost to make an X ray tube vs LHC. Science used to be able to snatch low hanging fruit. Now those are picked, but people still expect same rate of innovation.
- dartos 2y ago> but people still expect same rate of innovation This sounds like the decay, not the rising cost. I’m sure every researcher understands this, but not every executive or shareholder. The lack of trust in your workers / focus on hyper growth is the issue. Like you say, discoveries become more expensive over time, but we’re seeing similar institutional decay in non research institutions as well, so that can’t be the only factor. Like how much more expensive is it to teach a single college student now than it was 20 years ago. In all honesty, probably much cheaper. Especially when you start looking at online classes. So why is college in the states so much more expensive? It’s largely because of administrative bloat, which, among many other weird incentives, are symptoms of decay. And why would you need administrators? IMO, it’s because you’re unwilling to trust small departments or individuals to tech effectively.
- electrodank 2y ago[flagged]
- Archelaos 2y agoI think the example from the history of astronomy the author provides can by turned against him. It was the accumulation of astronomical observations and the refinement of ever and ever exacter measurements that provided the basis for the transition in ther scientific community from the heliocentric to the geocentric system, that took between one and two hundred years, depending when on lets it start (Copernicus or a predecessor) and end (Galileo, Kepler or Newton). If we look for example at the career of Kepler, we see that he had to accumulate expert knowledge of both systems and that he arrived at his later so-called laws not by a sudden insight, but through tireless work, trying again and again over several years to make sense out of series of measurements that did not really make sense in either the Ptolemaic or Copernican system. It was not the epicycles that worried the late medieval astronomers, but the fact that according to their geocentric theory the earth was not really exactly in the center of the deferent. Correspondingly, however, the sun was not really at the center of the Copernican system either. Both systems lacked elegance, and both systems were more or less equaly in accordance with the observational data. To solve this debate and to make progress, the strategy of the most famous astronomer of his time, Tycho Brahe, was to collect better measurements. And it were these accumulated measurements that enabled his pupil Kepler to develop a solution. In a sort, he was lucky. Only the data for the planet Mars, which he considered first, was exact enough to match an ellipsis. His data for Jupiter or Saturn would not have allowed him to come up with one. Accumulated knowledge was not an obstacle, but the basis for Keplers insights. That we might get a different impression is a result of simplifications that happen after a scientific community reaches a conclusion. At that stage such a theory becomes textbook knowledge: a student needs no longer acquire the accumulate knowledge of the previous period, but only the successful doctrine. These doctrines appear and are all too often presented as ingenious insights from geniuses, instead of cumulation points of a collective work of generations of scientists and scholars.
- passion__desire 2y agoParallax Nick is a thoroughly researched channel on these topics. https://youtube.com/@parallaxnick637 https://youtube.com/@parallaxnick637
- ahazred8ta 2y ago
- vinnyvichy 2y agoAnother article of not really independent interest: "Investing in the Unknown and Unknowable" https://www.hks.harvard.edu/publications/investing-unknown-and-unknowable https://www.hks.harvard.edu/publications/investing-unknown-a... PS: Since these are articles penned by academic economists rather than pro scientists, it'd be nice to hear criticism from an economic (or finance) angle..
- biophysboy 2y agoKnowledge is not always cumulative, but you have to be aware of its prevent contradictions to refute them. This is a point in Kuhn’s “Structure of scientific revolutions” that I think is neglected
- tsimionescu 2y agoThe comparison the author makes with astronomy doesn't support their point at all. First, it agrees with the notion of a burden of knowledge, by agreeing that improving the Ptolemaic model took huge amounts of time to learn its intricacies. Secondly, the Copernican alternative worked not because it was simpler, but because the Ptolemaic model was wrong. There is never a guarantee that a model we currently is is wrong and can be replaced by a substantially simpler model. Maybe someone brilliant will come along and find a simpler model than quantum electrodynamics, but maybe they won't: it's absolutely possible that, say, QED is the right model, and we can at best make it even more complex to explain more details of phenomena. Finally, tools only get you so far. Tools don't necessarily encode some of the scientific knowledge you need to advance fields. They are a separate track from it.
- animal_spirits 2y agoWe compile knowledge into institutions to reduce the burden of knowledge, distilling vast information into simple phrases. However, the rapid technological and cultural changes over the past 80 years have hindered the formation of new institutions. As technological improvements slow down, we will likely see more established institutions emerge, reducing the burden of knowledge and allowing us to break through the current knowledge ceiling.
- benreesman 2y agoNo, ideas are not getting hard to find. Ideas are getting harder to discuss. There has never been such a ripe time to find such ample reason to propose and discuss new and radical ideas. And there has never been such a hostile time to looking into it. In physics we’ve lost 50-70 years to string theory and related nonsense and the people still working on effective quantum field theories have had to re-brand as working on “quantum information / quantum computing” to stay in rent and lab desks. There aren’t many serious QFT people who doubt Everettian epistemology but it’s still nothing undergraduates hear about. Most serious non-string gravity work comes out of Perimeter and Marletto/Deutsch. In economics and finance we still put our hand into the sink shredder of the Chicago-style strong-form EMH, as resoundingly and empirically disproved by Simons (may he rest). And we continue to embrace Friedman-style supply side economic policy, without peer the nastiest wealth transfer from those with surplus away from those facing scarcity at the barrel of a gun in absolute terms ever. It never trickles down. In the fundamental epistemology of our day we are surrendering the reigns of influence to people who market chat bots trained to lie convincingly at scale. Knowledge, and the new ideas that approximate at the limit the derivative of knowledge are under attack. They’re under attack by some rich guy’s choice for the job he did well and decided to keep in the family.
- gampleman 2y agoIf we had a rather good way of attracting, selecting and retaining top scientists then by power laws inherent in any creative/competitive endeavour we’d expect that by increasing funding drastically to get more people employed we’d only see very marginal improvements, as the wider base of the talent won’t contribute all that much.
- tuatoru 2y agoYes. Heretical opinion: we should have much stronger merit-based gating on access to university. You can get in if you can demonstrate facility with real analysis, matrix algebra, and Bayesian reasoning. Imagine how much the general level of educational noise would be reduced in universities, and how much more space there would be to do meaningful research.
- contingencies 2y agoThe cost of complexity is inherently wicked. We could discuss its impacts on HR, on R&D, on supply chain, on education, on manufacturing or project management. However, let's stick to the core: the vast majority of investment-related decision making systems are still stuck in a finite-dimensional reality barely evolved from a spartan economic rationalism analytical milieu derivative of double entry book-keeping functionally based upon entrenched inefficiency and a lack of timeliness. Regulators, insurance, and other aspects of society with ill-deserved critical veto capability do not just support or require but effectively celebrate this uncompromising backwardness, whereas it should actually be banished. We desperately need holistic decision-making taking in to account factors lying outside of this purview. Academics, entrepreneurs, frustrated youth, competent industrial craftspeople and environmentalists are unlikely allies at the coal face.
- lumb63 2y agoPut another way, I think one of the author’s arguments is “what if we’re pouring more and more effort into research in an area, because our primary abstraction is wrong?” That would make it harder and harder to continue to roll the ball up an increasingly steep hill. And I think in some fields, they may be right. It’s akin to how in math some problems are very difficult in Cartesian coordinates but trivial in polar coordinates. For instance, it seems to me that the existence of the “wave-particle duality” in physics and our inability to explain quantum-scale phenomena in a way that makes intuitive or logical sense to people is the canary in the coal mine that we took a wrong turn a while back. That’s not to say it’s true in every field, but that’s one that has stood out to me for a while. I think another cause could be bureaucratic. From what I understand, several hundred years ago, research was funded by rich patrons who wanted some credit in relation to discoveries. Nowadays there is a giant, lurking, centrally planned grant machine that distributes money to researchers. And as we know from economics, central planning becomes increasingly untenable as the system becomes larger. Results get worse as the complexity skyrockets. Additionally, if we are going to blame the “burden of knowledge” in any capacity, we have to acknowledge the abysmal education system (in the US). An anecdote which I will never forget is, in 6th grade we had a student who had recently immigrated from India. By his own telling, he was an average student there. But compared to native students, he was light years ahead in math. We learned multiplication in third grade, division in fourth grade, long division and fractions in fifth grade, or something like that. He had learned multiplication in first grade, and all of division in second grade. While our smartest students in sixth grade were grappling with pre algebra, he was bored in classes with the eighth grade algebra students. Our education system has, since experiencing that, seemed to me deeply flawed; there really is no reason an efficient and effective public education system should take 12+ years to be able to have a child ready for college and then another 4+ in college to have them ready to contribute to an isolated field.
- aeim 2y ago> "what if we’re pouring more and more effort into research in an area, because our primary abstraction is wrong? Absolutely; perhaps "thinking of knowledge in terms of physical environments" also helps to illustrate why – sometimes the only way to get to where we want to be, is to redefine where-and-how we begin Concretely, looking up a unconquered mountain: it may seem as though many approaches are viable, but when attempt proves otherwise, the only option might be to return to base, and consider a new origin for approach I think this also helps illustrate why "progress by accrual alone" is so inherently flawed. Sometimes you must begin again, in whole, but also in part – because "you can't get to anywhere from anywhere" In fact, for almost all domains other than science, beginning again is such an optimal remedy to progress dead-ends, it's operational Writers know what's up – kill your darlings. Software engineers too, famously – measuring software progress by lines of code added is counter productive – we learn to be unsentimental, delete and reconsolidate often. This is the continual redefinition of the "origin" of where we continue-on-from Though it does require a change in perspective: written words, and lines of code – not as asset, but liability. Redundant words, code or literature, is contrived complexity. Unnecessary, and after a while, prohibitive to progress. I think science hasn't done this yet? – we know that naming things is hard, but perhaps for science, calling every artifact 'literature' doesn't help... I also think that science catastrophically misunderstands "beginning again". Finding new origins is easiest for those who haven't been conditioned to only see one. It doesn't take magical genius, just (selective) ignorance. This is why (non-trivial) revolutionary paradigms often come from outsiders (or historical scientists who did not complete the endurance test of the modern day burden) I can't help but think that scientific genius is based on "eh, fuck that, i'm sure there's a better way" It's as if, when Kuhn wrote "Structute", a million normies cried out in pain, and because they'd already been conditioned to one origin, and couldn't stand the idea of "newbs" getting credit, vowed to prove that "progress by iterative accrual alone" was all that was necessary. And here we are
- thntk 2y agoBesides the burden of knowledge and the flaws of academic funding, another factor to explain the slowdown in scientific progress is: the world has more things to be entertained with, including books, comics, games, movies, music, p@rn, social media. These things have dramatically increased over the past decades, meanwhile the number of hours in a day did not increase at all.
- highfrequency 2y agoGreat example: “ Ptolemy and his astronomical ancestors explained these “retrograde” motions with the extra loops you see in the map above called “epicycles.” By the 15th century astronomers had accumulated centuries of meticulous measurements and incorporated them into complex orbital paths, matching their observations. Learning these models and taking enough measurements to improve one of them took an entire lifetime of monastic devotion to studying the stars. The burden of knowledge was immense. But then, Copernicus came along with the heliocentric model which, in its simplest initial form, made worse prediction than the tuned-up Ptolemaic model. But the burden of knowledge was dissolved in an instant. Improving the Copernican model meant shifting orbital paths from perfect circles to ellipses. It had nothing to do with the epicycles and perihelions of the Ptolemaic model and none of that burdensome knowledge was necessary to expand the frontier anew.”
- its_ethan 2y agoYea a cool example, and sure there are others around this time where "we" figured out previous thinking was just fully wrong. But the author then also says: > There are thousands of other examples. Really? Thousands? Are they all similarly examples of pre-scientific method/ pre-industrial revolution knowledge? How many of these "thousand" examples have happened since 1850? Why didn't he list any that have happened in the last 50 years?
- sherazakram 2y ago[dead]
- madacol 2y agoAuthor's main idea is that knowledge is not always cumulative, sometimes new ideas disrupts and replaces old ones, and in some cases, even reduce the burden because it is also simpler. The problem is that most new ideas are, in fact, cumulative, and hence increases the burden at a faster rate than disruptive ideas reduce it The idea of "Burden of knowledge" only requires that most knowledge accumulates. The fact that there are exceptions that do not accumulate does not disprove it