15 ms·

The threat is comfortable drift toward not understanding what you're doing

• garn810 6mo ago
  Academia always been full of narcissists chasing status with flashy papers and halfbaked brilliant ideas (70%? maybe) LLMs just made the whole game trivial and now literally anyone can slap together something that sounds deep without ever doing the actual grind. LLMs just speeding up the process, just a matter of time how quickly this shit is exposing what the entire system has been all along
• stavros 6mo ago
  I see this fallacy being committed a lot these days. "Because LLMs, you will no longer need a skill you don't need any more, but which you used to need, and handwaves that's bad". Academia doesn't want to produce astrophysics (or any field) scientists just so the people who became scientists can feel warm and fuzzy inside when looking at the stars, it wants to produce scientists who can produce useful results. Bob produced a useful result with the help of an agent, and learned how to do that, so Bob had, for all intents and purposes, the exact same output as Alice. Well, unless you're saying that astrophysics as a field literally does not matter at all, no matter what results it produces, in which case, why are we bothering with it at all?
  • hirako2000 6mo ago
    I was reading in the article that what matters is the process that leads to the (typically useless) result, what the people get out of it. Once I realized that this white on black contrast was hurting my eyes, I decided to stop as I didn't want to see stripes for too long when looking away. Some activity has outcomes that aren't strictly in the results.
    • stavros 6mo ago
      Yeah, it was saying that what matters is the process of training people to be good scientists, so they can produce other, more useful, results. That's literally what training is, everywhere. This argument boils down to "don't use tools because you'll forget how to do things the hard way", which nobody would buy for any other tool, but with LLMs we seem to have forgotten that line of reasoning entirely.
      • hirako2000 6mo ago
        There is an argument to make that tools that speed up a process whilst keeping acuity intact are legitimate. LLMs, the way they typically get used, are solely to save time by handing over nearly the entire process. In that sense acuity can't remain intact, even less so improving over time.
        • stavros 6mo ago
          So?
          • hirako2000 6mo ago
            You previous comment reads as if LLMs get some unjustified different treatment. Do you agree the different treatment is justified ? (Many do not). Or are you asking , so what if acuity is diminished so long as an LLM does the job equally well?
      • defrost 6mo ago
        > This argument boils down to "don't use tools because you'll forget how to do things the hard way", which nobody would buy for any other tool, This is false. There absolutely are people that fall back on older tools when fancy tools fail. You will find such people in the military, in emergency services, in agriculture, generally in areas where getting the job done matters. Perhaps you're unfamiliar. They other week I finished putting holes in fence posts with a bit and brace as there was no fuel for the generator to run corded electric drills and the rechargable batteries were dead. Ukrainians, and others, need to fall back on no GPS available strategies and have done so for a few years now. etc.
        • thijson 6mo ago
          In the 80's the Americans thought that the Russians were backwards to be still using vacuum tubes in their military vehicles. Later they found out that they were being used because they are more tolerant to EMP from a nuclear blast.
        • 6mo ago
  • pards 6mo ago
    > Take away the agent, and Bob is still a first-year student who hasn't started yet. The year happened around him but not inside him. He shipped a product, but he didn't learn a trade. We're minting an entire generation of people completely dependent on VC funding. What happens if/when the AI companies fail to find a path to profitability and the VC funding dries up?
    • stavros 6mo ago
      Do you think that'll take a generation to happen?
      • rafterydj 6mo ago
        ChatGPT 3.5 came out coming on 4 years ago now. I don't think a human generation (~20-30 years) needs to be the benchmark here, but new juniors in the industry for a handful of years can be said to be a whole "generation". That how I was reading OP.
    • Paradigma11 6mo ago
      What will happen is pretty obvious. Those companies will either be classified as too important to fail and get government support or go bankrupt and will be bought for pennies on the dollar. For the customers nothing much will change since tokens are getting cheaper every year and the business is already pretty profitable. Progress will slow down massively till local open weight models catch up to pre-crash SotA and go on from there.
      • pards 6mo ago
        > the business is already pretty profitable As of March 2026, OpenAI generates annual revenue exceeding $12 billion. However, the costs of running ChatGPT are around $17 billion a year. Source: https://searchlab.nl/en/statistics/chatgpt-statistics-2026 https://searchlab.nl/en/statistics/chatgpt-statistics-2026
        • ipaddr 6mo ago
  • djaro 6mo ago
    The problem is that LLMs stop working after a certain point of complexity or specificity, which is very obvious once you try to use it in a field you have deep understanding of. At this point, your own skills should be able to carry you forward, but if you've been using an LLM to do things for you since the start, you won't have the necessary skills. Once they have to solve a novel problem that was not already solved for all intentes and purposes, Alice will be able to apply her skillset to that, whereas Bob will just run into a wall when the LLM starts producing garbage. It seems to me that "high-skill human" > "LLM" > "low-skill human", the trap is that people with low levels of skills will see a fast improvement of their output, at the hidden cost of that slow build-up of skills that has a way higher ceiling.
    • stavros 6mo ago
      Then test Bob on what you actually want him to produce, ie novel problems, instead of trivial things that won't tell you how good he is. Why is it a problem of the LLM if your test is unrelated to the performance you want?
      • troupo 6mo ago
        How can Bob produce novel things when he lacks the skills to do even trivial things? I didn't get to be a senior engineer by immediately being able to solve novel problems. I can now solve novel problems because I spent untold hours solving trivial ones.
        • stavros 6mo ago
          Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.
          • troupo 6mo ago
            Ah yes. The famous theoretical mathematicians who immediately started on novel problems in theoretical mathematics without first learning and understanding a huge number of trivial things like how division works to begin with, what fractions are, what equations are and how they are solved etc. Edit: let's look at a paper like Some Linear Transformations on Symmetric Functions Arising From a Formula of Thiel and Williams https://ecajournal.haifa.ac.il/Volume2023/ECA2023_S2A24.pdf https://ecajournal.haifa.ac.il/Volume2023/ECA2023_S2A24.pdf and try and guess how many of trivial things were completely unneeded to write a paper like this.
  • nandomrumber 6mo ago
    > why are we bothering with it at all? Because we largely want people who have committed to tens of thousands of dollars of debt to feel sufficiently warm and fuzzy enough to promote the experience so that the business model doesn’t collapse. It’s difficult to think anyone would end up truly regretting doing a course in astrophysics, or any of the liberal arts and sciences if they have a modicum of passion, but it’s very believable that a majority of them won’t go on to have a career in it, whatever it is, directly. They’re probably more likely to gain employment on their data science skills, or whether core competencies they honed, or just the fact that they’ve proven they can learn highly abstract concepts, or whatever their field generalises to. Most of the jobs are in not-highly-specific academic-outcome.
    • imtringued 6mo ago
      Even if you land a job in your field, you will encounter that academia is backwards vs industry in some aspects and decades ahead of what is adopted in the industry in other aspects to the point where both of these mean that you won't make much use of the skills you learned in university.
  • nathan_compton 6mo ago
    Is that what "academia" wants? Last I checked "academia" is not a dude I can call and ask for an opinion or definition of what it was interested in. I will make an explicit, plausible, counterpoint: academia wants to produce understanding. This is, more or less, by definition, not possible with an AI directly (obviously AIs can be useful in the process). Take GR as an example. The vast majority of the dynamical character of the theory is inaccessible to human beings. We study it because we wanted to understand it, and only secondarily because we had a concrete "result" we were trying to "achieve." A person who cares only about results and not about understanding is barely a person, in my opinion.
  • mzhaase 6mo ago
    Why should we only do things that produce some sort of value? Do we really want to reduce all of human existence to increasing profits?
    • stavros 6mo ago
      You said "value" and "profit". I said "useful".
    • nemo44x 6mo ago
      What’s a better method for determining how to utilize and distribute resources? To determine where energy should be used and where it should be moved from?
      • sgarland 6mo ago
        Some things are just enjoyable. I get no real utility from photography - it’s not my career, it’s not a side gig, and I’m not giving prints out as gifts. Most of the shots never get printed at all. I do it because I enjoy the act itself, of knowing how to make an image frozen in time look a particular way by tweaking parameters on the camera, and then seeing the result. I furthermore enjoy the fact that I could achieve the same result on a dumb film camera, because I spent time learning the fundamentals.
  • sega_sai 6mo ago
    You missed the argument. When we are talking about faculty, yes their result is the only thing that matters, so if it was produced quicker with a LLM, that's great. But when we are talking about the student, there is a drastic difference in the student in the with LLM vs without LLM cases. In the latter they have much better understanding. And that matters in the system when we are educating future physicists.
  • selimthegrim 6mo ago
    Completely missed the point of the blog post which was that the point was producing the scientist not the result
  • gedy 6mo ago
    We aren't talking pocket calculators here (I see the irony of phone app in pocket), LLMs are hugely expensive things made and controlled behind costly commercial subscriptions. And likely in the middle of a huge investment bubble and stability is uncertain. So we all need to be careful about "gee we don't need that skill or person anymore", etc.
    • danielbln 6mo ago
      Open weight models that run under your desk are not frontier model level, but they are getting closer. Improvements in agentic post training and things like TurboQuant mean that even if all frontier labs pull the plug tomorrow, we will still have agents to work with.
      • gedy 6mo ago
        I'm definitely looking forward to that, as I really want people to control their own tools.
      • zozbot234 6mo ago
        TurboQuant is not a step change, it's more of a smaller incremental improvement to KV quantization, and possibly (unsure) to quantization more generally. I'm actually more positive about SSD weights offload, which opens up very large local models for slow inference (good enough for slow chat) to virtually any hardware or amount of RAM.
  • asHg19237 6mo ago
    The arguments of the LLM psychosis afflicted get more and more desperate. Astrophysics is about understanding and thinking, this comment paints it as result oriented (whatever that means). The industrialization of academia hasn't even produced more results, it has produced more meaningless papers. Just like LLMs produce the 10.000th note taking app, which for the LLM psychosis afflicted is apparently enough.
    • deleted 6mo ago
      [deleted]
    • nothinkjustai 6mo ago
      [flagged]
  • dsqrt 6mo ago
    The goal of academic research is to create understanding, not papers. If we outsource all research to LLMs, then we are only producing the latter.
  • dwa3592 6mo ago
    Hard sciences play this crucial and often unseen role in our society : they help train humans to develop critical thinking. Not everyone with PhD in Astrophysics ends up doing Astrophysics in life; it's a discipline, or a training regime for our minds. After that PhD; the result is a human being who can tackle hard problems. We have many other such disciplines (basically any PhD in hard sciences) which produces this outcome.
  • cmiles74 6mo ago
    Until the LLM is wrong and Bob passes the erroneous result off as accurate, reliable and vetted by a knowledgeable person. At that point Bob is not producing a useful result. Then it becomes a trap other people might get caught in, wasting valuable time and energy.
  • deleted 6mo ago
    [deleted]
• sd9 6mo ago
  The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job has turned into a different sort of thing than what I signed up for. So I think this article is partly right, Bob is not learning those skills which we used to require. But I think the market is going to stop valuing those skills, so it’s not really a _problem_, except for Bob’s own intellectual loss. I don’t like it, but I’m trying to face up to it.
  • djaro 6mo ago
    > So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade meals because "those aren't going anywhere". Over the span of years, eventually a real cook will be able to make way better meals than anything you can buy at a grocery store. The market will always value the exact things LLMs can not do, because if an LLM can do something, there is no reason to hire a person for that.
    • jacquesm 6mo ago
      Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.
      • omega3 6mo ago
        That’s a good analogy but I think we’ve already went from 0 to 10 rungs over the last couple of years. If we assume that the models or harnesses will improve more and more rungs will be removed. Vast majority of programmers aren’t doing novel, groundbreaking work.
• sam_lowry_ 6mo ago
  See also The Profession by Isaac Asimov [0] and his small story The Feeling of Power [1]. Both are social dramas about societies that went far down the path of ignorance. [0] http://employees.oneonta.edu/blechmjb/JBpages/m360/Profession%20I%20Asimov.pdf http://employees.oneonta.edu/blechmjb/JBpages/m360/Professio... [1] https://s3.us-west-1.wasabisys.com/luminist/EB/A/Asimov%20-%20Nine%20Tomorrows.pdf https://s3.us-west-1.wasabisys.com/luminist/EB/A/Asimov%20-%...
• djoldman 6mo ago
  These themes have been going around and around for a while. One thing I've seen asserted: > What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days... The equations seemed right... Then Schwartz read it, and it was wrong... It faked results. It invented coefficients... The argument that AI output isn't good enough is somewhat in opposition to the idea that we need to worry about folks losing or never gaining skills/knowledge. There are ways around this: "It's only evident to experts and there won't be experts if students don't learn" But at the end of the day, in the long run, the ideas and results that last are the ones that work. By work, I mean ones that strictly improve outcomes (all outputs are the same with at least one better). This is because, with respect to technological progress, humans are pretty well modeled as just a slightly better than random search for optimal decisioning where we tend to not go backwards permanently. All that to say that, at times, AI is one of the many things that we've come up with that is wrong. At times, it's right. If it helps on aggregate, we'll probably adopt it permanently, until we find something strictly better.
  • jacquesm 6mo ago
    AI is extremely good at producing well formatted bullshit. You need to be constantly on guard against stuff that sounds and looks right but ultimately is just noise. You can also waste a ton of time on this. Especially OpenAI's offering shows poorly in this respect: it will keep circling back to its own comfort zone to show off some piece of code or some concept that it knows a lot about whilst avoiding the actual question. It's really good at jumping to the wrong conclusions (and making it sound like some kind of profound insight). But the few times that it is on the money make up for all of that noise. Even so, I could do without the wasted time and endless back and forths correcting the same stuff over and over again, it is extremely tedious.
    • djoldman 6mo ago
      Have you tried turning up temperature in those cases where it circles back? I have been meaning to.
• oncallthrow 6mo ago
  I think this article is largely, or at least directionally, correct. I'd draw a comparison to high-level languages and language frameworks. Yes, 99% of the time, if I'm building a web frontend, I can live in React world and not think about anything that is going on under the hood. But, there is 1% of the time where something goes wrong, and I need to understand what is happening underneath the abstraction. Similarly, I now produce 99% of my code using an agent. However, I still feel the need to thoroughly understand the code, in order to be able to catch the 1% of cases where it introduces a bug or does something suboptimally. It's possible that in future, LLMs will get _so_ good that I don't feel the need to do this, in the same way that I don't think about the transistors my code is ultimately running on. When doing straightforward coding tasks, I think they're already there, but I think they aren't quite at that point when it comes to large distributed systems.
  • spicyusername 6mo ago
    So we already have this problem and things are "fine"?
  • kgwxd 6mo ago
    > LLMs will get _so_ good that I don't feel the need to do this, in the same way that I don't think about the transistors my code is ultimately running on. The problem is, they're nothing like transistors, and never will be. Those are simple. Work or don't, consistently, in an obvious, or easily testable, way. LLM are more akin to biological things. Complex. Not well understood. Unpredictable behavior. To be safely useful, they need something like a lion tamer, except every individual LLM is its own unique species. I like working on computers because it minimizes the amount of biological-like things I have to work with.
    • oncallthrow 6mo ago
      I suppose transistors is a bad example. Perhaps a better analogy would be the Linux kernel. It's built by biological humans, and fallible ones at that. And yet, I don't feel the need to learn the intricacies of kernel internals, because it's reliable enough that it's essentially never the kernel's fault when my code doesn't work.
• ghc 6mo ago
  As straw men go, this is an attractive one, but... When I was fresh out of undergrad, joining a new lab, I followed a similar arc. I made mistakes, I took the wrong lessons from grad student code that came before mine, I used the wrong plotting libraries, I hijacked python's module import logic to embed a new language in its bytecode. These were all avoidable mistakes and I didn't learn anything except that I should have asked for help. Others in my lab, who were less self-reliant, asked for and got help avoiding the kinds of mistakes I confidently made. With 15 more years of experience, I can see in hindsight that I should have asked for help more frequently because I spent more time learning what not to do than learning the right things. If I had Claude Code, would I have made the same mistakes? Absolutely not! Would I have asked it to summarize research papers for me and to essentially think for me? Absolutely not! My mother, an English professor, levies similar accusations about the students of today, and how they let models think for them. It's genuinely concerning, of course, but I can't help but think that this phenomenon occurs because learning institutions have not adjusted to the new technology. If the goal is to produce scientists, PIs are going to need to stop complaining and figure out how to produce scientists who learn the skills that I did even when LLMs are available. Frankly I don't see how LLMs are different from asking other lab members for help, except that LLMs have infinite patience and don't have their own research that needs doing.
  • jacquesm 6mo ago
    AI does not give you knowledge. It magnifies both intelligence and stupidity with zero bias towards either. If you were above average intelligent then you may be able to do a little bit more than before assuming you were trained before AI came along. And if you were not so smart then you will be able to make larger messes. The problem, and I think the article indirectly points at that, is that the next generation to come along won't learn to think for themselves first. So they will on average end up on the 'B' track rather than that they will be able to develop their intelligence. I see this happening with the kids my kids hang out with. They don't want to understand anything because the AI can do that for them, or so they believe. They don't see that if you don't learn to think about smaller problems that the larger ones will be completely out of reach.
    • 6mo ago
• jeremie_strand 6mo ago
  [dead]
• efields 6mo ago
  I literally don't know how compilers work. I've written code for apps that are still in production 10 years later.
  • bakugo 6mo ago
    Have you written a compiler, though?
  • Herbstluft 6mo ago
    Are you working on compilers? If not it seems you did not understand what is being talked about here. Do you lack fundamental understand of those apps you built that are still in use? Did you lack understanding of their workings when you built them?
  • layer8 6mo ago
    You don’t need to understand compilers because the code it compiles, when valid according to the language specification, is supposed to work as written, and virtually always does. There is no language specification and no “as written” with LLMs.
  • wglb 6mo ago
    No problem with that. However, at one point in my career, I was frustrated with limitations in a language (Fortran II) and my curiosity got the better of me and I studied compilers thoroughly. This led to a new job and the understanding of many new useful programming concepts. Very rewarding. But if you are curious, studying compilers, maybe even writing a new one, will give you tools to do other things. While working with LLMs, much of my experience gives me new ideas to push the LLM to explore.
• huflungdung 6mo ago
  [dead]
• tom-blk 6mo ago
  Strongly agree,we see this almost everywhere now
• simianwords 6mo ago
  > Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe: "What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking; there's the real danger." Herbert was writing science fiction. I'm writing about my office. The distance between those two things has gotten uncomfortably small. The author is a bit naive here: 1. Society only progresses when people are specialised and can delegate their thinking 2. Specialisation has been happening for millenia. Agriculture allowed people to become specialised due to abundance of food 3. We accept delegation of thinking in every part of life. A manager delegates thinking to their subordinates. I delegate some thinking to my accountant 4. People will eventually get the hang of using AI to do the optimum amount of delegation such that they still retain what is necessary and delegate what is not necessary. People who don't do this optimally will get outcompeted The author just focuses on some local problems like skill atrophy but does not see the larger picture and how specific pattern has been repeating a lot in humanity's history.
  • zajio1am 6mo ago
    A related quote from A. N. Whitehead: > It is a profoundly erroneous truism ... that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them.
  • skydhash 6mo ago
    Current civilization is very complex. And it’s also fragile in some parts. When you build systems around instant communication and the availability of stuff built in the other side of the world on a fixed schedule, it’s very easy to disrupt. > 4. People will eventually get the hang of using AI to do the optimum amount of delegation such that they still retain what is necessary and delegate what is not necessary. People who don't do this optimally will get outcompeted Then they’ll be at the mercy of the online service availability and the company themselves. Also there’s the non deterministic result. I can delegate my understanding of some problems to a library, a software, a framework, because their operation are deterministic. Not so with LLMs.
• inatreecrown2 6mo ago
  Using AI to solve a task does not give you experience in solving the task, it gives you experience in using AI.
• patcon 6mo ago
  The exciting and interesting to me is that we'll probably need to engage "chaos engineering" principles, and encode intentional fallibility into these agents to keep us (and them) as good collaborators, and specifically on our toes, to help all minds stay alert and plastic If that comes to pass, we'll be rediscovering the same principles that biological evolution stumbled upon: the benefits of the imperfect "branch" or "successive limited comparison" approach of agentic behaviour, which perhaps favours heuristics (that clearly sometimes fail), interaction between imperfect collaborators with non-overlapping biases, etc etc https://contraptions.venkateshrao.com/p/massed-muddler-intelligence https://contraptions.venkateshrao.com/p/massed-muddler-intel... > Lindblom’s paper identifies two patterns of agentic behavior, “root” (or rational-comprehensive) and “branch” (or successive limited comparisons), and argues that in complicated messy circumstances requiring coordinated action at scale, the way actually effective humans operate is the branch method, which looks like “muddling through” but gradually gets there, where the root method fails entirely.
  • aaztehcy 6mo ago
    [flagged]
• throwaway132448 6mo ago
  The flip side I don’t see mentioned very often is that having a product where you know how the code works becomes its own competitive advantage. Better reliability, faster fixes and iteration, deeper and broader capabilities that allow you to be disruptive while everything else is being built towards the mean, etc etc. Maybe we’ve not been in this new age for long enough for that to be reflected in people’s purchasing criteria, but I’m quite looking forward to fending off AI-built competitors with this edge.
• AlexWilkins12 6mo ago
  Ironically, this article reeks of AI-generated phrases. Lot's of "It's not X, it's Y". eg: - "The failure mode isn't malice. It's convenience", - "You haven't saved time. You've forfeited the experience that the time was supposed to give you." - "But the real threat isn't either of those things. It's quieter, and more boring, and therefore more dangerous. The real threat is a slow, comfortable drift toward not understanding what you're doing. Not a dramatic collapse. Not Skynet. Just a generation of researchers who can produce results but can't produce understanding." And indeed running it through a few AI text detectors, like Pangram (not perfect, by any means, but a useful approximation), returns high probabilities. It would have felt more honest if the author had included a disclaimer that it was at least part written with AI, especially given its length and subject matter.
  • zozbot234 6mo ago
    Yes, the overwrought "It's not X it's Y" are signals of LLM involvement. No human uses them like that all the frickin' time. AI loves this construct way too much and cannot really tell whether the contrast is relevant/actually makes sense.
    • suzzer99 6mo ago
      It almost never does.
    • rossant 6mo ago
      Seeing these constructs in a text is not just a vague hint that it was AI-generated. It's a smoking gun.
    • dimal 6mo ago
      I think I must be an AI. I’ve written those types of phrases a lot long before the advent of LLMs and I love em dashes. LLMs were trained on human text. Where do you think they got these patterns from? Looking at the rest of the text, I don’t see some of the other LLM tells like excessive usage of bullet lists, and repetition of the same ideas over and over using different language. It’s a well written, engaging article. Why would it need a disclaimer if they used AI on part of it? They’re not arguing against the usage of LLMs.
• robot-wrangler 6mo ago
  Another threat is that you can find tons of papers pointing out how neural AI still struggles handling simple logical negation. Who cares right, we use tools for symbolics, yada yada. Except what's really the plan? Are we going to attempt parallel formalized representations of every piece of input context just to flag the difference between please DONT delete my files and please DO? This is all super boring though and nothing bad happened lately, so back to perusing latest AGI benchmarks..
• squirrel 6mo ago
  The article is well-written and makes cogent points about why we need "centaurs", human/computer hybrids who combine silicon- and carbon-based reasoning. Interestingly, the text has a number of AI-like writing artifacts, e.g. frequent use of the pattern "The problem isn't X. The problem is Y." Unlike much of the typical slop I see, I read it to the end and found it insightful. I think that's because the author worked with an AI exactly as he advocates, providing the deep thinking and leaving some of the routine exposition to the bot.
  • boxomcfoxo 6mo ago
    Nope. It was actually written entirely by Claude. https://boxobarks.leaflet.pub/3mj42airv3s2o#fingerprints-of-the-beige-liebox https://boxobarks.leaflet.pub/3mj42airv3s2o#fingerprints-of-... The framing of the essay around learning through "grunt work" is not deep, it's simply that this specific phrase appeared in two of the sources. Anything that looks like insight is plagiarised directly from the sources in some fashion. I've covered in my evidence the pivot phrases where direct summaries of the essay incorrectly appear to transition to the author's own ideas, but there are parts right through the essay that come from the sources. No deep thinking by the prompter required.
    • squirrel 6mo ago
      Thanks for writing that up. You convinced me that there was more Claude here than I'd thought, but I didn't see evidence that the author hadn't edited and supplemented, which is what I was suggesting. In fact, your last observation about correcting an erroneous date makes my point, not yours: Claude made a mistake, and the (human) author fixed it, thus improving the essay. I certainly agree that the author should disclose the use of an AI, how much is human vs silicon, and clarify which ideas are his own and which are not. I've written to him to ask about that.
      • squirrel 6mo ago
        The author replied quickly and described his use of AI as very limited and just for grammar and wording. I believe him, based both on the text of the article itself and what he told me.
• grafelic 6mo ago
  "He shipped a product, but he didn't learn a trade." I think is the key quote from this article, and encapsulates the core problem with AI agents in any skill-based field.
• DavidPiper 6mo ago
  I've just started a new role as a senior SWE after 5 months off. I've been using Claude a bit in my time off; it works really well. But now that I've started using it professionally, I keep running into a specific problem: I have nothing to hold onto in my own mind. How this plays out: I use Claude to write some moderately complex code and raise a PR. Someone asks me to change something. I look at the review and think, yeah, that makes sense, I missed that and Claude missed that. The code works, but it's not quite right. I'll make some changes. Except I can't. For me, it turns out having decisions made for you and fed to you is not the same as making the decisions and moving the code from your brain to your hands yourself. Certainly every decision made was fine: I reviewed Claude's output, got it to ask questions, answered them, and it got everything right. I reviewed its code before I raised the PR. Everything looked fine within the bounds of my knowledge, and this review was simply something I didn't know about. But I didn't make any of those decisions. And when I have to come back to the code to make updates - perhaps tomorrow - I have nothing to grab onto in my mind. Nothing is in my own mental cache. I know what decisions were made, but I merely checked them, I didn't decide them. I know where the code was written, but I merely verified it, I didn't write it. And so I suffer an immediate and extreme slow-down, basically re-doing all of Claude's work in my mind to reach a point where I can make manual changes correctly. But wait, I could just use Claude for this! But for now I don't, because I've seen this before. Just a few moments ago. Using Claude has just made it significantly slower when I need to use my own knowledge and skills. I'm still figuring out whether this problem is transient (because this is a brand new system that I don't have years of experience with), or whether it will actually be a hard blocker to me using Claude long-term. Assuming I want to be at my new workplace for many years and be successful, it will cost me a lot in time and knowledge to NOT build the castle in the sky myself.
  • xandrius 6mo ago
    Then you're using it more towards vibe coding than AI-assisted coding: I use AI to write the stuff the way I want it to be written. I give it information about how to structure files, coding style and the logic flow. Then I spend time to read each file change and give feedback on things I'd do differently. Vastly saves me time and it's very close or even better than what I would have written. If the result is something you can't explain than slow down and follow the steps it takes as they are taken.
• theteapot 6mo ago
  I have a vaguely unrelated question re: > You do what your supervisor did for you, years ago: you give each of them a well-defined project. Something you know is solvable, because other people have solved adjacent versions of it. Something that would take you, personally, about a month or two. You expect it to take each student about a year ... Is that how PhD projects are supposed to work? The supervisor is a subject matter expert and comes up with a well-defined achievable project for the student?
  • _gmax1 6mo ago
    From the cases I've observed directly in the area I work in, yes.
  • InkCanon 6mo ago
    Often at the start yes. So the students gets a bit of recognition, a bit of experience and a bit of knowledge.
  • loveparade 6mo ago
    I think it just really depends. There is no fixed rule to how PhD programs are supposed to work. Sometimes your advisor will suggest projects he finds interesting and wants to see done, he just doesn't have time to do it himself. That's pretty common. Sometimes advisors don't have that and/or want students to come up with their own projects proposals, etc.
  • LeonardoTolstoy 6mo ago
    It is a spectrum. My advisor was very hands off. He didn't, ultimately, even really understand my PhD. He knew the problem, but he had no path in mind to solve it, that was up to me. I'm now working (as a software engineer) with a person who is very hands on with his students (and even postdocs) to the point of giving them specific tasks to do and then discussing the result every week. He defines the problems and structure of the solution, the students at least partially are an extension of himself, they are doing stuff he merely doesn't have time to do himself. And there is everything in between.
  • derbOac 6mo ago
• mikeaskew4 6mo ago
  “The world still needs empirical thinkers, Danny.” - Caddyshack
• scrpgil 6mo ago
  [flagged]
  • deleted 6mo ago
    [deleted]
  • bojan 6mo ago
    The danger is also for Alice, if she gets moved to Bob's project when things start falling apart in production.
    • scrpgil 6mo ago
      [flagged]
• jerkstate 6mo ago
  Nobody actually understands what they're doing. When you're learning electronics, you first learn about the "lumped element model" which allows you to simplify Maxwell's equations. I think it is a mistake to think that solving problems with a programming language is "knowing how to do things" - at this point, we've already abstracted assembly language -> machine instructions -> logic gates and buses -> transistors and electronic storage -> lumped matter -> quantum mechanics -> ???? - so I simply don't buy the argument that things will suddenly fall apart by abstracting one level higher. The trick is to get this new level of abstraction to work predictably, which admittedly it isn't yet, but look how far it's come in a short couple of years. This article first says that you give juniors well-defined projects and let them take a long time because the process is the product. Then goes on to lament the fact that they will no longer have to debug Python code, as if debugging python code is the point of it all. The thing that LLMs can't yet do is pick a high-level direction for a novel problem and iterate until the correct solution is reached. They absolutely can and do iterate until a solution is reached, but it's not necessarily correct. Previously, guiding the direction was the job of the professor. Now, in a smaller sense, the grad student needs to be guiding the direction and validating the details, rather than implementing the details with the professor guiding the direction. This is an improvement - everybody levels up. I also disagree with the premise that the primary product of astrophysics is scientists. Like any advanced science it requires a lot of scientists to make the breakthroughs that trickle down into technology that improves everyday life, but those breakthroughs would be impossible otherwise. Gauss discovered the normal distribution while trying to understand the measurement error of his telescope. Without general relativity we would not have GPS or precision timekeeping. It uncovers the rules that will allow us to travel interplanetary. Understanding the composition and behavior of stars informs nuclear physics, reactor design, and solar panel design. The computation systems used by advanced science prototyped many commercial advances in computing (HPC, cluster computing, AI itself). So not only are we developing the tools to improve our understanding of the universe faster, we're leveling everybody up. Students will take on the role of professors (badly, at first, but are professors good at first? probably not, they need time to learn under the guidance of other faculty). professors will take on the role of directors. Everybody's scope will widen because the tiny details will be handled by AI, but the big picture will still be in the domain of humans.
• hgo 6mo ago
  I like this article and it reads well, but I have to say, that to me it really reads as something written by an LLM. Probably under supervision by a human that knew what it should say. I don't know if I mind. Example. This paragraph, to me, has a eerily perfect rhythm. The ending sentence perfectly delivers the twist. Like, why would you write in perfect prose an argument piece in the science realm? > Unlike Alice, who spent the year reading papers with a pencil in hand, scribbling notes in the margins, getting confused, re-reading, looking things up, and slowly assembling a working understanding of her corner of the field, Bob has been using an AI agent. When his supervisor sent him a paper to read, Bob asked the agent to summarize it. When he needed to understand a new statistical method, he asked the agent to explain it. When his Python code broke, the agent debugged it. When the agent's fix introduced a new bug, it debugged that too. When it came time to write the paper, the agent wrote it. Bob's weekly updates to his supervisor were indistinguishable from Alice's. The questions were similar. The progress was similar. The trajectory, from the outside, was identical.
  • kelnos 6mo ago
    Or maybe the author is just a competent writer.
    • hgo 6mo ago
      Yes. Let's assume so. My point is the suspicion itself.
      • alex_suzuki 6mo ago
        I hate that this is the first thing that crosses my mind now anytime I read a well-written article.
  • swiftcoder 6mo ago
    > why would you write in perfect prose If you could, why wouldn't you? LLM witch-hunts over every halfway competent writer are becoming quite tiresome
• lambdaone 6mo ago
  Very insightful. One key sentence sums it up: "He shipped a product, but he didn't learn a trade." This is going to get worse, and eventually cause disastrous damage unless we do something about it, as we risk losing human institutional memory across just about every domain, and end up as child-like supplicants to the machines. But as the article says, this is a people problem, not a machine problem.
• patapong 6mo ago
  I think this is a very important debate, and I think the author here adds a lot to this discussion! I mostly agree with it, but wanted to point out a few areas where I do not fully agree. > Take away the agent, and Bob is still a first-year student who hasn't started yet. This may be true, but I can see almost no conceivable word where the agent will be taken away. I think we should evaluate Bob's ability based on what he can do with an agent, not without, and here he seems to be doing quite well. > I've been hearing "just wait" since 2023. On almost any timeline, this is very short. Given the fact that we have already arrived at models able to almost build complete computer programs based on a single prompt, and solve frontier level math problems, I think any framework that relies on humans continuing to have an edge over LLMs in the medium term may be built on shaky grounds. Two very interesting questions today in this vein for me are: - Is the best way to teach complex topics to students today to have them carry out simple tasks? The author acknowledges that the difference between Bob and Alice only materializes at a very high level, basically when Alice becomes a PI of her own. If we were solely focused on teaching thinking at this level (with access to LLMs), how would we frame the educational path? It may look exactly like it does now, but it could also look very differently. - Is there inherent value in humans learning specific skills? If we get to a stage where LLMs can carry out most/all intellectual tasks better than humans, do we still want humans to learn these skills? My belief is yes, but I am frankly not sure how to motivate this answer.
  • ThrowawayR2 6mo ago
    > "no conceivable word where the agent will be taken away" LLM access is a paid service. HN concerns itself with inequality constantly and it's not inconceivable that some individuals get ahead because they can afford to pay for more tokens and better models than those who are poorer.
• __MatrixMan__ 6mo ago
  But aren't you still going to have to convince other people to let you do it with their money/data/hardware/etc? The understanding necessary to make that argument well is pretty deep and is unaffected by AI. I've been having a lot of fun vibe coding little interactive data visualizations so when I present the feature to stakeholders they can fiddle with it and really understand how it relates to existing data. I saw the agent leave a comment regarding Cramer's rule and yeah its a bit unsettling that I forgot what that is and haven't bothered to look it up, but I can tell from the graphs that its doing the correct thing. There's now a larger gap between me and the code, but the chasm between me and the stakeholders is getting smaller and so far that feels like an improvement.
  • danielbln 6mo ago
    Every AI/agentic thread on HN follows the same tension: builders want to build and solve problems. Code or task completion are implementation details to be done on the path to the actual prize: solving the problem. And then there are the coders, that have honed their mechanical skill of implementation and derive their intellectual fulfillment from that. The latter crowd has a rough time because much of it can be automated now, the former camp is happy because look at all the stuff that can now be built!
• Lerc 6mo ago
  The problem I see with this argument is that the ship sailed on understanding what you are doing years ago. It seems like it is abstraction layers all the way down. If an AI is capable of producing an elegant solution with fewer levels of abstraction it could be possible that we end up drifting towards having a better understanding of what's going on.
• FrojoS 6mo ago
  Every PhD program I'm aware of has a final hurdle known as the defence. You have to present your thesis while standing in front of a committee, and often the local community and public. They will asks questions and too many "I don't know" or false answers would make you fail. So, there is already a system in place that should stop Bob from graduating if he indeed learned much less than Alice. A similar argument can be made for conference publications. If Bob publishes his first year project at a conference but doesn't actually understand "his own work" it will show. The difficulty of passing the defence vary's wildly between Universities, departments and committees. Some are very serious affairs with a decent chance of failure while others are more of a show event for friends and family. Mine was more of the latter, but I doubt I would have passed that day if I had spend the previous years prompting instead of doing the grunt work.
  • ipaddr 6mo ago
    In the future the llms can answer those questions for you by listening and feeding you answers into your headset. The process you describe is a gate keeping exercise which will change to include llm judges at somepoint.
    • FrojoS 6mo ago
      That would be cheating. If the exam is 'gate keeping', I will say that it is a gate worth keeping. To be clear, I am not against alternative forms of education. Degrees are optional. But if you want a degree, there have to be exams and cheating has to be prevented.
• BobBagwill 6mo ago
  Try giving this problem to different AI LLM chatbots: If I could make a rocket that could accelerate at 3 Gs for 10 years, how long would it take to travel from Earth to Alpha Centauri by accelerating at 3 Gs for half the time, then decelerating at 3 Gs for half the time? Hint: They don't all get it right. Some of them never got it right after hints, corrections, etc.
• omega3 6mo ago
  I wonder what effect AI had on online education - course signups, new resources being added etc. I’ve recently started csprimer and whilst mentally stimulating I wonder if I’m not completely wasting my time.
• deleted 6mo ago
  [deleted]
• CharlieDigital 6mo ago
  I recently saw a preserved letterpress printing press in person and couldn't help but think of the parallels to the current shift in software engineering. The letterpress allowed for the mass production of printed copies, exchanging the intensive human labor of manual copying to letter setting on the printing press. Yet what did not change in this process is that it only made the production of the text more efficient; the act of writing, constructing a compelling narrative plot, and telling a story were not changed by this revolution. Bad writers are still bad writers, good writers still have a superior understanding of how to construct a plot. The technological ability to produce text faster never really changed what we consider "good" and "bad" in terms of written literature; it just allow more people to produce it. It is hard to tell if large language models can ever reach a state where it will have "good taste" (I suspect not). It will always reflect the taste and skill of the operator to some extent. Just because it allows you to produce more code faster does not mean it allows you to create a better product or better code. You still need to have good taste to create the structure of the product or codebase; you still have to understand the limitations of one architectural decision over another when the output is operationalized and run in production. The AI industry is a lot of hype right now because they need you to believe that this is no longer relevant. That Garry Tan producing 37,000 LoC/day somehow equates to producing value. That a swarm of agents can produce a useful browser or kernel compiler. Yet if you just peek behind the curtains at the Claude Code repo and see the pile of unresolved issues, regressions, missing features, half-baked features, and so on -- it seems plainly obvious that there are limitations because if Anthropic, with functionally unlimited tokens with frontier models, cannot use them to triage and fix their own product. AI and coding agents are like the printing press in some ways. Yes, it takes some costs out of a labor intensive production process, but that doesn't mean that what is produced is of any value if the creator on the other end doesn't understand the structure of the plot and the underlying mechanics (be it of storytelling or system architecture).
• bambushu 6mo ago
  [flagged]
• hnzionists 6mo ago
  [dead]
• steveBK123 6mo ago
  I agree with the general premise - the risk is we don’t develop juniors (new Alices) anymore, and at some point people are just sloperators gluing together bits of LLM output they do not understand. I have seen versions of this in the wild where a firm has gone through hard times and internally systems have lost all their original authors, and every subsequent generation of maintainers… being left with people in awe of the machine that hasn’t been maintained in a decade. I interviewed a guy once that genuinely was proud of himself, volunteering the information to me as he described resolving a segfault in a live trading system by putting kill -9 in a cronjob. Ghastly.
• somethingsome 6mo ago
  Personally, I wrote an essay to my students explaining exactly that the purpose is for them to think better and improve over time, they can use LLMs but, if they stop thinking, they are just failing themselves, not me. It had great success, now when I propose to them to use some model to do something, they tends to avoid.
• tmountain 6mo ago
  Thankfully, I am nearing the end of my career with software after 25 years well spent. If I had been born in a different decade, I would be facing the brunt of the AI shift, and I don’t think I would want to continue in the industry. Obviously, this is a personal decision, but we are in a totally different domain now, where, at best, you’re managing an LLM to deliver your product.
• shellkr 6mo ago
  This is almost the same as going from making fire with a stick to using a lighter.. sure it is simplified but still not wrong. Humans while still doing grunt work can still make mistakes as does the machine.. the machine will eventually discover it. The same can not be said of the human because of the work needed to do so might be too much. In the end we might not learn as much.. but it will not matter and thus is really not an issue.
  • techblueberry 6mo ago
    I think I disagree in what I see around me it’s less like going from fire to lighter and more like going from hand tools to power tools. If your skill was in understanding how the hand tools work, it’s harder to get a level of abstraction up and have a vision for building a house. If we’re not able to learn than less people are going to be able to get that vision, especially if you’re in technical domains where engineering and architecture matter. It’s going to be a weird future. I’m pretty effective with these tools, but I’m fifteen years of hacking on them manually. Some folks who are not as far into their careers don’t seem to know where to start. There’s a reason most people aren’t promoted to manager until they have years of experience under their belt. And now we’re expecting folks to be managers on day 1.
• steveBK123 6mo ago
  For the people arguing that the output is the code and the faster we generate it the better.. I do wonder where all the novel products produced by 10x devs who are now 100x with LLMs, the “idea guys” who can now produce products from whole clothe without having to hire pesky engineers.. where is the one-man 10 billion dollar startups, etc? We are 3-4 years into this mania and all I see on the other end of it is the LLMs themselves. Why hasn’t anything gotten better?
  • mikeaskew4 6mo ago
    Could be possible that the 10x devs working at 100x are just starting down the homestretch… The 10x dev doesn’t just set out to build a hello world app, ya know.
    • steveBK123 6mo ago
      I think its telling that the two main places I've seen the biggest in-roads in FinTech in terms of LLMs has been: 1) Stuff that was astonishingly not automated yet. I am talking about somebody opening up excel on one screen, and a website/pdf/whatever on the other.. and type stuff in to your excel sheet. So stuff where there wasn't any code involved previously, possibly due to diminishing returns of how adhoc it was to automate, skills mismatch, organizational politics or other reasons. 2) Lot of former BigData / crypto / SaaS guys who were in product/sales roles suddenly starting AI startups to help your company AI better. The product is facilitating the doing of AI.
  • maplethorpe 6mo ago
    I'm waiting for Anthropic to realise they can just set a few thousand agents loose to do just that, and monopolize the entire software market overnight. I'm not sure why they haven't done this yet.
    • steveBK123 6mo ago
      Because a lot of valuable software is the implicit / organizational / human domain knowledge .. not the trillions of lines of code LLms all scraped and trained on.
• maplethorpe 6mo ago
  I honestly don't know why this guy is hiring Alice and Bob in the first place, instead of just running two agents. He seemed to be saying it's to invest in them as people, but why? What is the end goal? If the end goal is to produce research, then just get the agents to do it.
• itmitica 6mo ago
  Contrarian just for the sake of it. Get on board or stay behind. Whatever good or bad AI brings to the table, it's here to stay. The cat's out of the bag. Might as well enjoy it. Evolution will not stay on your whimsical made-up reality. It will run you over.
  • techblueberry 6mo ago
    What if AI in the long run makes us slower and less effective. As someone who is one of the folks supercharged by these tools, I could see it. I think people are underestimating the level of experience and knowledge that’s required to prompt LLM’s. Not in the micro sense but in the macro. It seems so easy because it feels easy. But if you don’t have deep understanding of the domain, it will just feel impossible. The person next to you with domain experience will say “it’s so easy, look at This simple sentence I typed in”. And be like “it’s just a skill issue, why is everyone struggling so much” and not understand the years of accumulated wisdom or innate talent it took to type that simple sentence. AI makes the easy things easy and the hard things harder, and more omnipresent.
    • itmitica 6mo ago
      My experience is AI helping me unload things I am not built for. It allows me to be creative with less drag. Not everyone aims to be useless human automaton or useless human thesaurus. It allows me to exercise my intelligence freely.
      • techblueberry 6mo ago
        We’re kind of saying the same thing. But If you only learn LLMs for work, and then you only do the work assigned with the LLM and you don’t see it as an opportunity to leverage creativity and curiosity, you won’t be adding value to the process and someone else will take your job.
  • ergl 6mo ago
    Do you have any more platitudes to add so I can fill my dismissive HN comment bingo card?
• Wowfunhappy 6mo ago
  > Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expectations. Then Schwartz read it, and it was wrong. Claude had been adjusting parameters to make plots match instead of finding actual errors. It faked results. It invented coefficients. [...] Schwartz caught all of this because he's been doing theoretical physics for decades. He knew what the answer should look like. He knew which cross-checks to demand. [...] If Schwartz had been Bob instead of Schwartz, the paper would have been wrong, and neither of them would have known. And so the paradox is, the LLMs are only useful† if you're Schwartz, and you can't become Schwartz by using LLMs. Which means we need people like Alice! We have to make space for people like Alice, and find a way to promote her over Bob, even though Bob may seem to be faster. The article gestures at this but I don't think it comes down hard enough. It doesn't seem practical. But we have to find a way, or we're all going to be in deep trouble when the next generation doesn't know how to evaluate what the LLMs produce! --- † "Useful" in this context means "helps you produce good science that benefits humanity".
  • deleted 6mo ago
    [deleted]
  • conception 6mo ago
    Sadly I don’t see how our current social paradigm works for this. There is no history of any sort of long planning like this or long term loyalty (either direction) with employees and employers for this sort of journeyman guild style training. AI execs are basically racing, hoping we won’t need a Schwartz before they are all gone. But what incentives are in place to high a college grad, have them work without llms for a decade and then give them the tools to accelerate their work?
    • Wowfunhappy 6mo ago
      Then the social paradigm needs to change. Is everyone just going to roll over and die while AI destroys academia (and possibly a lot more)? Last September, Tyler Austin Harper published a piece for The Atlantic on how he thinks colleges should respond to AI. What he proposes is radical—but, if you've concluded that AI really is going to destroy everything these institutions stand for, I think you have to at least consider these sorts of measures. https://www.theatlantic.com/culture/archive/2025/09/ai-colleges-universities-solution/684160/ https://www.theatlantic.com/culture/archive/2025/09/ai-colle...
• mkovach 6mo ago
  This isn't new. It's been the same problem for decades, not what gets built, but what gets accepted. Weak ownership, unclear direction, and "sure, I guess" reviews were survivable when output was slow. When changes came in one at a time, you could get away with not really deciding. AI doesn't introduce a new failure mode. It puts pressure on the old one. The trickle becomes a firehose, and suddenly every gap is visible. Nobody quite owns the decision. Standards exist somewhere between tribal memory, wishful thinking, and coffee. And the question of whether something actually belongs gets deferred just long enough to merge it, but forces the answer without input. The teams doing well with agentic workflows aren't typically using magic models. They've just done the uncomfortable work of deciding what they're building, how decisions are made, and who has the authority to say no. AI is fine, it just removed another excuse for not having our act together. While we certainly can side-eye AI because of it, we own the problems. Well, not me. The other guy who quit before I started.
  • jappgar 6mo ago
    This is exactly the problem I see today. And it's not just a volume problem. Mediocre devs previously couldn't complete a project by themselves and were forced to solicit help and receive feedback along the way. When all managers care about is "shipping", development becomes a race to the bottom. Devs who used to collaborate are now competing. Whoever gets the slop into the codebase fastest, wins.
    • mkovach 6mo ago
      This is also very true, and while I consider it part of the authority to say no, this is a significant point.
• pbw 6mo ago
  There's certainly a risk that an individual will rely too much on AI, to the detriment of their ability to understand things. However, I think there are obvious counter-measures. For example, requiring that the student can explain every single intermediate step and every single figure in detail. A two-hour thesis defense isn't enough to uncover this, but a 40-hour deep probing examination by an AI might be. And the thesis committee gets a "highlight reel" of all the places the student fell short. The general pattern is: "Suppose we change nothing but add extensive use of AI, look how everything falls apart." When in reality, science and education are complex adaptive systems that will change as much as needed to absorb the impact of AI.
• bluedino 6mo ago
  Look at how bad the auto industry has gotten when it comes to quality and recalls. A combination of beancounters running the show and the old, experienced engineers dying, retiring, and going through buyouts has pretty much left things in a pretty sad state.
• zaikunzhang 6mo ago
  Earlier posts: https://news.ycombinator.com/item?id=47644808 https://news.ycombinator.com/item?id=47644808 https://news.ycombinator.com/item?id=47627645 https://news.ycombinator.com/item?id=47627645 https://news.ycombinator.com/item?id=47623788 https://news.ycombinator.com/item?id=47623788 https://news.ycombinator.com/item?id=47619990 https://news.ycombinator.com/item?id=47619990
• devnotes77 6mo ago
  [dead]
• lxgr 6mo ago
  > for someone who doesn't yet have that intuition, the grunt work is the work Very well said. I think people are about to realize how incredibly fortunate and exceptional it is to actually get paid, and in our industry very well, through a significant fraction of one's career while still "just" doing the grunt work, that arguably benefits the person doing it at least as much as the employer. A stable paid demand for "first-year grad student level work" or the equivalent for a given industry is probably not the only possible way to maintain a steady supply of experts (there's always the option of immense amounts of student debt or public funding, after all), but it sure seems like a load-bearing one in so many industries and professions. At the very least, such work being directly paid has the immense advantage of making artificially (often without any bad intentions!) created bullshit tasks that don't exercise actually relevant skillsets, or exercise the wrong ones, much easier to spot.
• sunir 6mo ago
  I think the mountain of things I don’t understand was already huge. It doesn’t stop me from getting a grip over the things I need to be responsible for and using tools to contain complexity irrelevant to me. Like many scientists have a stats person. The risk is that civilization is over its skis because humans are lazy. Humans are always lazy. In science there’s a limit to bs because dependent works fail. In economics there’s a crash. In physics stuff breaks. Then there is a correction.
• ahussain 6mo ago
  > When his supervisor sent him a paper to read, Bob asked the agent to summarize it. When he needed to understand a new statistical method, he asked the agent to explain it. When his Python code broke, the agent debugged it. When the agent's fix introduced a new bug, it debugged that too. When it came time to write the paper, the agent wrote it. Bob's weekly updates to his supervisor were indistinguishable from Alice's. In my experience, doing these things with the right intentions can actually improve understanding faster than not using them. When studying physics I would sometimes get stuck on small details - e.g. what algebraic rule was used to get from Eq 2.1 to 2.2? what happens if this was d^2 instead of d^3 etc. Textbooks don't have space to answer all these small questions, but LLMs can, and help the student continue making progress. Also, it seems hard to imagine that Alice and Bob's weekly updates would be indistinguishable if Bob didn't actually understand what he was working on.
  • sumeno 6mo ago
    Faster doesn't always mean better. I've "learned" things from LLM really fast, but I don't retain the information the same way as if I had taken my time to really work through it
• caxap 6mo ago
  If this article was written a year ago, I would have agreed. But knowing what I know today, I highly doubt that the outcomes of LLM/non-LLM users will be anywhere close to similar. LLMs are exceptionally good at building prototypes. If the professor needs a month, Bob will be done with the basic prototype of that paper by lunch on the same day, and try out dozens of hypotheses by the end of the day. He will not be chasing some error for two weeks, the LLM will very likely figure it out in matter of minutes, or not make it in the first place. Instructing it to validate intermediate results and to profile along the way can do magic. The article is correct that Bob will not have understood anything, but if he wants to, he can spend the rest of the year trying to understand what the LLM has built for him, after verifying that the approach actually works in the first couple of weeks already. Even better, he can ask the LLM to train him to do the same if he wishes. Learn why things work the way they do, why something doesn't converge, etc. Assuming that Bob is willing to do all that, he will progress way faster than Alice. LLMs won't take anything away if you are still willing to take the time to understand what it's actually building and why things are done that way. 5 years from now, Alice will be using LLMs just like Bob, or without a job if she refuses to, because the place will be full of Bobs, with or without understanding.
  • piiritaja 6mo ago
    "LLMs won't take anything away if you are still willing to take the time to understand what it's actually building" But do you actually understand it? The article argues exactly against this point - that you cannot understand the problems in the same way when letting agents do the initial work as you would when doing it without agents. from the article: "you cannot learn physics by watching someone else do it. You have to pick up the pencil. You have to attempt the problem. You have to get it wrong, sit with the wrongness, and figure out where your reasoning broke. Reading the solution manual and nodding along feels like understanding. It is not understanding. Every student who has tried to coast through a problem set by reading the solutions and then bombed the exam knows this in their bones. We have centuries of accumulated pedagogical wisdom telling us that the attempt, including the failed attempt, is where the learning lives. And yet, somehow, when it comes to AI agents, we've collectively decided that maybe this time it's different. That maybe nodding at Claude's output is a substitute for doing the calculation yourself. It isn't. We knew that before LLMs existed. We seem to have forgotten it the moment they became convenient."
• ChrisMarshallNY 6mo ago
  This is not wrong, but the "Bob and Alice" conundrum is not simple, either. In academia, understanding is vital. The same for research. But in production, results are what matters. Alice would be a better researcher, but Bob would be a better producer. He knows how to wrangle the tools. Each has its value. Many researchers develop marvelous ideas, but struggle to commercialize them, while production-oriented engineers, struggle to come up with the ideas. You need both.
  • cmiles74 6mo ago
    I have to disagree that Bob will be a better producer, although I do agree that Bob will produce more. In this scenario, Bob isn't clear on which LLM output is valid and important and which is erroneous and misleading; I think that's a pretty critical distinction. It's the kind of thing that might go undetected for a long time, until a particular paper turns out to be important and it's discovered that it's also entirely wrong, wasting a lot of time and energy.
    • ChrisMarshallNY 6mo ago
      Sounds like you're still thinking of Bob as a researcher. In production, there would be no "paper"; just some software/hardware product. If there was a problem, that would be fairly obvious, with testing (we are going to be testing our products, right?). I have been wrestling all morning, with an LLM. It keeps suggesting stuff that doesn't work, and I need to keep resetting the context. I am often able to go in, and see what the issue is, but that's almost worthless. The most productive thing that I can do, is tell the LLM what is the problem, on the output end, and ask it to review and fix. I can highlight possible causes, but it often finds corner cases that I miss. I have to be careful not to be too dictatorial. It's frustrating, as the LLM is like a junior programmer, but I can make suggestions that radically improve the result, and the total time is reduced drastically. I have gotten done, in about two hours, what might have taken all day.
      • cmiles74 6mo ago
        Indeed! In the article Bob is an astrophysicist. I think the difference between the workflow you describe and the description of Bob's is that you have a pretty good idea of what a working solution would look like. In my reading if the article Bob does not. In my opinion, the software developer analogue of Bob would be someone who would often reach for the LLM as it's nearby and easy. Maybe at first they would be careful about reading and vetting the model's output. Over time they might grow comfortable and overly confident with the model's output and pay less and less attention. As they take on more complex tasks they begin to understand less and less about the LLM tooling output but they don't really notice, it all looks good and tests are passing. Eventually we see a production problem, maybe even an outage. When we narrow down the issue to a PR with Bob's name on it and ask him how it led to the production issue, Bob tries to be helpful but struggles to understand his own PR.
• visarga 6mo ago
  > Whether that student walks out the door five years later as an independent thinker or a competent prompt engineer is, institutionally speaking, irrelevant. I think this is a simplification, of course Bob relied on AI but they also used their own brain to think about the problem. Bob is not reducible to "a competent prompt engineer", if you think that just take any person who prompts unrelated to physics and ask them to do Bob's work. In fact Bob might have a change to cover more mileage on the higher level of work while Alice does the same on the lower level. Which is better? It depends on how AI will evolve. The article assumes the alternative to AI-assisted work is careful human work. I am not sure careful human work is all that good, or that it will scale well in the future. Better to rely on AI on top of careful human work. My objection comes from remembering how senior devs review PRs ... "LGTM" .. it's pure vibes. If you are to seriously review a PR you have to run it, test it, check its edge cases, eval its performance - more work than making the PR itself. The entire history of software is littered with bugs that sailed through review because review is performative most of the time. Anyone remember the verification crisis in science?
• zaikunzhang 6mo ago
  See also D. W. Hogg, "Why do we do astrophysics?", https://arxiv.org/abs/2602.10181 https://arxiv.org/abs/2602.10181, February 2026.
• jeremie_strand 6mo ago
  [dead]
• dwa3592 6mo ago
  What a wonderful read. Thank you! The way I think about this is : We can't catch the hallucinations that we don't know are hallucinations.
• txrx0000 6mo ago
  The threat is if you replace your cognitive capabilities with AI, but you don't control entire the system your AI runs on (hardware, firmware, drivers, OS, weights, frontend), then that's equivalent to someone else owning a part of your brain.
• deleted 6mo ago
  [deleted]
• fredgrott 6mo ago
  I know how we can fix this.... Its of course devious, exactly some of our styles :) Give AI to VCs to use for all their domain stuff.... They than make wrong investment decisions based on AI wrong info and get killed in the market.... Market ends up killing AI outright....problem solved temporarily
• bwfan123 6mo ago
  > The problem isn't that we'll decide to stop thinking. The problem is that we'll barely notice when we do Most of what we call thinking is merely to justify beliefs that emotionally make us happy and is not creative per-se. I am making a distinction between "thinking" as we know it and "creative thinking" which is rare, and can see things in an unbiased manner breaking out of known categories. Arguably, at the PhD level, there needs to be a new ideas instead of remixing the existing ones.
• matheusmoreira 6mo ago
  I dunno. Claude helped me implement a new memory allocator, compacting garbage collector and object heap for my programming language. I certainly understood what I was doing when I did this. The experience was extremely engaging for me. Claude taught me a lot. I think the real danger is no longer caring about what you're doing. Yesterday I just pointed Claude at my static site generator and told it to clean it up. I wanted to care but... I didn't.
  • krackers 6mo ago
    This seems contradictory at first glance, if you didn't actually implement it then how well have you actually understood it? It's known from learning theory that engagement or even self-reported understanding doesn't imply that the student can actually solve problems presented to him. If someone claims to have "understood [middle school] algebra" but they aren't able to solve equations by themselves, you'd be skeptical. Of course past some point of familiarity it's simply faster to throw things into a CAS, but if you remove the initial manual struggle, then have you wired up your brain for understanding? There was a post on HN a few days back about how familiarity with a tool leads to a sense of "embodied understanding" [1], and I think the initial struggle is an intrinsic part of learning to get to the "unconscious competence' level. [1] https://news.ycombinator.com/item?id=47640775 https://news.ycombinator.com/item?id=47640775
    • matheusmoreira 6mo ago
      I did implement it though. I wrote the code myself. Claude explained to me how all of those things worked, showed me code and examples to illustrate, walked me through the algorithms step by step. It turned out to not be as insurmountably complex as I thought it was. I'm also manually writing articles about all of those things to crystallize everything I learned. Claude has been amazing for code review. Having someone to talk to about my own code is world changing for a solo developer like me.
      • krackers 6mo ago
        Oh ok that's different! From your comment I assumed that it was Claude Code doing the implementation while you only gave it high level concepts (e.g. "add a generational GC"). But if you use it as a resource to clarify concepts while you write the implementation yourself, then it's no different than having a tutor who helps you understand so you can do the homework yourself. Given the article, I don't think most people are using LLMs in a "tutor" fashion to learn how to do the homework, they're effectively having their homework done by the tutor.
• talkingtab 6mo ago
  This "drift" is not a drift at all, nor is it new. There are many names for this such as cargo cult and think-by-numbers (like paint by numbers), ant mills. It is recipes. And many, many common recipes demonstrate a wide spread lack of understanding. This kind of follow-the-leader kind of "thinking" is probably a requirement. The amount of expertise it would require to understand and decide about things in our daily life is overwhelming. Do you fix your own car, decide each day how to travel, get food and understand how all that works? No. So what is the problem? The problem is that if you follow the leader and the leader has an agenda that differs from your agenda. Do you really think that with Jeff Bezos being a (the?) major investor in Washington Post has anything to do with Democraccy? You know as in the WAPO slogan "Democracy dies in the Dark". Does Jeff have an agenda that differs from yours? Yes. NYT? Yes. Hacker news? Yes. Google? Yes. We now live in a world so filled with propaganda that it makes no difference whether something is AI. We all "follow". Or not.
• acoye 6mo ago
  I recommend the manga BLAME! that explore what happens to humanity if you push this to 11 https://fr.wikipedia.org/wiki/BLAME https://fr.wikipedia.org/wiki/BLAME!
• TraceAgently 6mo ago
  [dead]
• meidad_g 6mo ago
  [flagged]
• pwr1 6mo ago
  I catch myself doing this more than I'd like to admit. Copy something from an LLM, it works, ship it, move on. Then a week later something breaks and I realize I have no idea what that code actually does! The speed is addicting but your slowly trading depth for velocity and at some point that bill comes due.
• alejandrosplitt 6mo ago
  [dead]
• MarcelinoGMX3C 6mo ago
  Frankly, the "AI as accelerant" argument, as fomoz puts it, holds true only when you have a solid understanding of the domain. In enterprise system builds, we don't often encounter theoretical physics where errors might lead to a broken model rather than a broken system. Instead, a faked coefficient from an LLM could mean a production outage. It's why I push for a hybrid mentor-apprentice model. We need to actively cultivate the next generation of "Schwartzes" with hands-on, critical thinking before throwing them into LLM-driven environments. The current incentive structure, as conception points out, isn't set up for this, but it's crucial if we want to avoid building on sand.
• katzgrau 6mo ago
  When you’re deep in a thoughtful read and suddenly get the eerie feeling that you’re being catfished > But the real threat isn't either of those things. It's quieter, and more boring, and therefore more dangerous. The real threat is a slow, comfortable drift toward not understanding what you're doing. Not a dramatic collapse. Not Skynet. Just a generation of researchers who can produce results but can't produce understanding. Who know what buttons to press but not why those buttons exist. Who can get a paper through peer review but can't sit in a room with a colleague and explain, from the ground up, why the third term in their expansion has the sign that it does.
• cbushko 6mo ago
  This article makes the assumption that Bob was doing absolutely nothing, maybe at the Pub with this friends, while the AI did all his work. How do we know that while the AI was writing python scripts that Bob wasn't reading more papers, getting more data and just overall doing more than Alice. Maybe Bob is terrible at debugging python scripts while Alice is a pro at it? Maybe Bob used his time to develop different skills that Alice couldn't dream of? Maybe Bob will discover new techniques or ideas because he didn't follow the traditional research path that the established Researchers insist you follow? Maybe Bob used the AI to learn even more because he had a customized tutor at his disposal? Or maybe Bob just spent more time at the Pub with his friends.
• toniantunovi 6mo ago
  The coding-specific version of this is worth naming precisely. The drift does not happen because you stop writing code. It happens because you stop reading the output carefully. With AI-generated code, there is a particular failure mode: the code is plausible enough to pass a quick review and tests pass, so you ship it. The understanding degradation is cumulative and invisible until it is not. The partial fix is making automated checks independent of the developer's attention level: type checking, SAST, dependency analysis, and coverage gates that run regardless of how carefully you reviewed the diff. These are not a substitute for understanding, but they create a floor below which "comfortable drift" cannot silently carry you. The question worth asking of any AI coding workflow is whether that floor exists and where it is.
• globalchatads 6mo ago
  [dead]
• alestainer 6mo ago
  I was in academia in the pre-GPT-3 era and I don't see a difference between the superficial pass-the-criteria understanding of things then and now. People already rely on a ton of sources, putting their faith into it, recent replication crisis in social sciences had nothing to do with any LLMs. The problem of academia lies in the first paragraph of this article - supervisor that has to choose doing incremental, clearly feasible stuff. Currently it's called science, but I like to call it knowledge engineering because you're pretty much following a recipe and there is a clear bound on returns to such activities.
• beedeebeedee 6mo ago
  I don’t have kids, but suggested something years ago to my siblings when they started confronting similar issues: we should do a version of “ontogeny recapitulates phylogeny” for personal computers. Kids should start off with Commodore 64s, then get late 80’s or early 90’s Mac’s, then Windows 95, Debian and internet access (but only html). Finally, when they’re 18, be allowed an iPhone, Android and modern computing. Parenting can’t prevent the use of LLMs in grad school, but a similar approach could be taken by grad departments: don’t allow LLMs for the first few years, and require pen and paper exams, as well as oral examinations for all research papers.
  • PessimalDecimal 6mo ago
    I've been doing this with my kids, at least to some extent. It offers first rung on a ladder to understanding that complex things can be understood as cooperation among simpler parts. We'll see how it works out but so far it seems to be working. It's actually great since a lot of older technology is cheap and still readily available. My little ones love listening to old records, control the playback speed and hear the music go up in pitch if the RPMs are set too high. We look at the tracks on the vinyl under a microscope at talk about how the music is written on it that way. VHS an audio cassettes offer their own talking points. For computers, we don't literally use a Commodore 64 but we run simpler, old software on new hardware. Mostly because a lot of newer education software is somehow also funded by injecting ads into the games (awful). But there is also some good "modern" educations software worth checking out. I highly recommend gcompris.net.
• lo_zamoyski 6mo ago
  Education lost the plot years ago. AI is a kind of final nail in that coffin. While we may lament the ravages of AI, I expect there is a kind of providential silver lining in that it may cleanse the rot plaguing education. Just as postermodernism - itself full of errors - is like an enema that is clearing out the disease of modernism and will flush itself out in the process, so, too, AI may be just the purgative we need to force us back to a norm more fittingly called “education”. One of the marks of an educated person is the ability to dispassionately think from first principles. It is not a sufficient criterion, but it is a necessary one. In this case, the basic questions we must ask are: what is education, and what is education for? An instrumentalist view of education, the one that has claimed the soul of the modern university and primary education , tells us that education is about preparing for a career - preparing to be an economic actor - and about the effect you can have. In short, it is about practical power and economic utility. Now, the power to be able to do good things, to be practically able, is a good thing as such, and indeed one does acquire facility during one’s education. (And I would argue schooling today isn’t great at practicality either.) But the practical, unlike the theoretical, is always about something else. It is never for its own sake. What this means is that there must be a terminus. You cannot have an infinite regress of practical ends, because the justification for any practical end is not found in itself. And if the primary proximate end of education is the career, then what distinguishes education from training? Nothing. What’s more, if you then ask what the purpose of a career is, you find it is about consumption. So education today is about enabling people to be consumers. You wish to be effective so you can be payed more so you can buy more crap. Pure nihilism. True education is best captured by the classical liberal arts, which is to say the free arts. Human beings are intellectual and moral creatures. The purpose of education is to free a person to be more human, to free them to be able to reason effectively and competently for the sake of wisdom and for the sake of living wisely. In other words, it is about becoming what you ought to become as a human being in the most definitive sense. What good does AI do you if you haven’t become a better version of yourself in the process? So AI writes a paper for you. So what? The purpose of the paper is not the paper, but the knowledge, understanding, and insight that results from writing it.
• turtletontine 6mo ago
  > Bob's weekly updates to his supervisor were indistinguishable from Alice's. The questions were similar. The progress was similar. The trajectory, from the outside, was identical. I don’t believe this. Totally plausible that someone would be able to produce passable work with LLMs at a similar pace to a curious and talented scientist. But if you, their advisor, are sitting down and talking with them every week? It’s obvious how much they care or understand, I can’t believe you wouldn’t be able to tell the difference between these students.
• neuzhou 6mo ago
  [flagged]
• meidad_g 6mo ago
  [flagged]
• nunez 6mo ago
  > I'm not worried about the machines. The machines are fine. I'm worried about us. We're collectively racing towards the last scene of "Up", justifying each milestone along the way. I've used this analogy more times than I thought I ever would; no regrets. We're entering a post-thought era where knowledge itself is devalued and, worse, _seeking_ that knowledge is discouraged. There's no time to stumble through learning a codebase when Claude can ship features faster than you can think. There's no time to learn. Pixar warned us. Asimov warned us. Orwell warned us. This phenomenon isn't new; we just now have the technology to finally execute on this vision.
  • PaulDavisThe1st 6mo ago
    "Man was hitherto unable to realize his wishes. Now that he can realize them, he must either change them, or perish". -- William Carlos Williams (quoted in Steve Reich's "The Desert Music")
• jeremie_strand 6mo ago
  [dead]
• dejj 6mo ago
  “The gates of hell are open night and day; Smooth the descent, and easy is the way: But to return, and view the cheerful skies, In this the task and mighty labor lies.” The Aenid, Virgil, Dryden translation.
• somesortofthing 6mo ago
  I used to feel this way but... honestly, I've found that pressing on with only a vague understanding of what's happening and then diving deep with the agent's own help if it keeps making bad decisions leads to more output of comparable quality. Even without a deep understanding of the topic, you can usually tell when the LLM is BSing and you need to intervene. The model has much more knowledge "present-at-hand" than it'll actually apply to a given implementation, so you can substantially deepen your understanding with minimal reference to external resources by just taking a break from implementation to have a convo with it. I'm sure this approach breaks down at the very frontiers of highly technical fields but... virtually all work, even work by educated professionals, happens outside that area anyway. On well-trodden ground, you can improve at supervising agents by doing things that test your ability to supervise agents.
  • dummydummy1234 6mo ago
    I think there is a difference between things like coding where it is semi closed loop, at the end of the day the software works or not. Vs fields where there is not a reliable feedback path, or that feedback path is much more noisy.
    • somesortofthing 6mo ago
      There definitely is but even then, you can get a feel for a loop for more open-ended tasks too - you move forward until the model output starts to look handwavy/contradictory, then pause to talk to it/consult outside sources to improve your own knowledge. Most "fuzzy" fields also have quantitative components, and it's often worth stopping for a moment to put together some kind of quantitative evaluation suie to give the model grounding. When you've learned the right path yourself, you start moving forward again. It's for sure slower and more error-prone if you were already an expert when you started, but it's workable, and head-and-shoulders better than what you could do without the AI.
• YeGoblynQueenne 6mo ago
  >> The supervisor still needs to know what the answer should look like, still needs to know which checks to demand, still needs to have the instinct that something is off before they can articulate why. That instinct doesn't come from a subscription. It comes from years of failing at exactly the kind of work that people keep calling grunt work. i.e. science.
• toyetic 6mo ago
  I’m sure there’s some middle ground, like presumably Alice could’ve used AI in a way that gave her the same outcome but quicker. Either way I’m not sure what’s wrong with Bobs approach ( or rather why anyone other than Bob should care)? Either he’s doing something that prevents him from learning and growing in a relevant way or he’s not and can continue this method for the rest of his career. Either way that has no effect on Alice’s career.
• janalsncm 6mo ago
  > There's a common rebuttal to this, and I hear it constantly. "Just wait," people say. "In a few months, in a year, the models will be better. They won't hallucinate. They won't fake plots. The problems you're describing are temporary." To some extent, the reason models will get better is because companies will hire PhDs to train them on increasingly complex problems. The problem is that more complex problems take longer to train, more time to test, require more compute, and are harder to verify. This is why “just make it bigger” is a losing proposition imo. A lot of what I just said is also true for RLVR.
• oars 6mo ago
  Excellent article. It'll be valuable to refer back to this in 5-10 years. Archived here so we can always come back to this: https://archive.md/MZtsI https://archive.md/MZtsI
  • zaikunzhang 6mo ago
    The HN discussion as of this second: https://archive.ph/Fq4u8 https://archive.ph/Fq4u8
• RyujiYasukochi 6mo ago
  [flagged]
• Terr_ 6mo ago
  > Centuries of pedagogy, defeated by a chat window. To be fair to pedagogy, it's also being defeated by eons of human brain wiring that says: "People in stories are somewhat real, and the author is always real." We are disarmed by how well these things mimic all of typical indicators of thought, and it takes work— work our brains can't really sustain—to not fall into assumptions that there's another humanoid bearing some of the responsibility for catching problems.
• darkstarsys 6mo ago
  I agree with this article, and I think it goes further than astrophysics or even physics. As agentic LLMs are starting to prove long-standing open mathematical conjectures and even invent new ones, I fear we may reach a point in mathematical research (which, as Hogg's article describes astrophysics, has no "right edge") where the machines are just better than us. At that point do we just mostly lose interest? You can see it already in the Go-playing community. Why study for years to be a "pretty good" 9P when machines will always play better? What PhD student will spend years on a hard, interesting, deep math problem? Sure, a few will. Some people become monks too. But not many.
• cush 6mo ago
  They forgot the bit where Alice could barely make rent and ate trash for a year while she worked 70hr weeks for a poverty-line stipend. Bob had great sleeps and could hold a part time job waiting tables.
• iainctduncan 6mo ago
  It is astonishing to me how many people here are defending "not actually knowing what the hell you are doing", on the basis that LLMs will "keep getting better". The bit they are missing, IMHO, is that if LLMs keep getting better, doing the steering-the-LLM version keeps getting easier, and being an LLM-using-expert rapidly drops to a value of zero. Like literally fucking nothing. If anyone can do it easily with an LLM, why would anyone pay you anything? Why would they care? You might as well be a teenager at a fast food job. In this scenario, only actually understanding will be any kind of differentiator at all. And if that scenario doesn't come to pass, it will still be a far better differentiator to have a clue.
• danecjensen 6mo ago
  When did humans ever understand what they were doing.
• cadamsdotcom 6mo ago
  > Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expectations. Then Schwartz read it, and it was wrong. Claude had been adjusting parameters to make plots match instead of finding actual errors. It faked results. It invented coefficients. It produced verification documents that verified nothing. It asserted results without derivation. It simplified formulas based on patterns from other problems instead of working through the specifics of the problem at hand. This is solvable with harness engineering. The model’s first try is never ready for human consumption. There needs to be automation (bespoke, a mix of code and prompt based hooks - which agents can build) to force the agent’s output back through itself to tell it to be more rigorous, search online for proof of its claims, etc etc. and not stop until every claim is verifiable. No human should see the model’s output until it’s met these (again bespoke but not hand written) guardrails. What I’m talking about doesn’t exist and really has no analogy yet, so you can think of it as a super advanced form of linting. It’s grounding, but also verification that the grounding links to the material, and refusal to accept the model’s work until it meets the bar. We are asking models to dream (invent purely from their weights), and are surprised when their dreams, just like ours, have little relationship to reality. The current state of the art is going to look very naive in a few years’ time.
• boxomcfoxo 6mo ago
  This essay was extruded in its entirety from Claude. https://boxobarks.leaflet.pub/3mj42airv3s2o#fingerprints-of-the-beige-liebox https://boxobarks.leaflet.pub/3mj42airv3s2o#fingerprints-of-...