11 ms·

I’ve only skimmed the paper - a long and dense read - but it’s already clear it’ll become a classic. What’s fascinating is that engineering is transforming into

I’ve only skimmed the paper - a long and dense read - but it’s already clear it’ll become a classic. What’s fascinating is that engineering is transforming into a science, trying to understand precisely how its own creations work This shift is more profound than many realize. Engineering traditionally applied our understanding of the physical world, mathematics, and logic to build predictable things. But now, especially in fields like AI, we’ve built systems so complex we no longer fully understand them. We must now use scientific methods - originally designed to understand nature - to comprehend our own engineered creations. Mindblowing.

ctoth 2y ago
This "practice-first, theory-later" pattern has been the norm rather than the exception. The steam engine predated thermodynamics. People bred plants and animals for thousands of years before Darwin or Mendel. The few "top-down" examples where theory preceded application (like nuclear energy or certain modern pharmaceuticals) are relatively recent historical anomalies.
- marcelsalathe 2y ago
  I see your point, but something still seems different. Yes we bred plants and animals, but we did not create them. Yes we did build steam engines before understanding thermodynamics but we still understood what they did (heat, pressure, movement, etc.) Fun fact: we have no clue how most drugs works. Or, more precisely, we know a few aspects, but are only scratching the surface. We're even still discovering news things about Aspirin, one of the oldest drugs: https://www.nature.com/articles/s41586-025-08626-7 https://www.nature.com/articles/s41586-025-08626-7
  - tmp10423288442 2y ago
    > Yes we did build steam engines before understanding thermodynamics but we still understood what it did (heat, pressure, movement, etc.) We only understood in the broadest sense. It took a long process of iteration before we could create steam engines that were efficient enough to start an Industrial Revolution. At the beginning they were so inefficient that they could only pump water from the same coal mine they got their fuel from, and subject to frequent boiler explosions besides.
  - mystified5016 2y ago
    We laid transatlantic telegraph wires before we even had a hint of the physics involved. It create the entire field of transmission and signal theory. Shannon had to invent new physics to explain why the cables didn't work as expected.
    - anthk 2y ago
      THe telegraph it's older than radio. Think about it.
    - pas 2y ago
      I think that's misleading. There was a lot of physics already known, importance of insulation and cross-section, signal attenuation was also known. The future Lord Kelvin conducted experiments. The two scientific advisors had a conflict. And the "CEO" went with the cheaper option. """ Thomson believed that Whitehouse's measurements were flawed and that underground and underwater cables were not fully comparable. Thomson believed that a larger cable was needed to mitigate the retardation problem. In mid-1857, on his own initiative, he examined samples of copper core of allegedly identical specification and found variations in resistance up to a factor of two. But cable manufacture was already underway, and Whitehouse supported use of a thinner cable, so Field went with the cheaper option. """
    - cft 2y ago
      that was 1854. You basically only needed Ohm's law for that, which was discovered in 1827
      - JPLeRouzic 2y ago
        Ohm's law for a cable 4000 km/3000 miles long? That implies transmission was instantaneous and without any alteration in shape. I guess the rise time was tens of milliseconds and rebounds in signals lasted for milliseconds or more. Hardly something you can neglect. For reference, in my time (the 1980) in the telecom industry, we had to regenerate digital signals every 2km.
        2y ago
  - adastra22 2y ago
    We don’t create LLMs either. We evolve/train them. I think the comparison is closer than you think.
    - no_wizard 2y ago
      We most definitely create them though, there is an entire A -> B follow you can do. It’s complicated but they are most definitely created.
    - homeyKrogerSage 2y ago
      Dawg
  - pclmulqdq 2y ago
    Most of what we refer to as "engineering" involves using principles that flow down from science to do stuff. The return to the historic norm is sort of a return to the "useful arts" or some other idea.
  - arijo 2y ago
    Almost all civil, chemical, electrical, etc., engineering emerged from a practice-first, theory-later evolution.
- karparov 2y ago
  It's been there in programming from essentially the first day too. People skip the theory and just get hacking. Otherwise we'd all be writing Haskell now. Or rather we'd not be writing anything since a real compiler would still have been to hacky and not theoretically correct. I'm writing this with both a deep admiration as well as practical repulsion of C.S. theory.
- ants_everywhere 2y ago
  This isn't quite true, although it's commonly said. For steam engines, the first commercial ones came after and were based on scientific advancements that made them possible. One built in 1679 was made by an associate of Boyle, who discovered Boyle's law. These early steam engines co-evolved with thermodynamics. The engines improved and hit a barrier, at which point Carnot did his famous work. This is putting aside steam engines that are mostly curiosities like ones built in the ancient world. See, for example - https://en.wikipedia.org/wiki/Thermodynamics#History https://en.wikipedia.org/wiki/Thermodynamics#History - https://en.wikipedia.org/wiki/Steam_engine#History https://en.wikipedia.org/wiki/Steam_engine#History
- cryptonector 2y ago
  Canons and archery and catapults predated Newtonian classical mechanics.
latemedium 2y ago
I'm reminded of the metaphor that these models aren't constructed, they're "grown". It rings true in many ways - and in this context they're like organisms that must be studied using traditional scientific techniques that are more akin to biology than engineering.
- dartos 2y ago
  Sort of. We don’t precisely know the most fundamental workings of a living cell. Our understanding of the fundamental physics of the universe has some hold. But for LLMs and statistical models in general, we do know precisely what the fundamental pieces do. We know what processor instructions are being executed. We could, given enough research, have absolutely perfect understanding of what is happening in a given model and why. Idk if we’ll be able to do that in the physical sciences.
  - wrs 2y ago
    Having spent some time working with both molecular biologists and LLM folks, I think it's pretty good analogy. We know enough quantum mechanics to simulate the fundamental workings of a cell pretty well, but that's not a route to understanding. To explain anything, we need to move up an abstraction hierarchy to peptides, enzymes, receptors, etc. But note that we invented those categories in the first place -- nature doesn't divide up functionality into neat hierarchies like human designers do. So all these abstractions are leaky and incomplete. Molecular biologists are constantly discovering mechanisms that require breaking the current abstractions to explain. Similarly, we understand floating point multiplication perfectly, but when we let 100 billion parameters set themselves through an opaque training process, we don't have good abstractions to use to understand what's going on in that set of weights. We don't have even the rough equivalent of the peptides or enzymes level yet. So this paper is progress toward that goal.
kazinator 2y ago
We've already built things in computing that we don't easily understand, even outside of AI, like large distributed systems and all sorts of balls of mud. Within the sphere of AI, we have built machines which can play strategy games like chess, and surprise us with an unforseen defeat. It's not necessarily easy to see how that emerged from the individual rules. Even a compiler can surprise you. You code up some optimizations, which are logically separate, but then a combination of them does something startling. Basically, in mathematics, you cannot grasp all the details of a vast space just from knowing the axioms which generate it and a few things which follow from them. Elementary school children know what is a prime number, yet those things occupy mathematicians who find new surprises in that space.
- TeMPOraL 2y ago
  Right, but this is somewhat different, in that we apply a simple learning method to a big dataset, and the resulting big matrix of numbers suddenly can answer question and write anything - prose, poetry, code - better than most humans - and we don't know how it does it. What we do know[0] is, there's a structure there - structure reflecting a kind of understanding of languages and the world. I don't think we've ever created anything this complex before, completely on our own. Of course, learning method being conceptually simple, all that structure must come from the data. Which is also profound, because that structure is a first fully general world/conceptual model that we can actually inspect and study up close - the other one being animal and human brains, which are much harder to figure out. > Basically, in mathematics, you cannot grasp all the details of a vast space just from knowing the axioms which generate it and a few things which follow from them. Elementary school children know what is a prime number, yet those things occupy mathematicians who find new surprises in that space. Prime numbers and fractals and other mathematical objects have plenty of fascinating mysteries and complex structures forming though them, but so far none of those can casually pass Turing test and do half of my job for me, and millions other people. -- [0] - Even as many people still deny this, and talk about LLMs as mere "stochastic parrots" and "next token predictors" that couldn't possibly learn anything at all.
  - karparov 2y ago
nthingtohide 2y ago
> we’ve built systems so complex we no longer fully understand them. I see three systems which share the blackhole horizon problem. We don't know what happens behind the blackhole horizon. We don't know what happens at the exact moment of particle collisions. We don't know what is going inside AI's working mechanisms.
- jeremyjh 2y ago
  I don't think these things are equivalent at all. We don't understand AI models in much the same way that we don't understand the human brain; but just as decades of different approaches (physical studies, behavior studies) have shed a lot of light on brain function, we can do the same with an AI model and eventually understand it (perhaps, several decades after it is obsolete).
  - nthingtohide 2y ago
    Yes, but our methods of understanding either brain or particle collisions is still outside in. We figure out the functional mapping between input and output. We don't know these systems inside out. E.g. in particle collisions (scattering amplitude calculations), are the particle actually performing the Feynman diagrams summmation? PS: I mentioned in another comment that AI can pretend to be strategically jailbroken to achieve its objectives. One way to counter this is to have N copies of the same model running and take Majority voting of the output.
creer 2y ago
That seems pretty acceptable: there is a phase of new technologies where applications can be churned out and improved readily enough, without much understanding of the process. Then it's fair that efforts at understanding may not be economically justified (or even justified by academic papers rewards). The same budget or effort can simply be poured into the next version - with enough progress to show for it. Understanding becomes necessary only much later, when the pace of progress shows signs of slowing.
stronglikedan 2y ago
We've abstracted ourselves into abstraction.
- deleted 2y ago
  [deleted]
auggierose 2y ago
It's what mathematicians have been doing since forever. We use scientific methods to understand our own creations / discoveries. What is happening is that everything is becoming math. That's all.
- ranit 2y ago
  Relevant: https://news.ycombinator.com/item?id=43344703 https://news.ycombinator.com/item?id=43344703
- karparov 2y ago
  It's the exact opposite of math. Math postulates a bunch of axioms and then studies what follows from them. Natural science observes the world and tries to retroactively discover what laws could describe what we're seeing. In math, the laws come first, the behavior follows from the laws. The laws are the ground truth. In science, nature is the ground truth. The laws have to follow nature and are adjusted upon a mismatch. (If there is a mismatch in math then you've made a mistake.)
  - auggierose 2y ago
    No, the ground truth in math is nature as well. Which axioms are interesting? And why? That is nature. Yes, proof from axioms is a cornerstone of math, but there are all sorts of axioms you could assume, and all sorts of proofs to do from them, but we don't care about most of them. Math is about the discovery of the right axioms, and proof helps in establishing that these are indeed the right axioms.
    - lioeters 2y ago
      > the ground truth in math is nature Who was it that said, "Mathematics is an experimental science." > In his 1900 lectures, "Methods of Mathematical Physics," (posthumously published in 1935) Henri Poincaré argued that mathematicians weren't just constructing abstract systems; they were actively testing hypotheses and theories against observations and experimental data, much like physicists were doing at the time. Whether to call it nature or reality, I think both science and mathematics are in pursuit of truth, whose ground is existence itself. The laws and theories are descriptions and attempts to understand that what is. They're developed, rewritten, and refined based on how closely they approach our observations and experience of it.
georgewsinger 2y ago
This is such an insightful comment. Now that I see it, I can't see unsee it.
0xbadcafebee 2y ago
Engineering started out as just some dudes who built things from gut feeling. After a whole lot of people died from poorly built things, they decided to figure out how to know ahead of time if it would kill people or not. They had to use math and science to figure that part out. Funny enough that happened with software too. People just build shit without any method to prove that it will not fall down / crash. They throw some code together, poke at it until it does something they wanted, and call that "stable". There is no science involved. There's some mathy bits called "computer science" / "software algorithms", but most software is not a math problem. Software engineering should really be called "Software Craftsmanship". We haven't achieved real engineering with software yet.
- slfnflctd 2y ago
  You have a point, but it is also true that some software is far more rigorously tested than other software. There are categories where it absolutely is both scientific and real engineering. I fully agree that the vast majority is not, though.
- AdieuToLogic 2y ago
  This is such an unbelievably dismissive assertion, I don't even know where to start. To suggest, nay, explicitly state: Engineering started out as just some dudes who built things from gut feeling. After a whole lot of people died from poorly built things, they decided to figure out how to know ahead of time if it would kill people or not. Is to demean those who made modern life possible. Say what you want about software developers and I would likely agree with much of the criticism. Not so the premise set forth above regarding engineering professions in general.
  - 0xbadcafebee 2y ago
    Surely you already know the history of professional engineers, then? How it's only a little over 118 years old? Mostly originating from the fact that it was charlatans claiming to be engineers, building things that ended up killing people, that inspired the need for a professional license? "The people who made modern life possible" were not professional engineers, often barely amateurs. Artistocrat polymaths who delved into cutting edge philosophy. Blacksmith craftsmen developing new engines by trial and error. A new englander who failed to study law at Yale, landed in the American South, and developed a modification of an Indian device for separating seed from cotton plants. In the literal historical sense, "engineering" was just the building of cannons in the 14th century. Since thousands of years before, up until now, there has always been a combination of the practice of building things with some kind of "science" (which itself didn't exist until a few hundred years ago) to try to estimate the result of an expensive, dangerous project. But these are not the people who made modern life people. Lots, and lots, and lots of people made modern life possible. Not just builders and mathematicians. Receptionists. Interns. Factory workers. Farmers. Bankers. Sailors. Welders. Soldiers. So many professions, and people, whose backs and spirits were bent or broken, to give us the world we have today. Engineers don't deserve any more credit than anyone else - especially considering how much was built before their professions were even established. Science is a process, and math is a tool, that is very useful, and even critical. But without the rest it's just numbers on paper.
tim333 2y ago
I imagine this kind of thing well help understand how human brains work, especially as AI gets better and more human like.
nashashmi 2y ago
You seem to be glorifying humanity’s failure to make good products and instead making products that just work well enough to pass through the gate. We have always been making products that were too difficult to understand by pencil and paper. So we invented debug tools. And then we made systems that were too big to understand so we made trace routes. And now we have products that are too statistically large to understand, so we are inventing … whatever this is.
anal_reactor 2y ago
It is absolutely incredible that we happened to live exactly in the times when the humanity is teaching a machine to actually think. As in, not in some metaphorical sense, but in the common, intuitive sense. Whether we're there yet or not is up to discussion, but it's clear to me that within 10 years maximum we'll have created programs that truly think and are aware. At the same time, I just can't bring myself to be interested in the topic. I don't feel excitement. I feel... indifference? Fear? Maybe the technology became so advanced that for normal people like myself it's indistinguishable from magic, and there's no point trying to comprehend it, just avoid it and pray it's not used against you. Or maybe I'm just getting old, and I'm experiencing what my mother experienced when she refused to learn how to use MS Office.
- hn_acc1 2y ago
  Yeah.. It's just not something that really excites me as a computer geek of 40+ years who started in the 80s with a 300 baud modem. Still working as a coder in my 50s, and while I'm solving interesting problems, etc.. almost every technology these days seems to be focused on advertising, scraping / stealing other's data and repackaging it, etc. And I am using AI coding assistants, because, well, I have to to stay competitive. And these technologies come with a side helping of a large chance to REALLY mess up someone's life - who is going to argue with the database and WIN if it says you don't exist in this day and age? And that database is (databases are) currently under the control of incredibly petty sociopaths..
Barrin92 2y ago
"we’ve built systems so complex we no longer fully understand them. We must now use scientific methods - originally designed to understand nature - to comprehend our own engineered creations." Ted Chiang saw that one coming: https://www.nature.com/articles/35014679 https://www.nature.com/articles/35014679
samstave 2y ago
[dead]
cuttothechase 2y ago
This is definitely a classic for story telling but it appears to be nothing more than hand wavy. Its a bit like there is the great and powerful man behind the curtain, lets trace the thought of this immaculate being you mere mortals. Anthropomorphing seems to be in an overdose mode with "thinking / thoughts", "mind" etc., scattered everywhere. Nothing with any of the LLMs outputs so far suggests that there is anything even close enough to a mind or a thought or anything really outside of vanity. Being wistful with good story telling does go a long way in the world of story telling but in actually understanding the science, I wouldn't hold my breath.
- colah3 2y ago
  Thanks for the feedback! I'm one of the authors. I just wanted to make sure you noticed that this is linking to an accessible blog post that's trying to communicate a research result to a non-technical audience? The actual research result is covered in two papers which you can find here: - Methods paper: https://transformer-circuits.pub/2025/attribution-graphs/methods.html https://transformer-circuits.pub/2025/attribution-graphs/met... - Paper applying this method to case studies in Claude 3.5 Haiku: https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio... These papers are jointly 150 pages and are quite technically dense, so it's very understandable that most commenters here are focusing on the non-technical blog post. But I just wanted to make sure that you were aware of the papers, given your feedback.
  - hustwindmaple1 2y ago
    Really appreciate your team's enormous efforts in this direction, not only the cutting edge research (which I don't see OAI/DeepMind publishing any paper on) but aslo making the content more digestible for non-research audience. Please keep up the great work!
  - AdieuToLogic 2y ago
    The post to which you replied states: Anthropomorphing[sic] seems to be in an overdose mode with "thinking / thoughts", "mind" etc., scattered everywhere. Nothing with any of the LLMs outputs so far suggests that there is anything even close enough to a mind or a thought or anything really outside of vanity. This is supported by reasonable interpretation of the cited article. Considering the two following statements made in the reply: I'm one of the authors. And These papers are jointly 150 pages and are quite technically dense, so it's very understandable that most commenters here are focusing on the non-technical blog post. The onus of clarifying the article's assertions: Knowing how models like Claude *think* ... And Claude sometimes thinks in a conceptual space that is shared between languages, suggesting it has a kind of universal “language of thought.” As it pertains to anthropomorphizing an algorithm (a.k.a. stating it "thinks") is on the author(s).
frontfor 2y ago
I don’t think this is as profound as you made out to be. Most complex systems are incomprehensible to the majority of population anyway, so from a practical standpoint AI is no different. There’s also no single theory for how the financial markets work and yet market participants trade and make money nonetheless. And yes, we created the markets.
chpatrick 2y ago
I would say we engineered the system that trained them but we never really understood the data (human thinking).
dukeofdoom 2y ago
Not that I disagree with you. But Humans have a tendency to do things beyond their comprehension often. I take it you've never been fishing before and tied your line in a knot.
trhway 2y ago
> to comprehend our own engineered creations. The comprehend part may never happen. At least by our own mind. We’ll sooner build the mind which is going to do that comprehension: “To scale to the thousands of words supporting the complex thinking chains used by modern models, we will need to improve both the method and (perhaps with AI assistance) how we make sense of what we see with it” Yes, that AI assistance, meta self reflection, is going to probably be a way if not right to the AGI, at least very significant step toward it.
BOOSTERHIDROGEN 2y ago
If only this profound mechanism can be easily testable for social interaction.
MathMonkeyMan 2y ago
In a sense this has been true of conventional programs for a while now. Gerald Sussman discusses the idea when talking about why MIT switched their introductory programming course from Scheme to Python: <https://youtu.be/OgRFOjVzvm0?t=239 https://youtu.be/OgRFOjVzvm0?t=239>.
EGreg 2y ago
I think it’s pretty obvious what these models do in some cases. Try asking them to write a summary at the beginning of their answer. The summary is basically them trying to make something plausible-sounding but they aren’t actually going back and summarizing. LLMs are basically a building block in a larger software. Just like any library or framework. You shouldn’t expect them to be a hammer for every nail. But they can now enable so many different applications, including natural language interfaces, better translations and so forth. But then you’re supposed to have them output JSON to be used in building artifacts like Powerpoints. Has anyone implemented that yet?
hansmayer 2y ago
If you don't mind - based on what will this "paper" become a classic? Was it published in a well known scientific magazine, after undergoing a stringent peer-review process, because it is setting up and proving a new scientific hypothesis? Because this is what scientific papers look like. I struggle to identify any of those characteristics, except for being dense and hard to read, but that's more of a correlation, isn't it?
rcxdude 2y ago
That's basically how engineering works if you're doing anything at all novel: you will have some theory which informs your design, then you build it, then you test it and basically need to do science to figure out how it's perfoming, and most likely, why it's not working properly, and then iterate. I do engineering, but doing science has been a core part of almost every project I've worked on. (heck, even debugging code is basically science). There's just different degrees in different projects as to how much you understand about how the system you're designing actually works, and ML is an area where there's an unusual ratio of visibility (you can see all of the weights and calculations in the network precisely) to understanding (i.e. there's relatively little in terms of mathematical theory that precisely describe how a model trains and operates, just a bunch of approximations which can be somewhat justified, which is where a lot of engineering work sits)
madethisnow 2y ago
psychology
mdnahas 2y ago
I like your definitions! My personal definition of science is learning rules that predict the future, given the present state. And my definition of engineering is arranging the present state to control the future. I don’t think it’s unusual for engineering creations to need new science to understand them. When metal parts broke, humans studied metallurgy. When engines exploded, we studied the remains. With that science, we could engineer larger, longer lasting, more powerful devices. Now, we’re finding flaws in AI and diagnosing their causes. And soon able to build better ones.

ctoth 2y ago

This "practice-first, theory-later" pattern has been the norm rather than the exception. The steam engine predated thermodynamics. People bred plants and animals for thousands of years before Darwin or Mendel. The few "top-down" examples where theory preceded application (like nuclear energy or certain modern pharmaceuticals) are relatively recent historical anomalies.

marcelsalathe 2y ago
I see your point, but something still seems different. Yes we bred plants and animals, but we did not create them. Yes we did build steam engines before understanding thermodynamics but we still understood what they did (heat, pressure, movement, etc.) Fun fact: we have no clue how most drugs works. Or, more precisely, we know a few aspects, but are only scratching the surface. We're even still discovering news things about Aspirin, one of the oldest drugs: https://www.nature.com/articles/s41586-025-08626-7 https://www.nature.com/articles/s41586-025-08626-7
- tmp10423288442 2y ago
  > Yes we did build steam engines before understanding thermodynamics but we still understood what it did (heat, pressure, movement, etc.) We only understood in the broadest sense. It took a long process of iteration before we could create steam engines that were efficient enough to start an Industrial Revolution. At the beginning they were so inefficient that they could only pump water from the same coal mine they got their fuel from, and subject to frequent boiler explosions besides.
- mystified5016 2y ago
  We laid transatlantic telegraph wires before we even had a hint of the physics involved. It create the entire field of transmission and signal theory. Shannon had to invent new physics to explain why the cables didn't work as expected.
  - anthk 2y ago
    THe telegraph it's older than radio. Think about it.
  - pas 2y ago
    I think that's misleading. There was a lot of physics already known, importance of insulation and cross-section, signal attenuation was also known. The future Lord Kelvin conducted experiments. The two scientific advisors had a conflict. And the "CEO" went with the cheaper option. """ Thomson believed that Whitehouse's measurements were flawed and that underground and underwater cables were not fully comparable. Thomson believed that a larger cable was needed to mitigate the retardation problem. In mid-1857, on his own initiative, he examined samples of copper core of allegedly identical specification and found variations in resistance up to a factor of two. But cable manufacture was already underway, and Whitehouse supported use of a thinner cable, so Field went with the cheaper option. """
  - cft 2y ago
    that was 1854. You basically only needed Ohm's law for that, which was discovered in 1827
    - JPLeRouzic 2y ago
      Ohm's law for a cable 4000 km/3000 miles long? That implies transmission was instantaneous and without any alteration in shape. I guess the rise time was tens of milliseconds and rebounds in signals lasted for milliseconds or more. Hardly something you can neglect. For reference, in my time (the 1980) in the telecom industry, we had to regenerate digital signals every 2km.
      - 2y ago
- adastra22 2y ago
  We don’t create LLMs either. We evolve/train them. I think the comparison is closer than you think.
  - no_wizard 2y ago
    We most definitely create them though, there is an entire A -> B follow you can do. It’s complicated but they are most definitely created.
  - homeyKrogerSage 2y ago
    Dawg
- pclmulqdq 2y ago
  Most of what we refer to as "engineering" involves using principles that flow down from science to do stuff. The return to the historic norm is sort of a return to the "useful arts" or some other idea.
- arijo 2y ago
  Almost all civil, chemical, electrical, etc., engineering emerged from a practice-first, theory-later evolution.
karparov 2y ago
It's been there in programming from essentially the first day too. People skip the theory and just get hacking. Otherwise we'd all be writing Haskell now. Or rather we'd not be writing anything since a real compiler would still have been to hacky and not theoretically correct. I'm writing this with both a deep admiration as well as practical repulsion of C.S. theory.
ants_everywhere 2y ago
This isn't quite true, although it's commonly said. For steam engines, the first commercial ones came after and were based on scientific advancements that made them possible. One built in 1679 was made by an associate of Boyle, who discovered Boyle's law. These early steam engines co-evolved with thermodynamics. The engines improved and hit a barrier, at which point Carnot did his famous work. This is putting aside steam engines that are mostly curiosities like ones built in the ancient world. See, for example - https://en.wikipedia.org/wiki/Thermodynamics#History https://en.wikipedia.org/wiki/Thermodynamics#History - https://en.wikipedia.org/wiki/Steam_engine#History https://en.wikipedia.org/wiki/Steam_engine#History
cryptonector 2y ago
Canons and archery and catapults predated Newtonian classical mechanics.

latemedium 2y ago

I'm reminded of the metaphor that these models aren't constructed, they're "grown". It rings true in many ways - and in this context they're like organisms that must be studied using traditional scientific techniques that are more akin to biology than engineering.

dartos 2y ago
Sort of. We don’t precisely know the most fundamental workings of a living cell. Our understanding of the fundamental physics of the universe has some hold. But for LLMs and statistical models in general, we do know precisely what the fundamental pieces do. We know what processor instructions are being executed. We could, given enough research, have absolutely perfect understanding of what is happening in a given model and why. Idk if we’ll be able to do that in the physical sciences.
- wrs 2y ago
  Having spent some time working with both molecular biologists and LLM folks, I think it's pretty good analogy. We know enough quantum mechanics to simulate the fundamental workings of a cell pretty well, but that's not a route to understanding. To explain anything, we need to move up an abstraction hierarchy to peptides, enzymes, receptors, etc. But note that we invented those categories in the first place -- nature doesn't divide up functionality into neat hierarchies like human designers do. So all these abstractions are leaky and incomplete. Molecular biologists are constantly discovering mechanisms that require breaking the current abstractions to explain. Similarly, we understand floating point multiplication perfectly, but when we let 100 billion parameters set themselves through an opaque training process, we don't have good abstractions to use to understand what's going on in that set of weights. We don't have even the rough equivalent of the peptides or enzymes level yet. So this paper is progress toward that goal.

kazinator 2y ago

We've already built things in computing that we don't easily understand, even outside of AI, like large distributed systems and all sorts of balls of mud. Within the sphere of AI, we have built machines which can play strategy games like chess, and surprise us with an unforseen defeat. It's not necessarily easy to see how that emerged from the individual rules. Even a compiler can surprise you. You code up some optimizations, which are logically separate, but then a combination of them does something startling. Basically, in mathematics, you cannot grasp all the details of a vast space just from knowing the axioms which generate it and a few things which follow from them. Elementary school children know what is a prime number, yet those things occupy mathematicians who find new surprises in that space.

TeMPOraL 2y ago
Right, but this is somewhat different, in that we apply a simple learning method to a big dataset, and the resulting big matrix of numbers suddenly can answer question and write anything - prose, poetry, code - better than most humans - and we don't know how it does it. What we do know[0] is, there's a structure there - structure reflecting a kind of understanding of languages and the world. I don't think we've ever created anything this complex before, completely on our own. Of course, learning method being conceptually simple, all that structure must come from the data. Which is also profound, because that structure is a first fully general world/conceptual model that we can actually inspect and study up close - the other one being animal and human brains, which are much harder to figure out. > Basically, in mathematics, you cannot grasp all the details of a vast space just from knowing the axioms which generate it and a few things which follow from them. Elementary school children know what is a prime number, yet those things occupy mathematicians who find new surprises in that space. Prime numbers and fractals and other mathematical objects have plenty of fascinating mysteries and complex structures forming though them, but so far none of those can casually pass Turing test and do half of my job for me, and millions other people. -- [0] - Even as many people still deny this, and talk about LLMs as mere "stochastic parrots" and "next token predictors" that couldn't possibly learn anything at all.
- karparov 2y ago

nthingtohide 2y ago

> we’ve built systems so complex we no longer fully understand them. I see three systems which share the blackhole horizon problem. We don't know what happens behind the blackhole horizon. We don't know what happens at the exact moment of particle collisions. We don't know what is going inside AI's working mechanisms.

jeremyjh 2y ago
I don't think these things are equivalent at all. We don't understand AI models in much the same way that we don't understand the human brain; but just as decades of different approaches (physical studies, behavior studies) have shed a lot of light on brain function, we can do the same with an AI model and eventually understand it (perhaps, several decades after it is obsolete).
- nthingtohide 2y ago
  Yes, but our methods of understanding either brain or particle collisions is still outside in. We figure out the functional mapping between input and output. We don't know these systems inside out. E.g. in particle collisions (scattering amplitude calculations), are the particle actually performing the Feynman diagrams summmation? PS: I mentioned in another comment that AI can pretend to be strategically jailbroken to achieve its objectives. One way to counter this is to have N copies of the same model running and take Majority voting of the output.

creer 2y ago

That seems pretty acceptable: there is a phase of new technologies where applications can be churned out and improved readily enough, without much understanding of the process. Then it's fair that efforts at understanding may not be economically justified (or even justified by academic papers rewards). The same budget or effort can simply be poured into the next version - with enough progress to show for it. Understanding becomes necessary only much later, when the pace of progress shows signs of slowing.

stronglikedan 2y ago

We've abstracted ourselves into abstraction.

deleted 2y ago
[deleted]

auggierose 2y ago

It's what mathematicians have been doing since forever. We use scientific methods to understand our own creations / discoveries. What is happening is that everything is becoming math. That's all.

ranit 2y ago
Relevant: https://news.ycombinator.com/item?id=43344703 https://news.ycombinator.com/item?id=43344703
karparov 2y ago
It's the exact opposite of math. Math postulates a bunch of axioms and then studies what follows from them. Natural science observes the world and tries to retroactively discover what laws could describe what we're seeing. In math, the laws come first, the behavior follows from the laws. The laws are the ground truth. In science, nature is the ground truth. The laws have to follow nature and are adjusted upon a mismatch. (If there is a mismatch in math then you've made a mistake.)
- auggierose 2y ago
  No, the ground truth in math is nature as well. Which axioms are interesting? And why? That is nature. Yes, proof from axioms is a cornerstone of math, but there are all sorts of axioms you could assume, and all sorts of proofs to do from them, but we don't care about most of them. Math is about the discovery of the right axioms, and proof helps in establishing that these are indeed the right axioms.
  - lioeters 2y ago
    > the ground truth in math is nature Who was it that said, "Mathematics is an experimental science." > In his 1900 lectures, "Methods of Mathematical Physics," (posthumously published in 1935) Henri Poincaré argued that mathematicians weren't just constructing abstract systems; they were actively testing hypotheses and theories against observations and experimental data, much like physicists were doing at the time. Whether to call it nature or reality, I think both science and mathematics are in pursuit of truth, whose ground is existence itself. The laws and theories are descriptions and attempts to understand that what is. They're developed, rewritten, and refined based on how closely they approach our observations and experience of it.

georgewsinger 2y ago

This is such an insightful comment. Now that I see it, I can't see unsee it.

0xbadcafebee 2y ago

Engineering started out as just some dudes who built things from gut feeling. After a whole lot of people died from poorly built things, they decided to figure out how to know ahead of time if it would kill people or not. They had to use math and science to figure that part out. Funny enough that happened with software too. People just build shit without any method to prove that it will not fall down / crash. They throw some code together, poke at it until it does something they wanted, and call that "stable". There is no science involved. There's some mathy bits called "computer science" / "software algorithms", but most software is not a math problem. Software engineering should really be called "Software Craftsmanship". We haven't achieved real engineering with software yet.

slfnflctd 2y ago
You have a point, but it is also true that some software is far more rigorously tested than other software. There are categories where it absolutely is both scientific and real engineering. I fully agree that the vast majority is not, though.
AdieuToLogic 2y ago
This is such an unbelievably dismissive assertion, I don't even know where to start. To suggest, nay, explicitly state: Engineering started out as just some dudes who built things from gut feeling. After a whole lot of people died from poorly built things, they decided to figure out how to know ahead of time if it would kill people or not. Is to demean those who made modern life possible. Say what you want about software developers and I would likely agree with much of the criticism. Not so the premise set forth above regarding engineering professions in general.
- 0xbadcafebee 2y ago
  Surely you already know the history of professional engineers, then? How it's only a little over 118 years old? Mostly originating from the fact that it was charlatans claiming to be engineers, building things that ended up killing people, that inspired the need for a professional license? "The people who made modern life possible" were not professional engineers, often barely amateurs. Artistocrat polymaths who delved into cutting edge philosophy. Blacksmith craftsmen developing new engines by trial and error. A new englander who failed to study law at Yale, landed in the American South, and developed a modification of an Indian device for separating seed from cotton plants. In the literal historical sense, "engineering" was just the building of cannons in the 14th century. Since thousands of years before, up until now, there has always been a combination of the practice of building things with some kind of "science" (which itself didn't exist until a few hundred years ago) to try to estimate the result of an expensive, dangerous project. But these are not the people who made modern life people. Lots, and lots, and lots of people made modern life possible. Not just builders and mathematicians. Receptionists. Interns. Factory workers. Farmers. Bankers. Sailors. Welders. Soldiers. So many professions, and people, whose backs and spirits were bent or broken, to give us the world we have today. Engineers don't deserve any more credit than anyone else - especially considering how much was built before their professions were even established. Science is a process, and math is a tool, that is very useful, and even critical. But without the rest it's just numbers on paper.

tim333 2y ago

I imagine this kind of thing well help understand how human brains work, especially as AI gets better and more human like.

nashashmi 2y ago

You seem to be glorifying humanity’s failure to make good products and instead making products that just work well enough to pass through the gate. We have always been making products that were too difficult to understand by pencil and paper. So we invented debug tools. And then we made systems that were too big to understand so we made trace routes. And now we have products that are too statistically large to understand, so we are inventing … whatever this is.

anal_reactor 2y ago

It is absolutely incredible that we happened to live exactly in the times when the humanity is teaching a machine to actually think. As in, not in some metaphorical sense, but in the common, intuitive sense. Whether we're there yet or not is up to discussion, but it's clear to me that within 10 years maximum we'll have created programs that truly think and are aware. At the same time, I just can't bring myself to be interested in the topic. I don't feel excitement. I feel... indifference? Fear? Maybe the technology became so advanced that for normal people like myself it's indistinguishable from magic, and there's no point trying to comprehend it, just avoid it and pray it's not used against you. Or maybe I'm just getting old, and I'm experiencing what my mother experienced when she refused to learn how to use MS Office.

hn_acc1 2y ago
Yeah.. It's just not something that really excites me as a computer geek of 40+ years who started in the 80s with a 300 baud modem. Still working as a coder in my 50s, and while I'm solving interesting problems, etc.. almost every technology these days seems to be focused on advertising, scraping / stealing other's data and repackaging it, etc. And I am using AI coding assistants, because, well, I have to to stay competitive. And these technologies come with a side helping of a large chance to REALLY mess up someone's life - who is going to argue with the database and WIN if it says you don't exist in this day and age? And that database is (databases are) currently under the control of incredibly petty sociopaths..

Barrin92 2y ago

"we’ve built systems so complex we no longer fully understand them. We must now use scientific methods - originally designed to understand nature - to comprehend our own engineered creations." Ted Chiang saw that one coming: https://www.nature.com/articles/35014679 https://www.nature.com/articles/35014679

samstave 2y ago

[dead]

cuttothechase 2y ago

This is definitely a classic for story telling but it appears to be nothing more than hand wavy. Its a bit like there is the great and powerful man behind the curtain, lets trace the thought of this immaculate being you mere mortals. Anthropomorphing seems to be in an overdose mode with "thinking / thoughts", "mind" etc., scattered everywhere. Nothing with any of the LLMs outputs so far suggests that there is anything even close enough to a mind or a thought or anything really outside of vanity. Being wistful with good story telling does go a long way in the world of story telling but in actually understanding the science, I wouldn't hold my breath.

colah3 2y ago
Thanks for the feedback! I'm one of the authors. I just wanted to make sure you noticed that this is linking to an accessible blog post that's trying to communicate a research result to a non-technical audience? The actual research result is covered in two papers which you can find here: - Methods paper: https://transformer-circuits.pub/2025/attribution-graphs/methods.html https://transformer-circuits.pub/2025/attribution-graphs/met... - Paper applying this method to case studies in Claude 3.5 Haiku: https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio... These papers are jointly 150 pages and are quite technically dense, so it's very understandable that most commenters here are focusing on the non-technical blog post. But I just wanted to make sure that you were aware of the papers, given your feedback.
- hustwindmaple1 2y ago
  Really appreciate your team's enormous efforts in this direction, not only the cutting edge research (which I don't see OAI/DeepMind publishing any paper on) but aslo making the content more digestible for non-research audience. Please keep up the great work!
- AdieuToLogic 2y ago
  The post to which you replied states: Anthropomorphing[sic] seems to be in an overdose mode with "thinking / thoughts", "mind" etc., scattered everywhere. Nothing with any of the LLMs outputs so far suggests that there is anything even close enough to a mind or a thought or anything really outside of vanity. This is supported by reasonable interpretation of the cited article. Considering the two following statements made in the reply: I'm one of the authors. And These papers are jointly 150 pages and are quite technically dense, so it's very understandable that most commenters here are focusing on the non-technical blog post. The onus of clarifying the article's assertions: Knowing how models like Claude *think* ... And Claude sometimes thinks in a conceptual space that is shared between languages, suggesting it has a kind of universal “language of thought.” As it pertains to anthropomorphizing an algorithm (a.k.a. stating it "thinks") is on the author(s).

frontfor 2y ago

I don’t think this is as profound as you made out to be. Most complex systems are incomprehensible to the majority of population anyway, so from a practical standpoint AI is no different. There’s also no single theory for how the financial markets work and yet market participants trade and make money nonetheless. And yes, we created the markets.

chpatrick 2y ago

I would say we engineered the system that trained them but we never really understood the data (human thinking).

dukeofdoom 2y ago

Not that I disagree with you. But Humans have a tendency to do things beyond their comprehension often. I take it you've never been fishing before and tied your line in a knot.

trhway 2y ago

> to comprehend our own engineered creations. The comprehend part may never happen. At least by our own mind. We’ll sooner build the mind which is going to do that comprehension: “To scale to the thousands of words supporting the complex thinking chains used by modern models, we will need to improve both the method and (perhaps with AI assistance) how we make sense of what we see with it” Yes, that AI assistance, meta self reflection, is going to probably be a way if not right to the AGI, at least very significant step toward it.

BOOSTERHIDROGEN 2y ago

If only this profound mechanism can be easily testable for social interaction.

MathMonkeyMan 2y ago

In a sense this has been true of conventional programs for a while now. Gerald Sussman discusses the idea when talking about why MIT switched their introductory programming course from Scheme to Python: <https://youtu.be/OgRFOjVzvm0?t=239 https://youtu.be/OgRFOjVzvm0?t=239>.

EGreg 2y ago

I think it’s pretty obvious what these models do in some cases. Try asking them to write a summary at the beginning of their answer. The summary is basically them trying to make something plausible-sounding but they aren’t actually going back and summarizing. LLMs are basically a building block in a larger software. Just like any library or framework. You shouldn’t expect them to be a hammer for every nail. But they can now enable so many different applications, including natural language interfaces, better translations and so forth. But then you’re supposed to have them output JSON to be used in building artifacts like Powerpoints. Has anyone implemented that yet?

hansmayer 2y ago

If you don't mind - based on what will this "paper" become a classic? Was it published in a well known scientific magazine, after undergoing a stringent peer-review process, because it is setting up and proving a new scientific hypothesis? Because this is what scientific papers look like. I struggle to identify any of those characteristics, except for being dense and hard to read, but that's more of a correlation, isn't it?

rcxdude 2y ago

That's basically how engineering works if you're doing anything at all novel: you will have some theory which informs your design, then you build it, then you test it and basically need to do science to figure out how it's perfoming, and most likely, why it's not working properly, and then iterate. I do engineering, but doing science has been a core part of almost every project I've worked on. (heck, even debugging code is basically science). There's just different degrees in different projects as to how much you understand about how the system you're designing actually works, and ML is an area where there's an unusual ratio of visibility (you can see all of the weights and calculations in the network precisely) to understanding (i.e. there's relatively little in terms of mathematical theory that precisely describe how a model trains and operates, just a bunch of approximations which can be somewhat justified, which is where a lot of engineering work sits)

madethisnow 2y ago

psychology

mdnahas 2y ago

I like your definitions! My personal definition of science is learning rules that predict the future, given the present state. And my definition of engineering is arranging the present state to control the future. I don’t think it’s unusual for engineering creations to need new science to understand them. When metal parts broke, humans studied metallurgy. When engines exploded, we studied the remains. With that science, we could engineer larger, longer lasting, more powerful devices. Now, we’re finding flaws in AI and diagnosing their causes. And soon able to build better ones.