5 ms·

> Maybe we don't need to understand the code at all? Hard to fathom. Doesn't seem too crazy. This is the life of every customer (and most managers) of a softwa

by nomel 3mo ago

> Maybe we don't need to understand the code at all? Hard to fathom. Doesn't seem too crazy. This is the life of every customer (and most managers) of a software dev. To fathom it, think of yourself as the unicorn customer that can define projects, specs, and methods with competence, and even make architectural calls, with the AI being the dev. To me, this is the incredibly clear eventuality of our profession, within the next 5-10 years. You writing code will be the same as most devs who try to hand write ASM: you will very very rarely be able to beat the compiler at its own game. And, just as now, there's a very small chance you'll go read what compiler produces.

FloorEgg 3mo ago
I'm a decent full stack web developer, who knows almost nothing about kernel architecture or machine code. Before LLMs I could make a functional app, despite not understanding several layers the app ran on top of.
latentsea 3mo ago
Those layers are deterministic though...
- nomel 3mo ago
  Determinism is NOT a requirement, as proven by your existence of software devs. Again, shift the perspective to the people you work for, or to your fellow dev. They get along fine with you, who are also not deterministic. They check your output by setting requirements that you can prove are true in a way that satisfies them. You yourself guarantee correctness by tests, because you learned long ago that you yourself are not to be trusted, and are not capable of writing 100% correct code on first pass, except for the most trivial tasks. Your lack of determinism is built into your workflow, with tools like linters, code analysis, etc, to help deal with it all.
  - hermitShell 3mo ago
    Do you think this necessarily sends us as a society (as a species) towards a future where we eventually stop understanding technology and science itself? Like in Asimov's Foundation or Liu Cixin's 'Taking Care of God'. Maybe we need two tiers of technology. Human in the loop up to the point of AI, and then the AI layer which depends on the human stack.
    - nomel 3mo ago
      As knowledge progresses, less and less people will be able to understand it, as has always been the case with the progression of technology. This is an inevitability since intelligence is something that can only be slightly learned [1]. The passionate, high intelligence, nerds will continue to do what they've always done, being the only, very very small, percentage of society that bothers to understand any of it. Most humans are dumb, from a science perspective. The ability to be successful in science has an intelligence threshold of somewhere around 120 [2], with true innovation requiring much more (as history has shown). Do you see any alternative to 'Taking Care of God', if we continue progressing? But, for coding specifically, coding as an unfortunate consequence of working with computers. We've been increasing abstraction since their inception, so we can do the actual work with them that we're trying to implement with code as a means to implement it. Go ask and older programmer, and he'll say the same sort of "kids these days don't understand computers", because he had to wire up the logic on a breadboard and write machine code in a hex editor, where you use a javascript library without even knowing how to implement an adder!!! [1] https://www.nature.com/articles/mp2014105 https://www.nature.com/articles/mp2014105 [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC6480791/ https://pmc.ncbi.nlm.nih.gov/articles/PMC6480791/
      - hermitShell 3mo ago
        I take your point, statistically we're already there. Unless humanity was to bootstrap its own biological intelligence to be consistently higher, there's little point in any individual attempting to safeguard the tech stack to first principles. Accept AI or don't accept AI, history tells us that if it works, people will move to the higher abstraction layer. I can imagine an organization that devotes itself to preservation of the history of technology... like monastic groups that copied texts endlessly. In a post-scarcity world, that would be a sensible outcome... people are free to devote themselves to art or science or debauchery as they please...
- win311fwg 3mo ago
  Not really. It is true that supply chain attack concerns has put increased pressure on compiler makers to make compilers deterministic, but historically they haven't been, generally speaking. It was long thought that determinism didn't matter. Where compilers do differ from LLMs is that compiler output correctness is binary, while LLM output correctness is probabilistic. However, compilers have long suffered from inputs not being correct, so we have put a lot of effort into finding ways to validate that the input is correct relative to the output. Which turns out works just as well for LLMs, to the point that validating that the input is correct relative to the output ends up offering the same assurances for both compilers and LLMs alike. Compiler output correctness alone is not actually all that beneficial in practice because, again, it is only as useful as the correctness of the input and inputs are not trustworthy without "double entry accounting" to go along with it. Once you have "double entry accounting" in place then you already have what you need to turn LLM correctness into being binary as well.

by nomel 3mo ago

FloorEgg 3mo ago

I'm a decent full stack web developer, who knows almost nothing about kernel architecture or machine code. Before LLMs I could make a functional app, despite not understanding several layers the app ran on top of.

latentsea 3mo ago

Those layers are deterministic though...

nomel 3mo ago
Determinism is NOT a requirement, as proven by your existence of software devs. Again, shift the perspective to the people you work for, or to your fellow dev. They get along fine with you, who are also not deterministic. They check your output by setting requirements that you can prove are true in a way that satisfies them. You yourself guarantee correctness by tests, because you learned long ago that you yourself are not to be trusted, and are not capable of writing 100% correct code on first pass, except for the most trivial tasks. Your lack of determinism is built into your workflow, with tools like linters, code analysis, etc, to help deal with it all.
- hermitShell 3mo ago
  Do you think this necessarily sends us as a society (as a species) towards a future where we eventually stop understanding technology and science itself? Like in Asimov's Foundation or Liu Cixin's 'Taking Care of God'. Maybe we need two tiers of technology. Human in the loop up to the point of AI, and then the AI layer which depends on the human stack.
  - nomel 3mo ago
    As knowledge progresses, less and less people will be able to understand it, as has always been the case with the progression of technology. This is an inevitability since intelligence is something that can only be slightly learned [1]. The passionate, high intelligence, nerds will continue to do what they've always done, being the only, very very small, percentage of society that bothers to understand any of it. Most humans are dumb, from a science perspective. The ability to be successful in science has an intelligence threshold of somewhere around 120 [2], with true innovation requiring much more (as history has shown). Do you see any alternative to 'Taking Care of God', if we continue progressing? But, for coding specifically, coding as an unfortunate consequence of working with computers. We've been increasing abstraction since their inception, so we can do the actual work with them that we're trying to implement with code as a means to implement it. Go ask and older programmer, and he'll say the same sort of "kids these days don't understand computers", because he had to wire up the logic on a breadboard and write machine code in a hex editor, where you use a javascript library without even knowing how to implement an adder!!! [1] https://www.nature.com/articles/mp2014105 https://www.nature.com/articles/mp2014105 [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC6480791/ https://pmc.ncbi.nlm.nih.gov/articles/PMC6480791/
    - hermitShell 3mo ago
      I take your point, statistically we're already there. Unless humanity was to bootstrap its own biological intelligence to be consistently higher, there's little point in any individual attempting to safeguard the tech stack to first principles. Accept AI or don't accept AI, history tells us that if it works, people will move to the higher abstraction layer. I can imagine an organization that devotes itself to preservation of the history of technology... like monastic groups that copied texts endlessly. In a post-scarcity world, that would be a sensible outcome... people are free to devote themselves to art or science or debauchery as they please...
win311fwg 3mo ago
Not really. It is true that supply chain attack concerns has put increased pressure on compiler makers to make compilers deterministic, but historically they haven't been, generally speaking. It was long thought that determinism didn't matter. Where compilers do differ from LLMs is that compiler output correctness is binary, while LLM output correctness is probabilistic. However, compilers have long suffered from inputs not being correct, so we have put a lot of effort into finding ways to validate that the input is correct relative to the output. Which turns out works just as well for LLMs, to the point that validating that the input is correct relative to the output ends up offering the same assurances for both compilers and LLMs alike. Compiler output correctness alone is not actually all that beneficial in practice because, again, it is only as useful as the correctness of the input and inputs are not trustworthy without "double entry accounting" to go along with it. Once you have "double entry accounting" in place then you already have what you need to turn LLM correctness into being binary as well.