5 ms·

I found it. All that tells you is that it's a simple problem. Had I not been told it was broken I likely would not have. It's the kind of bug that makes you

I found it. All that tells you is that it's a simple problem. Had I not been told it was broken I likely would not have. It's the kind of bug that makes you stop breathing for a brief moment. So I ran this function through Gemini 2.5 Pro, ChatGPT o3 and Grok 3. No context other than the source code fragment was provided. All three also clearly identified the problem. There is no longer a reason for trivial flaws such as this surviving to release code. We're past that now, and that's an astonishing thing. These are the questions I ponder: Should we consider the possibility that the ongoing, incomplete and decades long pursuit of "better" languages is now misguided? Or, that perhaps "better" might now mean whatever properties make code easier for LLMs to analyze and validate against specifications, as opposed to merely preventing the flaws humans are prone to make?

nabla9 1y ago
[flagged]
- topspin 1y ago
  I don't believe I could have been more self-deprecating there. I know for certain I don't care to.
kalaksi 1y ago
> whatever properties make code easier for LLMs to analyze So double down on languages that are more represented in training data? I think it's still worthwhile to actually improve programming languages. Relying only on LLMs is fragile. So ideally do both: improve language and AI.
- topspin 1y ago
  > So double down on languages that are more represented in training data? The pragmatic answer to that is: this appears to be highly effective. What I have in mind is something else, however. Consider the safety nets we try to build with, for example, elaborate type systems. These new analysis tools can operate on enormous contexts and infer and enforce "type" with a capacity far greater what a human mind can hope to approach. So perhaps there is little value to complex types. Instead, a simple type system supported by specification will be the superior approach. seL4 is an example of the concept: C, but exhaustively specified and verified.
  - kccqzy 1y ago
    In my experience good type systems help human programmers and AI agents alike. AI can only infer "type" probabilistically and haphazardly. It is far from able to enforce anything. A good type system helps both humans and AI agents by correcting their mistakes sooner.
dwroberts 1y ago
> I found it. All that tells you is that it's a simple problem Not totally clear what you mean here - are you saying you’re the author of the article or PR, or that you independently discovered the same issue (after the fact)?
- topspin 1y ago
  Ok, so somehow that is causing confusion. I will clarify. The author asked that the reader attempt to find the flaw by inspecting the provided code. The flaw was obvious enough that I was able to find it. The implication is that if it were less obvious, I might not have. I was not attempting to take any credit at all: exactly the opposite.
piker 1y ago
No way is it practical to run complex code like this through a sycophantic parrot. Try it again with some old, well known code and see how many “Certainly! Your error is …” you get.
- spacecadet 1y ago
  Doesnt make one much more than a sycophantic parrot when they also repeat platitudes without anything else to add.
- perching_aix 1y ago
  There was an article here about a month or so ago doing exactly that unpractical thing against I think NFS, and finding a small handful of confirmed vulnerabilities that way. Wish granted I suppose?
  - piker 1y ago
    Fine, but TFA suggests that even -Wunused is too impractical. Taking that at face value, it's not clear how "Hey use this thing that has an even higher latency and lower signal-to-noise ratio" is a solution.
    - perching_aix 1y ago
      Well yeah, you're definitely not going to hear me suggest sending a codebase through LLMs sooner than suggest promoting warnings into errors or using well established tools, that's for sure :p
rini17 1y ago
Don't forget author did 99% of the work for you by finding that function.
- topspin 1y ago
  Having wrote "Had I not been told it was broken I likely would not have," should make it clear that this wasn't lost on me.
spacecadet 1y ago
Like gibberlink. Programming languages are insane to me. In a way we created them to communicate with machines, but they evolved to complement a very human workflow. It makes little sense outside of assisting humans to train LLMs on human computer languages, why not let the AIs generate their own optimized languages?
- topspin 1y ago
  > why not let the AIs generate their own optimized languages? Excellent question. What would it end up looking like? Would it be full of monads and HKTs? Lambda calculus? WebAssembly? I hope to learn the answer one day.
  - spacecadet 1y ago
    Appreciated. I might be a researcher in this space :P
- Roark66 1y ago
  >It makes little sense outside of assisting humans to train LLMs on human computer languages, why not let the AIs generate their own optimized languages? They produce so much BS, but at least now we can catch most of it. If we the languages are suddenly optimised for the AIs we couldn't do that anymore.
0points 1y ago
> There is no longer a reason for trivial flaws such as this surviving to release code. This is unrelated to "no longer due to ChatGPT times". We've been able to detect this issue for decades already, see -Wunused-variable.
- topspin 1y ago
  But: DIV_ROUND_UP(psize, cols); I need sleep, so I'm not going to check, but that may appear to be an intermediate value to a C compiler, and thus escape -Wunused-variable.
  - aaronmdjones 1y ago
    > that may appear to be an intermediate value to a C compiler, and thus escape -Wunused-variable. It does indeed (I checked): $ cat test.c #include <stdint.h> #define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d)) struct vdev; typedef struct vdev vdev_t; struct vdev { uint64_t vdev_ashift; void *vdev_tsd; vdev_t *vdev_top; }; struct vdev_raidz; typedef struct vdev_raidz vdev_raidz_t; struct vdev_raidz { int vd_original_width; int vd_physical_width; int vd_nparity; }; extern uint64_t vdev_raidz_get_logical_width(vdev_raidz_t *, uint64_t); static uint64_t vdev_raidz_asize_to_psize(vdev_t *vd, uint64_t asize, uint64_t txg) { vdev_raidz_t *vdrz = vd->vdev_tsd; uint64_t psize; uint64_t ashift = vd->vdev_top->vdev_ashift; uint64_t cols = vdrz->vd_original_width; uint64_t nparity = vdrz->vd_nparity; cols = vdev_raidz_get_logical_width(vdrz, txg); psize = (asize >> ashift); psize -= nparity * DIV_ROUND_UP(psize, cols); psize <<= ashift; return (asize); } $ gcc-14 -Wall -Wextra -Wpedantic -Wunused -Wunused-variable -Wunused-but-set-variable -c test.c -o test.o test.c:28:5: warning: ‘vdev_raidz_asize_to_psize’ defined but not used [-Wunused-function] 28 | vdev_raidz_asize_to_psize(vdev_t *vd, uint64_t asize, uint64_t txg) | ^~~~~~~~~~~~~~~~~~~~~~~~~ $ clang-19 -Weverything -c test.c -o test.o test.c:33:31: warning: implicit conversion changes signedness: 'int' to 'uint64_t' (aka 'unsigned long') [-Wsign-conversion] 33 | uint64_t cols = vdrz->vd_original_width; | ~~~~ ~~~~~~^~~~~~~~~~~~~~~~~ test.c:34:34: warning: implicit conversion changes signedness: 'int' to 'uint64_t' (aka 'unsigned long') [-Wsign-conversion] 34 | uint64_t nparity = vdrz->vd_nparity; | ~~~~~~~ ~~~~~~^~~~~~~~~~ test.c:28:5: warning: unused function 'vdev_raidz_asize_to_psize' [-Wunused-function] 28 | vdev_raidz_asize_to_psize(vdev_t *vd, uint64_t asize, uint64_t txg) | ^~~~~~~~~~~~~~~~~~~~~~~~~ 3 warnings generated. (The unused-function diagnostic is expected for this test) I would have also expected clang to catch the double assignment to cols, to be honest.
gf000 1y ago
I mean, once you pinpoint 5 lines of code and say that there is an error, it is not particularly hard to find it. Now give the whole repo before the fix to an LLM and ask it to find this bug with some context the author started with. I'm sure you can fix every software issue that way as well!
kccqzy 1y ago
The pursuit of "better" languages might be incomplete, but it has long been known that this specific problem can be solved by the newtype pattern. It might have originated in Haskell but it's equally applicable in more traditional languages like C++. All you need is to define a named struct containing an integer called PhysicalSize and another one called AllocatedSize. In C++ add operator overloading to it so you can do arithmetic. In Haskell add instances such as Integral and Num.

nabla9 1y ago

[flagged]

topspin 1y ago
I don't believe I could have been more self-deprecating there. I know for certain I don't care to.

kalaksi 1y ago

> whatever properties make code easier for LLMs to analyze So double down on languages that are more represented in training data? I think it's still worthwhile to actually improve programming languages. Relying only on LLMs is fragile. So ideally do both: improve language and AI.

topspin 1y ago
> So double down on languages that are more represented in training data? The pragmatic answer to that is: this appears to be highly effective. What I have in mind is something else, however. Consider the safety nets we try to build with, for example, elaborate type systems. These new analysis tools can operate on enormous contexts and infer and enforce "type" with a capacity far greater what a human mind can hope to approach. So perhaps there is little value to complex types. Instead, a simple type system supported by specification will be the superior approach. seL4 is an example of the concept: C, but exhaustively specified and verified.
- kccqzy 1y ago
  In my experience good type systems help human programmers and AI agents alike. AI can only infer "type" probabilistically and haphazardly. It is far from able to enforce anything. A good type system helps both humans and AI agents by correcting their mistakes sooner.

dwroberts 1y ago

> I found it. All that tells you is that it's a simple problem Not totally clear what you mean here - are you saying you’re the author of the article or PR, or that you independently discovered the same issue (after the fact)?

topspin 1y ago
Ok, so somehow that is causing confusion. I will clarify. The author asked that the reader attempt to find the flaw by inspecting the provided code. The flaw was obvious enough that I was able to find it. The implication is that if it were less obvious, I might not have. I was not attempting to take any credit at all: exactly the opposite.

piker 1y ago

No way is it practical to run complex code like this through a sycophantic parrot. Try it again with some old, well known code and see how many “Certainly! Your error is …” you get.

spacecadet 1y ago
Doesnt make one much more than a sycophantic parrot when they also repeat platitudes without anything else to add.
perching_aix 1y ago
There was an article here about a month or so ago doing exactly that unpractical thing against I think NFS, and finding a small handful of confirmed vulnerabilities that way. Wish granted I suppose?
- piker 1y ago
  Fine, but TFA suggests that even -Wunused is too impractical. Taking that at face value, it's not clear how "Hey use this thing that has an even higher latency and lower signal-to-noise ratio" is a solution.
  - perching_aix 1y ago
    Well yeah, you're definitely not going to hear me suggest sending a codebase through LLMs sooner than suggest promoting warnings into errors or using well established tools, that's for sure :p

rini17 1y ago

Don't forget author did 99% of the work for you by finding that function.

topspin 1y ago
Having wrote "Had I not been told it was broken I likely would not have," should make it clear that this wasn't lost on me.

spacecadet 1y ago

Like gibberlink. Programming languages are insane to me. In a way we created them to communicate with machines, but they evolved to complement a very human workflow. It makes little sense outside of assisting humans to train LLMs on human computer languages, why not let the AIs generate their own optimized languages?

topspin 1y ago
> why not let the AIs generate their own optimized languages? Excellent question. What would it end up looking like? Would it be full of monads and HKTs? Lambda calculus? WebAssembly? I hope to learn the answer one day.
- spacecadet 1y ago
  Appreciated. I might be a researcher in this space :P
Roark66 1y ago
>It makes little sense outside of assisting humans to train LLMs on human computer languages, why not let the AIs generate their own optimized languages? They produce so much BS, but at least now we can catch most of it. If we the languages are suddenly optimised for the AIs we couldn't do that anymore.

0points 1y ago

> There is no longer a reason for trivial flaws such as this surviving to release code. This is unrelated to "no longer due to ChatGPT times". We've been able to detect this issue for decades already, see -Wunused-variable.

topspin 1y ago
But: DIV_ROUND_UP(psize, cols); I need sleep, so I'm not going to check, but that may appear to be an intermediate value to a C compiler, and thus escape -Wunused-variable.
- aaronmdjones 1y ago
  > that may appear to be an intermediate value to a C compiler, and thus escape -Wunused-variable. It does indeed (I checked): $ cat test.c #include <stdint.h> #define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d)) struct vdev; typedef struct vdev vdev_t; struct vdev { uint64_t vdev_ashift; void *vdev_tsd; vdev_t *vdev_top; }; struct vdev_raidz; typedef struct vdev_raidz vdev_raidz_t; struct vdev_raidz { int vd_original_width; int vd_physical_width; int vd_nparity; }; extern uint64_t vdev_raidz_get_logical_width(vdev_raidz_t *, uint64_t); static uint64_t vdev_raidz_asize_to_psize(vdev_t *vd, uint64_t asize, uint64_t txg) { vdev_raidz_t *vdrz = vd->vdev_tsd; uint64_t psize; uint64_t ashift = vd->vdev_top->vdev_ashift; uint64_t cols = vdrz->vd_original_width; uint64_t nparity = vdrz->vd_nparity; cols = vdev_raidz_get_logical_width(vdrz, txg); psize = (asize >> ashift); psize -= nparity * DIV_ROUND_UP(psize, cols); psize <<= ashift; return (asize); } $ gcc-14 -Wall -Wextra -Wpedantic -Wunused -Wunused-variable -Wunused-but-set-variable -c test.c -o test.o test.c:28:5: warning: ‘vdev_raidz_asize_to_psize’ defined but not used [-Wunused-function] 28 | vdev_raidz_asize_to_psize(vdev_t *vd, uint64_t asize, uint64_t txg) | ^~~~~~~~~~~~~~~~~~~~~~~~~ $ clang-19 -Weverything -c test.c -o test.o test.c:33:31: warning: implicit conversion changes signedness: 'int' to 'uint64_t' (aka 'unsigned long') [-Wsign-conversion] 33 | uint64_t cols = vdrz->vd_original_width; | ~~~~ ~~~~~~^~~~~~~~~~~~~~~~~ test.c:34:34: warning: implicit conversion changes signedness: 'int' to 'uint64_t' (aka 'unsigned long') [-Wsign-conversion] 34 | uint64_t nparity = vdrz->vd_nparity; | ~~~~~~~ ~~~~~~^~~~~~~~~~ test.c:28:5: warning: unused function 'vdev_raidz_asize_to_psize' [-Wunused-function] 28 | vdev_raidz_asize_to_psize(vdev_t *vd, uint64_t asize, uint64_t txg) | ^~~~~~~~~~~~~~~~~~~~~~~~~ 3 warnings generated. (The unused-function diagnostic is expected for this test) I would have also expected clang to catch the double assignment to cols, to be honest.

gf000 1y ago

I mean, once you pinpoint 5 lines of code and say that there is an error, it is not particularly hard to find it. Now give the whole repo before the fix to an LLM and ask it to find this bug with some context the author started with. I'm sure you can fix every software issue that way as well!

kccqzy 1y ago

The pursuit of "better" languages might be incomplete, but it has long been known that this specific problem can be solved by the newtype pattern. It might have originated in Haskell but it's equally applicable in more traditional languages like C++. All you need is to define a named struct containing an integer called PhysicalSize and another one called AllocatedSize. In C++ add operator overloading to it so you can do arithmetic. In Haskell add instances such as Integral and Num.