6 ms·

I'm reminded, once again, of the recent "vibe coded" OCaml fiasco[1]. The PR author had zero understanding why their entirely LLM-generated contribution was vi

by Alupis 8mo ago

I'm reminded, once again, of the recent "vibe coded" OCaml fiasco[1]. The PR author had zero understanding why their entirely LLM-generated contribution was viewed so suspiciously. The article validates a significant point: it is one thing to have passing tests and be able to produce output that resembles correctness - however it's something entirely different for that output to be good and maintainable. [1] https://github.com/ocaml/ocaml/pull/14369 https://github.com/ocaml/ocaml/pull/14369

Culonavirus 8mo ago
Damn... "AI has a very deep understanding of how this code works. Please challenge me on this." this person is something else. Just... wow.
- copilot_king_2 8mo ago
  This is the norm in my experience. If you haven't come across a significant number of AI addicts as obnoxiously delusional as @Culonavirus describes, you must be getting close to retirement age. People with any connection to new college graduates understand that this sort of idiotic LLM-backed arrogance is extremely common among low-to-mid-functioning twenty-somethings.
- co_king_3 8mo ago
  [flagged]
  - monegator 8mo ago
    *grifters
    - co_king_3 8mo ago
      Successful grifters are not this stupid. Terminal AI users are genuinely *slow* individuals. These people are functionally illiterate; they are not able to extract information by reading a bit of text in the same ways that you or I are. These people are very, very *gullible*.
  - ratrace 8mo ago
    This comment does not deserve to be greyed out.
- nyantaro1 8mo ago
  nice username
tossandthrow 8mo ago
The Ai legal analysis seemed to be the nail in the coffin. Adding Ai generated comments are IMHO some of the most rude uses of Ai.
- usrusr 8mo ago
  Not sure what exactly you're referring to, but legal is a very interesting field to observe, right? I've been wondering about that since quite early in my LLM awareness: A slightly sarcastic (or perhaps not so slightly..) mental model of legal conflict resolution is that much of it boils down to throwing lots of content at the opposing side, claiming that it shows that the represented side is right and creating a task for the opposite side to find a flaw in that material. I believe that this game of quantity fits through the whole range from "I'll have my lawyer repeat my argument in a letter featuring their letter head" all the way to paper-tsunamis like the Google-Oracle trial. Now give both sides access to LLM... I wonder if the legal profession will eventually settle on some format of in-person offline resolution with strict limits to recess and/or limits to word count for both documents and notes, because otherwise conflicts fail to get settled in anyone's lifetime (or won by whoever does not run out of tokens first - come thinking of it, the technogarchs would love this, so I guess this is exactly what will happen barring a revolution)
  - tossandthrow 8mo ago
    Ah, sorry. I am not referring to using LLMs for legal work. I am referring to the act of merely pasting the output of a model as a comment. Have the decency to understand what the LLM is writing and write your own message.
    - agile-gift0262 8mo ago
      That comment is wild > Here's the AI-written copyright analysis... I'm not going to spend more time reading than what you have spent writing!
  - 63stack 8mo ago
    What do you mean "not exactly sure what you are referring to"? The guy just posted a huge ai slop pr, do you think that's the correct place for "very interesting field observations about legal"? What else could it refer to than you can't back up copyright ownership questions with "ai said so"??
kleiba 8mo ago
I just read that whole thread and I think the author made the mistake of submitting a 13k loc PR, but other than that - while he gets downvoted to hell on every comment - he's actually acting professionally and politely. I wouldn't call this a fiasco, it reads to me more that being able to create huge amounts of code - whether the end result works well or not - breaks the traditional model of open source. Small contributions can be verified and the merrit-vs-maintenance-effort can at least be assessed somewhat more realistically. I have no bones in the "vibe coding sucks" vs "vibe coding rocks" discussion and I reading that thread as an outsider. I cannot help but find the PR author's attitude absolutely okay while the compiler folks are very defensive. I do agree with them that submitting a huge PR request without prior discussion cannot be the way forward. But that's almost orthogonal to the question of whether AI-generated code is or is not of value. If I were the author, I would probably take my 13k loc proof-of-concept implementation and chop it down into bite-size steps that are easy to digest, and try to get them to get integrated into the compiler successively, with being totally upfront about what the final goal is. You'd need to be ready to accept criticism and requests for change, but it should not be too hard to have your AI of choice incorporate these into your code base. I think the main mistake of the author was not to use vibe coding, it was to dream up his own personal ideal of a huge feature, and then go ahead and single-handedly implement the whole thing without involving anyone from the actual compiler project. You cannot blame the maintainers for not being crazy about accepting such a huge blob.
- vasco 8mo ago
  He is not polite, he is of the utmost rudeness. As a reply to being pointed to the fact that he copied so much code that the generated code included someone else's name in the License, his reply was https://github.com/ocaml/ocaml/pull/14369/changes/ce372a60bd97e0a19b336c3042c4036cfda3c01e https://github.com/ocaml/ocaml/pull/14369/changes/ce372a60bd... I struggle to think how someone thinks this is polite. Is politeness to you just not using curse words?
  - kleiba 8mo ago
    Admittedly, his handling of this aspect was perhaps less than ideal, but I cannot see any impoliteness here whatsoever. As a matter of fact, I struggle to think how you could think otherwise. But I am biased. After having lived a number of years in a country where I would say the average understanding of politeness is vastly different from where I've grown up, I've learned that there is just a difference of opinion of what is polite and what isn't. I have probably been affected by that too.
onion2k 8mo ago
however it's something entirely different for that output to be good and maintainable People aren't prompting LLMs to write good, maintainable code though. They're assuming that because we've made a collective assumption that good, maintainable code is the goal then it must also be the goal of an LLM too. That isn't true. LLMs don't care about our goals. They are solving problems in a probabilistic way based on the content of their training data, context, and prompting. Presumably if you take all the code in the world and throw it in mixer what comes out is not our Platonic ideal of the best possible code, but actually something more like a Lovecraftian horror that happens to get the right output. This is quite positive because it shows that with better prompting+context+training we might actually be able to guide an LLM to know what good and bad looks like (based on the fact that we know). The future is looking great. However, we also need to be aware that 'good, maintainable code' is often not what we think is the ideal output of a developer. In businesses everywhere the goal is 'whatever works right now, and to hell with maintainability'. When a business is 3 months from failing spending time to write good code that you can continue to work on in 10 years feels like wasted effort. So really, for most code that's written, it doesn't actually need to be good or maintainable. It just needs to work. And if you look at the code that a lot of businesses are running, it doesn't. LLMs are a step forward in just getting stuff to work in the first place. If we can move to 'bug free' using AI, at the unit level, then AI is useful. Above individual units of code, like logic, architecture, security, etc things still have to come from the developer because AI can't have the context of a complete application yet. When that's ready then we can tackle 'tech debt free' because almost all tech debt lives at that higher level. I don't think we'll get there for a long time.
- codebolt 8mo ago
  > People aren't prompting LLMs to write good, maintainable code though. Then they're not using the tools correctly. LLMs are capable of producing good clean code, but they need to be carefully instructed as to how. I recently used Gemini to build my first Android app, and I have zero experience with Kotlin or most of the libraries (but I have done many years of enterprise Java in my career). When I started I first had a long discussion with the AI about how we should set up dependency injection, Material3 UI components, model-view architecture, Firebase, logging, etc and made a big Markdown file with a detailed architecture description. Then I let the agent mode implement the plan over several steps and with a lot of tweaking along the way. I've been quite happy with the result, the app works like a charm and the code is neatly structured and easy to jump into whenever I need to make changes. Finishing a project like this in a couple of dozen hours (especially being a complete newbie to the stack) simply would not have been possible 2-3 years ago.
Incipient 8mo ago
>Here's my question: why did the files that you submitted name Mark Shinwell as the author? >Beats me. AI decided to do so and I didn't question it. Haha that's comedy gold, and honestly a good interview screening situation - you'd instantly pass on the candidate!
- thewhitetulip 8mo ago
  I once had a PR. I told the dev that "LLM is ok but you own the code" He told me "I spent n days to architect the solution" He shows me claude generated system design .. and then i say ok, I went to review the code. 1hr later i asked why did you repeat the code all over at the end. Dude replies "junk the entire PR it's AI generated"
  - pseudohadamard 8mo ago
    Has anyone who's familiar with compiler source code tried to compare it to other compilers? Given that LLMs have been trained on data sets that include the source code for numerous C compilers, is this just (say) pcc extruded in Rust form?
simgt 8mo ago
I'm humbled by the maintainer's answer [0]. Must be great to work with people like him who have infinite patience and composure. [0] https://github.com/ocaml/ocaml/pull/14369#issuecomment-3556593972 https://github.com/ocaml/ocaml/pull/14369#issuecomment-35565...
- amoss 8mo ago
  gasche has been active on various forums over the years, and yes can confirm that he has infinite patience.
- zozbot234 8mo ago
  Yes, that comment by gasche is a very good general explanation for why vibe coded slop still doesn't cut it for contributing to any non-trivial FLOSS project. When you're building towards a large feature (DWARF support in this case) it's critical for contributions to be small and self-contained so that maintainers and reviewers don't get overwhelmed. As things stand, this means that human effort is an absolute requirement. When contributions are small and tightly human-controlled it's also less likely that potential legal concerns will arise, since it means that any genuinely creative decisions about the code are a lot easier to trace. (In this case, the AI seems to have ripped off a lot of the work from OxCaml with inconsistent attribution. OxCaml is actually license compatible (and friendly) with Ocaml but obviously any merge of that work should happen on its own terms, not as a side effect of ripoff slop code.)
- ordu 8mo ago
  > Must be great to work with people like him who have infinite patience and composure. It is not just patience, he is ready to spent a shitload of time explaining basics to strangers. Such an answer would take, I believe would take a very least half an hour to compose, not counting the time you need to read all the relevant discussion to get the context. But yeah, it would be great to have more people like him around.
63stack 8mo ago
I get infuriated just from reading that, I wish I had as much patience as the maintainers on that project.

by Alupis 8mo ago

Culonavirus 8mo ago

Damn... "AI has a very deep understanding of how this code works. Please challenge me on this." this person is something else. Just... wow.

copilot_king_2 8mo ago
This is the norm in my experience. If you haven't come across a significant number of AI addicts as obnoxiously delusional as @Culonavirus describes, you must be getting close to retirement age. People with any connection to new college graduates understand that this sort of idiotic LLM-backed arrogance is extremely common among low-to-mid-functioning twenty-somethings.
co_king_3 8mo ago
[flagged]
- monegator 8mo ago
  *grifters
  - co_king_3 8mo ago
    Successful grifters are not this stupid. Terminal AI users are genuinely *slow* individuals. These people are functionally illiterate; they are not able to extract information by reading a bit of text in the same ways that you or I are. These people are very, very *gullible*.
- ratrace 8mo ago
  This comment does not deserve to be greyed out.
nyantaro1 8mo ago
nice username

tossandthrow 8mo ago

The Ai legal analysis seemed to be the nail in the coffin. Adding Ai generated comments are IMHO some of the most rude uses of Ai.

usrusr 8mo ago
Not sure what exactly you're referring to, but legal is a very interesting field to observe, right? I've been wondering about that since quite early in my LLM awareness: A slightly sarcastic (or perhaps not so slightly..) mental model of legal conflict resolution is that much of it boils down to throwing lots of content at the opposing side, claiming that it shows that the represented side is right and creating a task for the opposite side to find a flaw in that material. I believe that this game of quantity fits through the whole range from "I'll have my lawyer repeat my argument in a letter featuring their letter head" all the way to paper-tsunamis like the Google-Oracle trial. Now give both sides access to LLM... I wonder if the legal profession will eventually settle on some format of in-person offline resolution with strict limits to recess and/or limits to word count for both documents and notes, because otherwise conflicts fail to get settled in anyone's lifetime (or won by whoever does not run out of tokens first - come thinking of it, the technogarchs would love this, so I guess this is exactly what will happen barring a revolution)
- tossandthrow 8mo ago
  Ah, sorry. I am not referring to using LLMs for legal work. I am referring to the act of merely pasting the output of a model as a comment. Have the decency to understand what the LLM is writing and write your own message.
  - agile-gift0262 8mo ago
    That comment is wild > Here's the AI-written copyright analysis... I'm not going to spend more time reading than what you have spent writing!
- 63stack 8mo ago
  What do you mean "not exactly sure what you are referring to"? The guy just posted a huge ai slop pr, do you think that's the correct place for "very interesting field observations about legal"? What else could it refer to than you can't back up copyright ownership questions with "ai said so"??

kleiba 8mo ago

I just read that whole thread and I think the author made the mistake of submitting a 13k loc PR, but other than that - while he gets downvoted to hell on every comment - he's actually acting professionally and politely. I wouldn't call this a fiasco, it reads to me more that being able to create huge amounts of code - whether the end result works well or not - breaks the traditional model of open source. Small contributions can be verified and the merrit-vs-maintenance-effort can at least be assessed somewhat more realistically. I have no bones in the "vibe coding sucks" vs "vibe coding rocks" discussion and I reading that thread as an outsider. I cannot help but find the PR author's attitude absolutely okay while the compiler folks are very defensive. I do agree with them that submitting a huge PR request without prior discussion cannot be the way forward. But that's almost orthogonal to the question of whether AI-generated code is or is not of value. If I were the author, I would probably take my 13k loc proof-of-concept implementation and chop it down into bite-size steps that are easy to digest, and try to get them to get integrated into the compiler successively, with being totally upfront about what the final goal is. You'd need to be ready to accept criticism and requests for change, but it should not be too hard to have your AI of choice incorporate these into your code base. I think the main mistake of the author was not to use vibe coding, it was to dream up his own personal ideal of a huge feature, and then go ahead and single-handedly implement the whole thing without involving anyone from the actual compiler project. You cannot blame the maintainers for not being crazy about accepting such a huge blob.

vasco 8mo ago
He is not polite, he is of the utmost rudeness. As a reply to being pointed to the fact that he copied so much code that the generated code included someone else's name in the License, his reply was https://github.com/ocaml/ocaml/pull/14369/changes/ce372a60bd97e0a19b336c3042c4036cfda3c01e https://github.com/ocaml/ocaml/pull/14369/changes/ce372a60bd... I struggle to think how someone thinks this is polite. Is politeness to you just not using curse words?
- kleiba 8mo ago
  Admittedly, his handling of this aspect was perhaps less than ideal, but I cannot see any impoliteness here whatsoever. As a matter of fact, I struggle to think how you could think otherwise. But I am biased. After having lived a number of years in a country where I would say the average understanding of politeness is vastly different from where I've grown up, I've learned that there is just a difference of opinion of what is polite and what isn't. I have probably been affected by that too.

onion2k 8mo ago

however it's something entirely different for that output to be good and maintainable People aren't prompting LLMs to write good, maintainable code though. They're assuming that because we've made a collective assumption that good, maintainable code is the goal then it must also be the goal of an LLM too. That isn't true. LLMs don't care about our goals. They are solving problems in a probabilistic way based on the content of their training data, context, and prompting. Presumably if you take all the code in the world and throw it in mixer what comes out is not our Platonic ideal of the best possible code, but actually something more like a Lovecraftian horror that happens to get the right output. This is quite positive because it shows that with better prompting+context+training we might actually be able to guide an LLM to know what good and bad looks like (based on the fact that we know). The future is looking great. However, we also need to be aware that 'good, maintainable code' is often not what we think is the ideal output of a developer. In businesses everywhere the goal is 'whatever works right now, and to hell with maintainability'. When a business is 3 months from failing spending time to write good code that you can continue to work on in 10 years feels like wasted effort. So really, for most code that's written, it doesn't actually need to be good or maintainable. It just needs to work. And if you look at the code that a lot of businesses are running, it doesn't. LLMs are a step forward in just getting stuff to work in the first place. If we can move to 'bug free' using AI, at the unit level, then AI is useful. Above individual units of code, like logic, architecture, security, etc things still have to come from the developer because AI can't have the context of a complete application yet. When that's ready then we can tackle 'tech debt free' because almost all tech debt lives at that higher level. I don't think we'll get there for a long time.

codebolt 8mo ago
> People aren't prompting LLMs to write good, maintainable code though. Then they're not using the tools correctly. LLMs are capable of producing good clean code, but they need to be carefully instructed as to how. I recently used Gemini to build my first Android app, and I have zero experience with Kotlin or most of the libraries (but I have done many years of enterprise Java in my career). When I started I first had a long discussion with the AI about how we should set up dependency injection, Material3 UI components, model-view architecture, Firebase, logging, etc and made a big Markdown file with a detailed architecture description. Then I let the agent mode implement the plan over several steps and with a lot of tweaking along the way. I've been quite happy with the result, the app works like a charm and the code is neatly structured and easy to jump into whenever I need to make changes. Finishing a project like this in a couple of dozen hours (especially being a complete newbie to the stack) simply would not have been possible 2-3 years ago.

Incipient 8mo ago

>Here's my question: why did the files that you submitted name Mark Shinwell as the author? >Beats me. AI decided to do so and I didn't question it. Haha that's comedy gold, and honestly a good interview screening situation - you'd instantly pass on the candidate!

thewhitetulip 8mo ago
I once had a PR. I told the dev that "LLM is ok but you own the code" He told me "I spent n days to architect the solution" He shows me claude generated system design .. and then i say ok, I went to review the code. 1hr later i asked why did you repeat the code all over at the end. Dude replies "junk the entire PR it's AI generated"
- pseudohadamard 8mo ago
  Has anyone who's familiar with compiler source code tried to compare it to other compilers? Given that LLMs have been trained on data sets that include the source code for numerous C compilers, is this just (say) pcc extruded in Rust form?

simgt 8mo ago

I'm humbled by the maintainer's answer [0]. Must be great to work with people like him who have infinite patience and composure. [0] https://github.com/ocaml/ocaml/pull/14369#issuecomment-3556593972 https://github.com/ocaml/ocaml/pull/14369#issuecomment-35565...

amoss 8mo ago
gasche has been active on various forums over the years, and yes can confirm that he has infinite patience.
zozbot234 8mo ago
Yes, that comment by gasche is a very good general explanation for why vibe coded slop still doesn't cut it for contributing to any non-trivial FLOSS project. When you're building towards a large feature (DWARF support in this case) it's critical for contributions to be small and self-contained so that maintainers and reviewers don't get overwhelmed. As things stand, this means that human effort is an absolute requirement. When contributions are small and tightly human-controlled it's also less likely that potential legal concerns will arise, since it means that any genuinely creative decisions about the code are a lot easier to trace. (In this case, the AI seems to have ripped off a lot of the work from OxCaml with inconsistent attribution. OxCaml is actually license compatible (and friendly) with Ocaml but obviously any merge of that work should happen on its own terms, not as a side effect of ripoff slop code.)
ordu 8mo ago
> Must be great to work with people like him who have infinite patience and composure. It is not just patience, he is ready to spent a shitload of time explaining basics to strangers. Such an answer would take, I believe would take a very least half an hour to compose, not counting the time you need to read all the relevant discussion to get the context. But yeah, it would be great to have more people like him around.

63stack 8mo ago

I get infuriated just from reading that, I wish I had as much patience as the maintainers on that project.