4 ms·

Idk from when even id this article? Got me LLMs currently are broke and the majority is already aware of this. Copilot fails the cleanly refactor complex Java

by javaunsafe2019 2y ago

Idk from when even id this article? Got me LLMs currently are broke and the majority is already aware of this. Copilot fails the cleanly refactor complex Java methods in a way that I’m better of writing that stuff by my own as I have to understand it anyways. And the news that they don’t scale as predicted is too bad compared to how weak they currently perform…

lxgr 2y ago
Why does an LLM have to be better than you to be useful to you? Personally, I use them for the things they can do, and for the things they can't, I just don't, exactly as I would for any other tool. People assuming they can do more than they are actually capable of is a problem (compounded by our tendency to attribute intelligence to entities with eloquent language, which might be more of a surface level thing than we used to believe), but that's literally been one for as long as we had proverbial hammers and nails.
- lou1306 2y ago
  > Why does an LLM have to be better than you to be useful to you? If ((time to craft the prompt) + (time required to fix LLM output)) ~ (time to achieve the task on my own) it's not hard to see that working on my own is a very attractive proposition. It drives down complexity, does not require me to acquire new skills (i.e., prompt engineering), does not require me to provide data to a third party nor to set up an expensive rig to run a model locally, etc.
  - lxgr 2y ago
    Then they might indeed not be the right tool for what you're trying to do. I'm just a little bit tired of sweeping generalizations like "LLMs are completely broken". You can easily use them as a tool part of a process that then ends up being broken (because it's the wrong tool!), yet that doesn't disqualify them for all tool use.
    - lou1306 2y ago
      Yeah but the sampling process required to determine what they are good at is not free either. (For starters, it's consuming huge amounts of public research funding and compute, but let's not go down that rabbit-hole)
vonneumannstan 2y ago
If you can't find a use for the best LLMs it is 100% a skill issue. IF the only way you can think to use them is re-factoring complex java codebases you're ngmi.
- lxgr 2y ago
  So far I haven't found one that does my dishes and laundry. I really wish I knew how to properly use them. My point being: Why would anyone have to find a use for a new tool? Why wouldn't "it doesn't help me with what I'm trying to do" be an acceptable answer in many cases?
  - Workaccount2 2y ago
    I have found more often than not that people in the "LLMs are useless" camp are actually in the "I need LLMs to be useless" camp.
    - exe34 2y ago
      nice example of poisoning the well!
    - mdp2021 2y ago
      Do not forget the very linear reality of those people that shout "The car does not work!" in frustration because they would gladly use a car.
    - tokinonagare 2y ago
      Well I found the exact reverse: people saying LLM are useful actually need them to be useful, so they could boast about their "prompt engineering" "skill" (i.e. typing a sentence) and "AI knowledge". I've seen a caricature of this a few hours ago on LinkedIn, from a "data guy" saying devs not using AI are gonna replaced by those who do. Yet it was very clear from his reply to comments he never wrote code and wasn't a position to give an opinion on the matter, especially one like the extreme and rude one he wrote.

lxgr 2y ago

Why does an LLM have to be better than you to be useful to you? Personally, I use them for the things they can do, and for the things they can't, I just don't, exactly as I would for any other tool. People assuming they can do more than they are actually capable of is a problem (compounded by our tendency to attribute intelligence to entities with eloquent language, which might be more of a surface level thing than we used to believe), but that's literally been one for as long as we had proverbial hammers and nails.

lou1306 2y ago
> Why does an LLM have to be better than you to be useful to you? If ((time to craft the prompt) + (time required to fix LLM output)) ~ (time to achieve the task on my own) it's not hard to see that working on my own is a very attractive proposition. It drives down complexity, does not require me to acquire new skills (i.e., prompt engineering), does not require me to provide data to a third party nor to set up an expensive rig to run a model locally, etc.
- lxgr 2y ago
  Then they might indeed not be the right tool for what you're trying to do. I'm just a little bit tired of sweeping generalizations like "LLMs are completely broken". You can easily use them as a tool part of a process that then ends up being broken (because it's the wrong tool!), yet that doesn't disqualify them for all tool use.
  - lou1306 2y ago
    Yeah but the sampling process required to determine what they are good at is not free either. (For starters, it's consuming huge amounts of public research funding and compute, but let's not go down that rabbit-hole)

vonneumannstan 2y ago

If you can't find a use for the best LLMs it is 100% a skill issue. IF the only way you can think to use them is re-factoring complex java codebases you're ngmi.

lxgr 2y ago
So far I haven't found one that does my dishes and laundry. I really wish I knew how to properly use them. My point being: Why would anyone have to find a use for a new tool? Why wouldn't "it doesn't help me with what I'm trying to do" be an acceptable answer in many cases?
- Workaccount2 2y ago
  I have found more often than not that people in the "LLMs are useless" camp are actually in the "I need LLMs to be useless" camp.
  - exe34 2y ago
    nice example of poisoning the well!
  - mdp2021 2y ago
    Do not forget the very linear reality of those people that shout "The car does not work!" in frustration because they would gladly use a car.
  - tokinonagare 2y ago
    Well I found the exact reverse: people saying LLM are useful actually need them to be useful, so they could boast about their "prompt engineering" "skill" (i.e. typing a sentence) and "AI knowledge". I've seen a caricature of this a few hours ago on LinkedIn, from a "data guy" saying devs not using AI are gonna replaced by those who do. Yet it was very clear from his reply to comments he never wrote code and wasn't a position to give an opinion on the matter, especially one like the extreme and rude one he wrote.