7 ms·

You're sort of presuming that LLMs are going to be a massive service to the world there, aren't you? I think the jury is still out on that one.

jimmaswell 2y ago
They already have been. Even just in programming, even just Copilot has been a life changing productivity booster.
- wruza 2y ago
  Are you sure it’s a life changing productivity booster? Sometimes I look at my projects and wonder how would I explain it to an LLM what this code should have done if it didn’t exist yet. Must be a shitton of boilerplate programming for copilot to be a life-changing experience.
  - AYBABTME 2y ago
    You haven't used them enough. Everytime an LLM reduces my search from 1min to 5s, the LLM pays. Just summary features: save me 20min of reading a transcript, turn it into 20s. That's a huge enabler.
    - mecsred 2y ago
      If 20 mins of informations can legitimately be condensed into 20 seconds, it sounds like the original wasn't worth reading in the first place. Could have skipped the llm entirely.
      - bostik 2y ago
        I upvoted you, because I think you have a valid point. The tone is unnecessarily aggressive though. Effective and information-dense communication is really hard. That doesn't mean we should just accept the useless fluff surrounding the actual information and/or analysis. People could learn a lot from the Ignoble Prize ceremony's 24/7 presentation model. Sadly, it seems we are heading towards a future where you may need an LLM to distill the relevant information out of a sea of noise.
        mecsred 2y ago
        Didn't intend for it to be aggressive, just concise. Spare me from the llm please :)
      - AYBABTME 2y ago
        Think of the summary of a zoom call. Or of a chapter that you're not sure if you care to read or not. Not all content is worth consuming, and not all content is dense.
        postalrat 2y ago
        If I had a recording of the zoom call I could generate a summary on demand with better tools than were available at the time the zoom call was made.
      - crazygringo 2y ago
        > it sounds like the original wasn't worth reading in the first place But if that's the only place that contained the information you needed, then you have no choice. There's a lot of material out there that is badly written, badly organized, badly presented. LLM's can be a godsend for extracting the information you actually need without wasting 20 minutes wading through the muck.
        mecsred 2y ago
        Yeah I can see that use case, I just wouldn't trust an LLM to decide "is this worth reading". May as well flip a coin.
    - wruza 2y ago
      Overviews aren’t code though. In code, for me, they don’t pass 80/20 tests well enough, sometimes even on simple cases. (You get 50-80% of an existing function/block with some important context prepended and a comment, let it write the rest and check if it succeeds). It doesn’t mean that LLMs are useless. Or that I am antillamist or a denier - I’m actually an enthusiast. But this specific claim I hear often and don’t find true. Maybe true for repetitive code in boring environments where typing and remembering formats/params over and over is the main issue. Not in actual code. If I paste the actual non-trivial code, it starts deviating fast. And it isn’t too complex, it’s just less like “parallel sort two arrays” and more like “wait for an image on a screenshot by execing scrot (with no sound) repeatedly and passing the result to this detect-cv2.py script and use all matching options described in this ts type, get stdout json as in this ts type, and if there’s a match, wait for the specified anim timeout and test again to get the settled match coords after an animation finishes; throw after a total timeout”. Not a rocket science, pretty dumb shit, but right there they fall flat and start imagining things, heavily. I guess it shines if you ask it to make an html form, but I couldn’t call that life-changing unless I had to make these damn forms all day.
    - andrei_says_ 2y ago
      My experience with overviews is that they are often subtly or not so subtly inaccurate. LLMs not understanding meaning or intent carries risk of misrepresentation.
- giraffe_lady 2y ago
  "Even just in programming" the jury is still out. None of my coworkers using these are noticeably more productive than the ones who don't. Outside of programming no one gives a shit except scammers and hype chasers.
  - JonChesterfield 2y ago
    The people writing articles for journals that aggregate and approximate other sources are in mortal terror of LLMs. Likewise graphic designers and anyone working in (human language) translation. I don't fear that LLMs are going to take my job as a developer. I'm pretty sure they mark a further decrease in the quality and coherence of software, along with a rapid increase in the quantity of code out there, and that seems likely to provide me with reliable employment forever. I'm basically employed in fixing bugs that didn't need to exist in the first place and that seems to cover a lot of software dev.
    - giraffe_lady 2y ago
      They're not scared of LLMs because of anything about LLMs. It's just that everyone with power is publicly horny to delete the remaining middle class jobs and are happy to use LLMs as a justification whether it can functionally replace those workers or not. So it's not that everyone has evaluated chatgpt and cannily realized it can do their job, they're just reading the room.
- recursive 2y ago
  I've been using copilot for several months. If I could figure out a way to measure its impact on my productivity, I'd probably see a single digit percentage boost in "productivity". This is not life-changing for me. And for some tasks, it's actually worse than nothing. As in, I spend time feeding it a task, and it just completely fails to do anything useful.
  - jimmaswell 2y ago
    I've been using it for over a year I think. I don't often feed it tasks with comments so much as go about things the same as usual and let it autocomplete. The time and cognitive load saved adds up massively. I've had to go without it for a bit while my workplace gets its license in order for the corporate version and the personal version has an issue with the proxy, and it's been agonizing going without it again. I almost forgot how much it sucks having to jump to google every other minute, and it was easy to start to take for granted how much context copilot was letting me not have to hold onto in my head. It really lets me work on the problem as opposed to being mired in immaterial details. It feels like I'm at least 2x slower overall without it.
    - rockskon 2y ago
      I don't know about you but LLMs spit out garbage nonsense frequent enough that I can't trust their output in any context I cannot personally verify the validity of.
      - ab5tract 2y ago
        Same here. The Cody autocomplete is so off base all the time that it’s deactivated. I serve Cody a direct question about 1-3 times a week. Of that, it gets maybe 50% correct on the first try. I don’t bother with a second try because by then I’ve already spent the equivalent amount of time looking at the relevant library source code and/or docs would have taken.
    - atq2119 2y ago
      > I almost forgot how much it sucks having to jump to google every other minute Even allowing for some hyperbole, your programming experience is extremely different from mine. Looking anything up outside the IDE, let alone via Google, is by far the exception for me rather than the rule. I've long suspected that this kind of difference explains a lot of the difference in how Copilot is perceived.
- haakonhr 2y ago
  And here you're assuming that making software engineers more productive would be a service to the world. I think the jury is out on that one as well. At least for the majority of software engineering since 2010.
- vrighter 2y ago
  actually, studies seem to show it makes code worse. Just like llms can confidently spout junk, devs using llms confidently check in more bugs.

jimmaswell 2y ago

They already have been. Even just in programming, even just Copilot has been a life changing productivity booster.

wruza 2y ago
Are you sure it’s a life changing productivity booster? Sometimes I look at my projects and wonder how would I explain it to an LLM what this code should have done if it didn’t exist yet. Must be a shitton of boilerplate programming for copilot to be a life-changing experience.
- AYBABTME 2y ago
  You haven't used them enough. Everytime an LLM reduces my search from 1min to 5s, the LLM pays. Just summary features: save me 20min of reading a transcript, turn it into 20s. That's a huge enabler.
  - mecsred 2y ago
    If 20 mins of informations can legitimately be condensed into 20 seconds, it sounds like the original wasn't worth reading in the first place. Could have skipped the llm entirely.
    - bostik 2y ago
      I upvoted you, because I think you have a valid point. The tone is unnecessarily aggressive though. Effective and information-dense communication is really hard. That doesn't mean we should just accept the useless fluff surrounding the actual information and/or analysis. People could learn a lot from the Ignoble Prize ceremony's 24/7 presentation model. Sadly, it seems we are heading towards a future where you may need an LLM to distill the relevant information out of a sea of noise.
      - mecsred 2y ago
        Didn't intend for it to be aggressive, just concise. Spare me from the llm please :)
    - AYBABTME 2y ago
      Think of the summary of a zoom call. Or of a chapter that you're not sure if you care to read or not. Not all content is worth consuming, and not all content is dense.
      - postalrat 2y ago
        If I had a recording of the zoom call I could generate a summary on demand with better tools than were available at the time the zoom call was made.
    - crazygringo 2y ago
      > it sounds like the original wasn't worth reading in the first place But if that's the only place that contained the information you needed, then you have no choice. There's a lot of material out there that is badly written, badly organized, badly presented. LLM's can be a godsend for extracting the information you actually need without wasting 20 minutes wading through the muck.
      - mecsred 2y ago
        Yeah I can see that use case, I just wouldn't trust an LLM to decide "is this worth reading". May as well flip a coin.
  - wruza 2y ago
    Overviews aren’t code though. In code, for me, they don’t pass 80/20 tests well enough, sometimes even on simple cases. (You get 50-80% of an existing function/block with some important context prepended and a comment, let it write the rest and check if it succeeds). It doesn’t mean that LLMs are useless. Or that I am antillamist or a denier - I’m actually an enthusiast. But this specific claim I hear often and don’t find true. Maybe true for repetitive code in boring environments where typing and remembering formats/params over and over is the main issue. Not in actual code. If I paste the actual non-trivial code, it starts deviating fast. And it isn’t too complex, it’s just less like “parallel sort two arrays” and more like “wait for an image on a screenshot by execing scrot (with no sound) repeatedly and passing the result to this detect-cv2.py script and use all matching options described in this ts type, get stdout json as in this ts type, and if there’s a match, wait for the specified anim timeout and test again to get the settled match coords after an animation finishes; throw after a total timeout”. Not a rocket science, pretty dumb shit, but right there they fall flat and start imagining things, heavily. I guess it shines if you ask it to make an html form, but I couldn’t call that life-changing unless I had to make these damn forms all day.
  - andrei_says_ 2y ago
    My experience with overviews is that they are often subtly or not so subtly inaccurate. LLMs not understanding meaning or intent carries risk of misrepresentation.
giraffe_lady 2y ago
"Even just in programming" the jury is still out. None of my coworkers using these are noticeably more productive than the ones who don't. Outside of programming no one gives a shit except scammers and hype chasers.
- JonChesterfield 2y ago
  The people writing articles for journals that aggregate and approximate other sources are in mortal terror of LLMs. Likewise graphic designers and anyone working in (human language) translation. I don't fear that LLMs are going to take my job as a developer. I'm pretty sure they mark a further decrease in the quality and coherence of software, along with a rapid increase in the quantity of code out there, and that seems likely to provide me with reliable employment forever. I'm basically employed in fixing bugs that didn't need to exist in the first place and that seems to cover a lot of software dev.
  - giraffe_lady 2y ago
    They're not scared of LLMs because of anything about LLMs. It's just that everyone with power is publicly horny to delete the remaining middle class jobs and are happy to use LLMs as a justification whether it can functionally replace those workers or not. So it's not that everyone has evaluated chatgpt and cannily realized it can do their job, they're just reading the room.
recursive 2y ago
I've been using copilot for several months. If I could figure out a way to measure its impact on my productivity, I'd probably see a single digit percentage boost in "productivity". This is not life-changing for me. And for some tasks, it's actually worse than nothing. As in, I spend time feeding it a task, and it just completely fails to do anything useful.
- jimmaswell 2y ago
  I've been using it for over a year I think. I don't often feed it tasks with comments so much as go about things the same as usual and let it autocomplete. The time and cognitive load saved adds up massively. I've had to go without it for a bit while my workplace gets its license in order for the corporate version and the personal version has an issue with the proxy, and it's been agonizing going without it again. I almost forgot how much it sucks having to jump to google every other minute, and it was easy to start to take for granted how much context copilot was letting me not have to hold onto in my head. It really lets me work on the problem as opposed to being mired in immaterial details. It feels like I'm at least 2x slower overall without it.
  - rockskon 2y ago
    I don't know about you but LLMs spit out garbage nonsense frequent enough that I can't trust their output in any context I cannot personally verify the validity of.
    - ab5tract 2y ago
      Same here. The Cody autocomplete is so off base all the time that it’s deactivated. I serve Cody a direct question about 1-3 times a week. Of that, it gets maybe 50% correct on the first try. I don’t bother with a second try because by then I’ve already spent the equivalent amount of time looking at the relevant library source code and/or docs would have taken.
  - atq2119 2y ago
    > I almost forgot how much it sucks having to jump to google every other minute Even allowing for some hyperbole, your programming experience is extremely different from mine. Looking anything up outside the IDE, let alone via Google, is by far the exception for me rather than the rule. I've long suspected that this kind of difference explains a lot of the difference in how Copilot is perceived.
haakonhr 2y ago
And here you're assuming that making software engineers more productive would be a service to the world. I think the jury is out on that one as well. At least for the majority of software engineering since 2010.
vrighter 2y ago
actually, studies seem to show it makes code worse. Just like llms can confidently spout junk, devs using llms confidently check in more bugs.