4 ms·

The funny thing about the 10x claims (I've even seen folks say 100x) is that results like these would be readily apparent. Even a broad 1.5x productivity increa

The funny thing about the 10x claims (I've even seen folks say 100x) is that results like these would be readily apparent. Even a broad 1.5x productivity increase would be world changing. And yet, we clearly are not seeing this currently. I'm not saying it will never happen, just that it will be very obvious if it does. 5-10% sounds about right to me.

larve 3y ago
The problem with these claims is that you can't really quantify a "10x" change. I have found a lot of emergent benefits to using LLMs. For example, because my cognitive load of wrangling APIs and understanding and refactoring legacy code and all the other nonsense of my day to day can be so heavily delegated, I actually feel refreshed at the end of the day, and can bang out a decently chunked feature on my opensource software on the couch (admittedly also boilerplate heavy code). This means that I went from 0 opensource commits to 4000 since chatgpt came out. Not just that, but I've gotten not only more adventurous, but have the time to consider doing drastic refactors and spend much more time thinking about my software. I won't call it 10x or 100x, because that wouldn't mean anything, but surely it is a paradigm shift for me, completely world changing.
- pm_me_your_quan 3y ago
  I'd be really curious if you're willing to expand more on how it has helped with those workflows. Do you copy/paste chunks in and ask it to explain them? Have it try to refactor them and then clean up?
  - larve 3y ago
    For legacy code: - generate comments (hit or miss, but at least it can rewrite my random notes into consistent notes) - generate type annotations - refactor "broadly" (say, "rename all variables to match the following style" or "turn this class into a dataclass like XXX" or "transform the SQL queries into builder queries using XYZ"). Often requires some manual work but it gets a lot of tedious stuff out of the way - reverse-engineer clean API specs by just pasting in recorded HTTP logs - clean up logs into proper enums by generating the regexps - write CLI tools to probe the system (say, CLI tool to exercise the APIs mentioned above) - generate synthetic test data - transform HTML garbage into using a modern component system / react - transform legacy react/js into consistent redux actions - generate SQL queries at the speed of mouth I could go on forever...
    - kjkjadksj 3y ago
      Are you testing chatgpts output in any way? I’ve considered using it for tasks but after hearing all the talk of how it can write good looking code that ends up not working as you might expect, I started wondering if the time savings from generating that block are wasted from interpreting and testing.
      - simonw 3y ago
        I have access to ChatGPT Code Interpreter mode, where it can both write Python and then execute it. I use that to write code all the time, because ChatGPT can write the code, run it, get an error, then re-write the code to address the error. Here are two recent transcripts whereI used it in this way: - https://chat.openai.com/share/b062955d-3601-4051-b6d9-80cef9228233 https://chat.openai.com/share/b062955d-3601-4051-b6d9-80cef9... - https://chat.openai.com/share/b9873d04-5978-489f-8c6b-4b948db7724d https://chat.openai.com/share/b9873d04-5978-489f-8c6b-4b948d...
    - aleph_minus_one 3y ago
      > - generate SQL queries at the speed of mouth Because of the points this is the nearest to the work that some colleagues do, I anakyze this point (but you could ask similar questions about many of the other points): In my experience, writing correct SQL queries (which often tend to be quite non-trivial because of the internal complexity of the projects) typically involves a lot of knowledge about the whole system that my colleagues and I work on. Even if I could copy-paste this information, written down once, into the AI chat window: - I seriously doubt that any of these AI chat bots would be able to generate a remotely decent SQL query based on this information, if only because these SQL queries look really different from what you would see in typical CRUD web applications (for a very instructive example think into the direction of ETL for unifying historically separated lines of business where you often have lots of discussions with the respective colleagues to clear up very subtle details what the code is actually supposed to do in some strange boundary cases that exist because of some historical reasons (which one wants to get rid of)) - even explaining what the SQL query is supposed to do would in my opinion take more time than simply writing it down. Even ignoring the previous point: it is very typical that explaining in sufficient detail what the code is supposed to do would take far more time than simply writing it. A lot of programming work is not writing some scaffolding of some CRUD app or implementing a textbook algorithm.
- dontupvoteme 3y ago
  Side projects are also where I found it invaluable. I'm spending most of my time thinking about actual problems in linguistics and language rather than how to setup an NLTK pipeline or docker image for your own wikimedia database or what not I never heard of type hints before but ironically I use them on everything now, since it's easier to lint.
dmarchand90 3y ago
I dunno computers should be world changing. And they are. Yet the lives of ordinary people have hardly improved and are arguably worse since the 1980s
- JimtheCoder 3y ago
  Worse in what sense? Computers have done amazing things for humanity. I would not want to go back to a computerless world...
  - dmarchand90 3y ago
    If you work in the US for fang or similar your life is probably much better. I love computers. I love what they do for recreation and keeping in touch long distance. But work hours are longer. Education and housing is no longer affordable for most people. In the US life expectancy is beginning to drop. You would think people would have easier less stressful lives with greater financial security. But it's not the case for 99% of us. Quick example. A doctor in the 90s would expect to look forward to less time wasted on paperwork thanks to computers but administration work is increasing. Likewise most professors at most universities will tell you they spend more and more time on administration. Shouldn't this be one of the primary things a computer could address. My point is great gains brought by a technology such as computers may not translate to great end user benefit. Largely due to compensating inefficiencies elsewhere in the system. So for example. It may be that chatgpt makes people 10x more productive. But if management gives bad direction it doesn't matter. If most software being developed is redundant then it doesn't matter. If most software is just making ads run faster then it doesn't matter. The technology needs to be appropriately directed to be of noticeable social benefit.
    - waldohatesyou 3y ago
      I think the catch here is that we need to know what makes admin work increase. While I can’t comment on education, at least for housing in Canada this is all a result of NIMBYism and government intervention. In other words, we could have slouched towards utopia much further than we have if it were not for NIMBYs.
sebzim4500 3y ago
I don't think anyone is claiming that chatgpt has made everyone 100x more efficient. Either they are retarded or you misunderstood them.

larve 3y ago

The problem with these claims is that you can't really quantify a "10x" change. I have found a lot of emergent benefits to using LLMs. For example, because my cognitive load of wrangling APIs and understanding and refactoring legacy code and all the other nonsense of my day to day can be so heavily delegated, I actually feel refreshed at the end of the day, and can bang out a decently chunked feature on my opensource software on the couch (admittedly also boilerplate heavy code). This means that I went from 0 opensource commits to 4000 since chatgpt came out. Not just that, but I've gotten not only more adventurous, but have the time to consider doing drastic refactors and spend much more time thinking about my software. I won't call it 10x or 100x, because that wouldn't mean anything, but surely it is a paradigm shift for me, completely world changing.

pm_me_your_quan 3y ago
I'd be really curious if you're willing to expand more on how it has helped with those workflows. Do you copy/paste chunks in and ask it to explain them? Have it try to refactor them and then clean up?
- larve 3y ago
  For legacy code: - generate comments (hit or miss, but at least it can rewrite my random notes into consistent notes) - generate type annotations - refactor "broadly" (say, "rename all variables to match the following style" or "turn this class into a dataclass like XXX" or "transform the SQL queries into builder queries using XYZ"). Often requires some manual work but it gets a lot of tedious stuff out of the way - reverse-engineer clean API specs by just pasting in recorded HTTP logs - clean up logs into proper enums by generating the regexps - write CLI tools to probe the system (say, CLI tool to exercise the APIs mentioned above) - generate synthetic test data - transform HTML garbage into using a modern component system / react - transform legacy react/js into consistent redux actions - generate SQL queries at the speed of mouth I could go on forever...
  - kjkjadksj 3y ago
    Are you testing chatgpts output in any way? I’ve considered using it for tasks but after hearing all the talk of how it can write good looking code that ends up not working as you might expect, I started wondering if the time savings from generating that block are wasted from interpreting and testing.
    - simonw 3y ago
      I have access to ChatGPT Code Interpreter mode, where it can both write Python and then execute it. I use that to write code all the time, because ChatGPT can write the code, run it, get an error, then re-write the code to address the error. Here are two recent transcripts whereI used it in this way: - https://chat.openai.com/share/b062955d-3601-4051-b6d9-80cef9228233 https://chat.openai.com/share/b062955d-3601-4051-b6d9-80cef9... - https://chat.openai.com/share/b9873d04-5978-489f-8c6b-4b948db7724d https://chat.openai.com/share/b9873d04-5978-489f-8c6b-4b948d...
  - aleph_minus_one 3y ago
    > - generate SQL queries at the speed of mouth Because of the points this is the nearest to the work that some colleagues do, I anakyze this point (but you could ask similar questions about many of the other points): In my experience, writing correct SQL queries (which often tend to be quite non-trivial because of the internal complexity of the projects) typically involves a lot of knowledge about the whole system that my colleagues and I work on. Even if I could copy-paste this information, written down once, into the AI chat window: - I seriously doubt that any of these AI chat bots would be able to generate a remotely decent SQL query based on this information, if only because these SQL queries look really different from what you would see in typical CRUD web applications (for a very instructive example think into the direction of ETL for unifying historically separated lines of business where you often have lots of discussions with the respective colleagues to clear up very subtle details what the code is actually supposed to do in some strange boundary cases that exist because of some historical reasons (which one wants to get rid of)) - even explaining what the SQL query is supposed to do would in my opinion take more time than simply writing it down. Even ignoring the previous point: it is very typical that explaining in sufficient detail what the code is supposed to do would take far more time than simply writing it. A lot of programming work is not writing some scaffolding of some CRUD app or implementing a textbook algorithm.
dontupvoteme 3y ago
Side projects are also where I found it invaluable. I'm spending most of my time thinking about actual problems in linguistics and language rather than how to setup an NLTK pipeline or docker image for your own wikimedia database or what not I never heard of type hints before but ironically I use them on everything now, since it's easier to lint.

dmarchand90 3y ago

I dunno computers should be world changing. And they are. Yet the lives of ordinary people have hardly improved and are arguably worse since the 1980s

JimtheCoder 3y ago
Worse in what sense? Computers have done amazing things for humanity. I would not want to go back to a computerless world...
- dmarchand90 3y ago
  If you work in the US for fang or similar your life is probably much better. I love computers. I love what they do for recreation and keeping in touch long distance. But work hours are longer. Education and housing is no longer affordable for most people. In the US life expectancy is beginning to drop. You would think people would have easier less stressful lives with greater financial security. But it's not the case for 99% of us. Quick example. A doctor in the 90s would expect to look forward to less time wasted on paperwork thanks to computers but administration work is increasing. Likewise most professors at most universities will tell you they spend more and more time on administration. Shouldn't this be one of the primary things a computer could address. My point is great gains brought by a technology such as computers may not translate to great end user benefit. Largely due to compensating inefficiencies elsewhere in the system. So for example. It may be that chatgpt makes people 10x more productive. But if management gives bad direction it doesn't matter. If most software being developed is redundant then it doesn't matter. If most software is just making ads run faster then it doesn't matter. The technology needs to be appropriately directed to be of noticeable social benefit.
  - waldohatesyou 3y ago
    I think the catch here is that we need to know what makes admin work increase. While I can’t comment on education, at least for housing in Canada this is all a result of NIMBYism and government intervention. In other words, we could have slouched towards utopia much further than we have if it were not for NIMBYs.

sebzim4500 3y ago

I don't think anyone is claiming that chatgpt has made everyone 100x more efficient. Either they are retarded or you misunderstood them.