4 ms·

I've used GPT-4 for programming since it came out and it's massively improved my productivity, despite being frustrating at times. It quickly forgets details an

I've used GPT-4 for programming since it came out and it's massively improved my productivity, despite being frustrating at times. It quickly forgets details and starts hallucinating, so I have to constantly remind it by pasting in code. After a few hours it gets so confused I have to start a new chat to reset things. I've been using Claude pretty intensively over the last week and it's so much better than GPT. The larger context window (200k tokens vs ~16k) means that it can hold almost the entire codebase in memory and is much less likely to forget things. The low request quota even for paid users is a pain though.

bredren 3y ago
Which languages are you primarily writing in? Do you mean you are throttled by it more often? (~“You must wait X mins to make more queries”)
varispeed 3y ago
It is still remarkably better than working with junior developers. I can ask it to code something based on specification and most of the time it will do better job than if I send the same task to a human. It often makes some small mistakes, but little nudge here and there corrects them, whereas with human you have to spend a lot of time explaining why something is wrong and why this and that way would rectify it. The difference is probably because GPT has access to a vast array of knowledge a human can't reasonably compete with.
- aiisjustanif 3y ago
  > It is still remarkably better than working with junior developers. Definitely not all junior developers. I have yet to see it do well at handling code migrations, updating APIs, writing end to end tests, and front end code changes with existing UX specifications to name a few things.
  - greenavocado 3y ago
    Wow, you have good junior developers.
    - throwaway35777 3y ago
      Work at a good company and you'll find that the juniors are pretty capable.
- throwaway35777 3y ago
  > It is still remarkably better than working with junior developers It really bugs me when people imply AI will replace only certain devs based on their title. Seniority does not equal skill. There are plenty of extremely talented "junior" developers who don't have the senior title because of a slow promo process and minimum time in role requirements. They can and do own entire projects and take on senior-level responsibilities. I've also worked with a "senior" dev who struggled for over a month to make a logging change.
- bigcoke 3y ago
  that's more or less how I use gpt-4, "how to <describe some algorithm in some language><maybe some useful context with a short code example>", most of time it output something useful that I can work with, but if a junior work is to be my gpt-4, that's a waste of human resource.
cosmojg 3y ago
> The low request quota even for paid users is a pain though. Why not use the API directly? Their Workbench interface is pretty nifty if you don't feel like hooking the API up to your tools of choice.
- danielbln 3y ago
  The workbench UI sucks, what's nifty about it? It's cumbersome and slow. I would recommend using a ChatUI (huggingface ChatUI, or https://github.com/lobehub/lobe-chat https://github.com/lobehub/lobe-chat) and use the API that way.
  - cosmojg 3y ago
    Right, but it's certainly easier for people who might not even know what "API" stands for, and that's quite nifty. As far as self-hosted frontends go, I can personally recommend SillyTavern[1] in the browser, ChatterUI[2] on mobile, and ShellGPT[3] for CLI. LobeChat looks pretty cool, though! I'll definitely check it out. [1] https://github.com/SillyTavern/SillyTavern https://github.com/SillyTavern/SillyTavern [2] https://github.com/Vali-98/ChatterUI https://github.com/Vali-98/ChatterUI [3] https://github.com/TheR1D/shell_gpt https://github.com/TheR1D/shell_gpt
biddit 3y ago
> The larger context window (200k tokens vs ~16k) Just to add some clarification - the newer GPT4 models from OpenAI have 128k context windows[1]. I regularly load in the entirety of my React/Django project, via Aider. 1. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turb...
- danielbln 3y ago
  GPT-4 has much worse recall compared to Claude-3 though, compare these two haystack tests: https://github.com/gkamradt/LLMTest_NeedleInAHaystack/raw/main/img/GPT_4_testing.png https://github.com/gkamradt/LLMTest_NeedleInAHaystack/raw/ma... https://cdn.arstechnica.net/wp-content/uploads/2024/03/claude3_chart6.-640x317.png https://cdn.arstechnica.net/wp-content/uploads/2024/03/claud...
  - aubanel 3y ago
    Be aware the haystack test is not good at all (in its current form). It's a single piece of information inserted in the same text each time, a very poor measurement of how well the LLM can retrieve info.
    - bufferoverflow 3y ago
      Seems like a very good test for recall.
      - aubanel 3y ago
        Even in the most restrictive définition of recall as in "retrieve a short contiguous piece of information inside an unrelated context", it's not that good. It's always the exact same needle inserted in the exact same context. Not the slightest variation apart from the location of the needle. Then if you want to test for recall of sparse information or multi-hop information, it's useless.
        3y ago
ramijames 3y ago
How do both compare to copilot?
lfkdev 3y ago
Just use the API, do as many requests as you want
acedTrex 3y ago
I see so many people say that llms have improved their programming and i am truly baffled by how. They are useful on occasion and i do use chatgpt and copilot daily but if they disappeared today it would be at worst a minor annoyance.
- dontreact 3y ago
  It’s a tool you learn to use and improve with. How many hours of practice have you given the tool? Personally I’ve been practicing for almost 3 years now going back to CoPilot so I would guess I have at very minimum a few hundred hours of thinking about how (what to prompt for, what to check the output for) and probably more importantly when (how to decompose the task I am doing into parts for which some the LLM can reliably execute) to use the tool.
  - stavros 3y ago
    I don't think it's a failure of the prompting, but of the tool itself. It just isn't good for any higher-level tasks, and those I know how to do anyway. It's good for menial API work, as a sibling comment says, and using it I can avoid reading API docs, but that's about it. I tried to have it rewrite in Go a small Python script (a hundred or so lines) that I wrote, and it basically refused, saying "here's the start, you write the rest now". As others said, if it went away it'd be a minor inconvenience of me having to read a bit more documentation.
    - dontreact 3y ago
      Yes to me that’s obviously not a good use, and I have experience to know not to attempt hundred line rewrites of files. I order to reflect on my mental about this. It’s that i more often break down parts problem into sub routines that are 30 or so lines. These are nailed about 95% of the time and when there’s something wrong it’s obvious.
      - dontreact 3y ago
        I also wonder if it’s a language effect and the models are just way better at python
danw1979 3y ago
The lack of API credits for Claude Pro subscribers is also a bit of an oversight. I’d like to be able to consume my daily quota via the API as well as the chatbot.

bredren 3y ago

Which languages are you primarily writing in? Do you mean you are throttled by it more often? (~“You must wait X mins to make more queries”)

varispeed 3y ago

It is still remarkably better than working with junior developers. I can ask it to code something based on specification and most of the time it will do better job than if I send the same task to a human. It often makes some small mistakes, but little nudge here and there corrects them, whereas with human you have to spend a lot of time explaining why something is wrong and why this and that way would rectify it. The difference is probably because GPT has access to a vast array of knowledge a human can't reasonably compete with.

aiisjustanif 3y ago
> It is still remarkably better than working with junior developers. Definitely not all junior developers. I have yet to see it do well at handling code migrations, updating APIs, writing end to end tests, and front end code changes with existing UX specifications to name a few things.
- greenavocado 3y ago
  Wow, you have good junior developers.
  - throwaway35777 3y ago
    Work at a good company and you'll find that the juniors are pretty capable.
throwaway35777 3y ago
> It is still remarkably better than working with junior developers It really bugs me when people imply AI will replace only certain devs based on their title. Seniority does not equal skill. There are plenty of extremely talented "junior" developers who don't have the senior title because of a slow promo process and minimum time in role requirements. They can and do own entire projects and take on senior-level responsibilities. I've also worked with a "senior" dev who struggled for over a month to make a logging change.
bigcoke 3y ago
that's more or less how I use gpt-4, "how to <describe some algorithm in some language><maybe some useful context with a short code example>", most of time it output something useful that I can work with, but if a junior work is to be my gpt-4, that's a waste of human resource.

cosmojg 3y ago

> The low request quota even for paid users is a pain though. Why not use the API directly? Their Workbench interface is pretty nifty if you don't feel like hooking the API up to your tools of choice.

danielbln 3y ago
The workbench UI sucks, what's nifty about it? It's cumbersome and slow. I would recommend using a ChatUI (huggingface ChatUI, or https://github.com/lobehub/lobe-chat https://github.com/lobehub/lobe-chat) and use the API that way.
- cosmojg 3y ago
  Right, but it's certainly easier for people who might not even know what "API" stands for, and that's quite nifty. As far as self-hosted frontends go, I can personally recommend SillyTavern[1] in the browser, ChatterUI[2] on mobile, and ShellGPT[3] for CLI. LobeChat looks pretty cool, though! I'll definitely check it out. [1] https://github.com/SillyTavern/SillyTavern https://github.com/SillyTavern/SillyTavern [2] https://github.com/Vali-98/ChatterUI https://github.com/Vali-98/ChatterUI [3] https://github.com/TheR1D/shell_gpt https://github.com/TheR1D/shell_gpt

biddit 3y ago

> The larger context window (200k tokens vs ~16k) Just to add some clarification - the newer GPT4 models from OpenAI have 128k context windows[1]. I regularly load in the entirety of my React/Django project, via Aider. 1. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turb...

danielbln 3y ago
GPT-4 has much worse recall compared to Claude-3 though, compare these two haystack tests: https://github.com/gkamradt/LLMTest_NeedleInAHaystack/raw/main/img/GPT_4_testing.png https://github.com/gkamradt/LLMTest_NeedleInAHaystack/raw/ma... https://cdn.arstechnica.net/wp-content/uploads/2024/03/claude3_chart6.-640x317.png https://cdn.arstechnica.net/wp-content/uploads/2024/03/claud...
- aubanel 3y ago
  Be aware the haystack test is not good at all (in its current form). It's a single piece of information inserted in the same text each time, a very poor measurement of how well the LLM can retrieve info.
  - bufferoverflow 3y ago
    Seems like a very good test for recall.
    - aubanel 3y ago
      Even in the most restrictive définition of recall as in "retrieve a short contiguous piece of information inside an unrelated context", it's not that good. It's always the exact same needle inserted in the exact same context. Not the slightest variation apart from the location of the needle. Then if you want to test for recall of sparse information or multi-hop information, it's useless.
      - 3y ago

ramijames 3y ago

How do both compare to copilot?

lfkdev 3y ago

Just use the API, do as many requests as you want

acedTrex 3y ago

I see so many people say that llms have improved their programming and i am truly baffled by how. They are useful on occasion and i do use chatgpt and copilot daily but if they disappeared today it would be at worst a minor annoyance.

dontreact 3y ago
It’s a tool you learn to use and improve with. How many hours of practice have you given the tool? Personally I’ve been practicing for almost 3 years now going back to CoPilot so I would guess I have at very minimum a few hundred hours of thinking about how (what to prompt for, what to check the output for) and probably more importantly when (how to decompose the task I am doing into parts for which some the LLM can reliably execute) to use the tool.
- stavros 3y ago
  I don't think it's a failure of the prompting, but of the tool itself. It just isn't good for any higher-level tasks, and those I know how to do anyway. It's good for menial API work, as a sibling comment says, and using it I can avoid reading API docs, but that's about it. I tried to have it rewrite in Go a small Python script (a hundred or so lines) that I wrote, and it basically refused, saying "here's the start, you write the rest now". As others said, if it went away it'd be a minor inconvenience of me having to read a bit more documentation.
  - dontreact 3y ago
    Yes to me that’s obviously not a good use, and I have experience to know not to attempt hundred line rewrites of files. I order to reflect on my mental about this. It’s that i more often break down parts problem into sub routines that are 30 or so lines. These are nailed about 95% of the time and when there’s something wrong it’s obvious.
    - dontreact 3y ago
      I also wonder if it’s a language effect and the models are just way better at python

danw1979 3y ago

The lack of API credits for Claude Pro subscribers is also a bit of an oversight. I’d like to be able to consume my daily quota via the API as well as the chatbot.