5 ms·

It’s the same for me. I genuinely don’t understand how I can be having such a completely different experience from the people who rave about ChatGPT. Every time

It’s the same for me. I genuinely don’t understand how I can be having such a completely different experience from the people who rave about ChatGPT. Every time I’ve tried it’s been useless. How can some people think it’s amazing and has completely changed how they work, while for me it makes mistakes that a static analyser would catch? It’s not like I’m doing anything remarkable, for the past couple of months I’ve been doing fairly standard web dev and it can’t even fix basic problems with HTML. It will suggest things that just don’t work at all and my IDE catches, it invents APIs for packages. One guy I work with uses it extensively and what it produces is essentially black boxes. If I find a problem with something “he” (or rather ChatGPT) has produced it takes him ages to commune with the machine spirit again to figure out how to fix it, and then he still doesn’t understand it. I can’t help but see this as a time-bomb, how much completely inscrutable shite are these tools producing? In five years are we going to end up with a bunch of “senior engineers” who don’t actually understand what they’re doing? Before people cry “o tempora o mores” at me and make parallels with the introduction of high-level languages, at least in order to write in a high-level language you need some basic understanding of the logic that is being executed.

globular-toast 2y ago
The ones who use it extensively are the same that used to hit up stackoverflow as the first port of call for every trivial problem that came their way. They're not really engineers, they just want to get stuff done.
- phist_mcgee 2y ago
  No ad hominem please.
  - globular-toast 2y ago
    Hmm... calling people "not engineers" is considered an attack now? I'm afraid this is actually revealing your own bias towards engineers. I never said engineers were superior or that we'd be better off with a whole world full of them.
    - phist_mcgee 2y ago
      Nice try mate, but you're not flipping this one on me.
lm28469 2y ago
> How can some people think it’s amazing and has completely changed how they work, while for me it makes mistakes that should a static analyser would catch? There are a lot of code monkeys working on boilerplate code, these people used to rely on stack overflow and now that chatgpt is here it's a huge improvement for them If you work on anything remotely complex or which hasn't been solved 10 times on stack overflow chatgpt isn't remotely as useful
- skinner_ 2y ago
  I work on very complex problems. Some of my solutions have small, standard substeps that now I can reliably outsource to ChatGPT. Here are a few just from last week: - write cvxpy code to find the chromatic number of a graph, and an optimal coloring, given its adjecency matrix. - given an adjecency matrix write numpy code that enumerates all triangle-free vertex subsets. - please port this old code from tensorflow to pytorch: ... - in pytorch, i'd like to code a tensor network defining a 3-tensor of shape (d, d, d). my tensor consists of first projecting all three of its d-dimensional inputs to a k-dimensional vector, typically k=d/10, and then applying a (k, k, k) 3-tensor to contract these to a single number. All were solved by ChatGPT on the first try.
  - lazypenguin 2y ago
    To be honest, these don’t sound like hard problems. These sound like they have very specific answers that I might find in the more specialized stackoverflow sections. These are also the kind of questions (not in this domain) that I’ve found yield the best results from LLMs. In comparison asking an LLM a more project specific question “this code has a race condition where is it” while including some code usually is a crapshoot and really depends if you were lucky enough to give it the right context anyway.
    - skinner_ 2y ago
      Sure, these are standard problems, I’ve said so myself. My point is that my productivity is multiplied by ChatGPT, even if it can only solve standard problems. This is because, although I work on highly non-standard problems (see https://arxiv.org/abs/2311.10069 https://arxiv.org/abs/2311.10069 for an example), I can break them down into smaller, standard components, which ChatGPT can solve in seconds. I never ask ChatGPT "where's the race condition" kind of questions.
ben_w 2y ago
> How can some people think it’s amazing and has completely changed how they work, while for me it makes mistakes that should a static analyser would catch? It’s not like I’m doing anything remarkable, for the past couple of months I’ve been doing fairly standard web dev and it can’t even fix basic problems with HTML. Part of this is, I think, anchoring and expectation management: you hear people say it's amazing and wonderful, and then you see it fall over and you're naturally disappointed. My formative years started off with Commodore 64 basic going "?SYNTAX ERROR" from most typos plus a lot of "I don't know what that means" from the text adventures, then Metrowerks' C compiler telling me there were errors on every line *after but not including* the one where I forgot the semicolon, then surprises in VisualBasic and Java where I was getting integer division rather than floats, then the fantastic oddity where accidentally leaning on the option key on a mac keyboard while pressing minus turns the minus into an n-dash which looked completely identical to a minus on the Xcode default font at the time and thus produced a very confusing compiler error… So my expectations have always been low for machine generated output. And it has wildly exceeded those low expectations. But the expectation management goes both ways, especially when the comparison is "normal humans" rather than "best practices". I've seen things you wouldn't believe... Entire files copy-pasted line for line, "TODO: deduplicate" and all, 20 minute app starts passed off as "optimized solutions." FAQs filled with nothing but Bob Ross quotes, a zen garden of "happy little accidents." I watched iOS developers use UI tests as a complete replacement for storyboards, bi-weekly commits, each a sprawling novel of despair, where every change log was a tragic odyssey. Google Spreadsheets masquerading as bug trackers, Swift juniors not knowing their ! from their ?, All those hacks and horrors… lost in time, Time to deploy. (All true, and all pre-dating ChatGPT). > It will suggest things that just don’t work at all and my IDE catches, it invents APIs for packages. Aye. I've even had that with models forgetting the APIs they themselves have created, just outside the context window. To me, these are tools. They're fantastic tools, but they're not something you can blindly fire-and-forget… …fortunately for me, because my passive income is not quite high enough to cover mortgage payments, and I'm looking for work. > In five years are we going to end up with a bunch of “senior engineers” who don’t actually understand what they’re doing? Yes, if we're lucky. If we're not, the models keep getting better and we don't have any "senior engineers" at all.
williamcotton 2y ago
I found it very useful for writing a lexer and parser for a search DSL and React component recently: https://github.com/williamcotton/search-input-query https://github.com/williamcotton/search-input-query
- zeroonetwothree 2y ago
  Interesting. I implemented something very similar (if not identical) a couple years ago (at work so not open source). I used a simple grammar and standard parser generator. It’s been nice to have the grammar as we’ve made tweaks over the years to change various behaviours and add features.
vrighter 2y ago
first time I tried it, I asked it to find bugs in a piece of very well tested C code. It introduced an off-by-one error by miscounting the number of arguments in an sprintf call, breaking the program. And then proceeded to fail to find that bug that it introduced.
jonas21 2y ago
I think the difference comes down to interacting with it like IDE autocomplete vs. interacting with it like a colleague. It sounds like you're doing the former -- and yeah, it can make mistakes that autocomplete wouldn't or generate code that's wrong or overly complex. On the other hand, I've found that if you treat it more like a colleague, it works wonderfully. Ask it to do something, then read the code and ask follow-up questions. If you see something that's wrong or just seems off, tell it, and ask it to fix it. If you don't understand something, ask for an explanation. I've found that this process generates great code that I often understand better than if I had written it from scratch, and in a fraction of the time. It also sounds like you're asking it to do basic tasks that you already know how to do. I find that it's most useful in tackling things that I don't know how to do. It'll already have read all of the documentation and know the right way to call whatever APIs, etc, and -- this is key -- you can have a conversation with it to clear up anything that's confusing. This takes a big shift in mindset if you've been using IDEs all your life and have expectations of LLMs being a fancy autocomplete. And you really have to unlearn a lot of stuff to get the most out of them.
- LinearEntropy 2y ago
  I'm in the same boat as the person you're responding to. I really don't understand how to get anything helpful out of ChatGPT, or more than anything basic out of Claude. > I've found that if you treat it more like a colleague, it works wonderfully. This is what I've been trying to do. I don't use LLM code completion tools. I'll ask anything from how to do something "basicish" with html & css, and it'll always output something that doesn't work as expected. Question it and I'll get into a loop of the same response code, regardless of how I explain that it isn't correct. On the other end of the scale, I'll ask about an architectural or design decision. I'll often get a response that is in the realm of what I'd expect. When drilling down and asking specifics however, the responses really start to fall apart. I inevitably end up in the loop of asking if an alternative is [more performant/best practice/the language idiomatic way] and getting the "Sorry, you're correct" response. The longer I stay in that loop, the more it contradicts itself, and the less cohesive the answers get. I _wish_ I could get the results from LLMs that so many people seem to. It just doesn't happen for me.
CSMastermind 2y ago
I mean if you're getting no value out of ChatGPT I'd love to have a session seeing how you use it.

globular-toast 2y ago

The ones who use it extensively are the same that used to hit up stackoverflow as the first port of call for every trivial problem that came their way. They're not really engineers, they just want to get stuff done.

phist_mcgee 2y ago
No ad hominem please.
- globular-toast 2y ago
  Hmm... calling people "not engineers" is considered an attack now? I'm afraid this is actually revealing your own bias towards engineers. I never said engineers were superior or that we'd be better off with a whole world full of them.
  - phist_mcgee 2y ago
    Nice try mate, but you're not flipping this one on me.

lm28469 2y ago

> How can some people think it’s amazing and has completely changed how they work, while for me it makes mistakes that should a static analyser would catch? There are a lot of code monkeys working on boilerplate code, these people used to rely on stack overflow and now that chatgpt is here it's a huge improvement for them If you work on anything remotely complex or which hasn't been solved 10 times on stack overflow chatgpt isn't remotely as useful

skinner_ 2y ago
I work on very complex problems. Some of my solutions have small, standard substeps that now I can reliably outsource to ChatGPT. Here are a few just from last week: - write cvxpy code to find the chromatic number of a graph, and an optimal coloring, given its adjecency matrix. - given an adjecency matrix write numpy code that enumerates all triangle-free vertex subsets. - please port this old code from tensorflow to pytorch: ... - in pytorch, i'd like to code a tensor network defining a 3-tensor of shape (d, d, d). my tensor consists of first projecting all three of its d-dimensional inputs to a k-dimensional vector, typically k=d/10, and then applying a (k, k, k) 3-tensor to contract these to a single number. All were solved by ChatGPT on the first try.
- lazypenguin 2y ago
  To be honest, these don’t sound like hard problems. These sound like they have very specific answers that I might find in the more specialized stackoverflow sections. These are also the kind of questions (not in this domain) that I’ve found yield the best results from LLMs. In comparison asking an LLM a more project specific question “this code has a race condition where is it” while including some code usually is a crapshoot and really depends if you were lucky enough to give it the right context anyway.
  - skinner_ 2y ago
    Sure, these are standard problems, I’ve said so myself. My point is that my productivity is multiplied by ChatGPT, even if it can only solve standard problems. This is because, although I work on highly non-standard problems (see https://arxiv.org/abs/2311.10069 https://arxiv.org/abs/2311.10069 for an example), I can break them down into smaller, standard components, which ChatGPT can solve in seconds. I never ask ChatGPT "where's the race condition" kind of questions.

ben_w 2y ago

> How can some people think it’s amazing and has completely changed how they work, while for me it makes mistakes that should a static analyser would catch? It’s not like I’m doing anything remarkable, for the past couple of months I’ve been doing fairly standard web dev and it can’t even fix basic problems with HTML. Part of this is, I think, anchoring and expectation management: you hear people say it's amazing and wonderful, and then you see it fall over and you're naturally disappointed. My formative years started off with Commodore 64 basic going "?SYNTAX ERROR" from most typos plus a lot of "I don't know what that means" from the text adventures, then Metrowerks' C compiler telling me there were errors on every line *after but not including* the one where I forgot the semicolon, then surprises in VisualBasic and Java where I was getting integer division rather than floats, then the fantastic oddity where accidentally leaning on the option key on a mac keyboard while pressing minus turns the minus into an n-dash which looked completely identical to a minus on the Xcode default font at the time and thus produced a very confusing compiler error… So my expectations have always been low for machine generated output. And it has wildly exceeded those low expectations. But the expectation management goes both ways, especially when the comparison is "normal humans" rather than "best practices". I've seen things you wouldn't believe... Entire files copy-pasted line for line, "TODO: deduplicate" and all, 20 minute app starts passed off as "optimized solutions." FAQs filled with nothing but Bob Ross quotes, a zen garden of "happy little accidents." I watched iOS developers use UI tests as a complete replacement for storyboards, bi-weekly commits, each a sprawling novel of despair, where every change log was a tragic odyssey. Google Spreadsheets masquerading as bug trackers, Swift juniors not knowing their ! from their ?, All those hacks and horrors… lost in time, Time to deploy. (All true, and all pre-dating ChatGPT). > It will suggest things that just don’t work at all and my IDE catches, it invents APIs for packages. Aye. I've even had that with models forgetting the APIs they themselves have created, just outside the context window. To me, these are tools. They're fantastic tools, but they're not something you can blindly fire-and-forget… …fortunately for me, because my passive income is not quite high enough to cover mortgage payments, and I'm looking for work. > In five years are we going to end up with a bunch of “senior engineers” who don’t actually understand what they’re doing? Yes, if we're lucky. If we're not, the models keep getting better and we don't have any "senior engineers" at all.

williamcotton 2y ago

I found it very useful for writing a lexer and parser for a search DSL and React component recently: https://github.com/williamcotton/search-input-query https://github.com/williamcotton/search-input-query

zeroonetwothree 2y ago
Interesting. I implemented something very similar (if not identical) a couple years ago (at work so not open source). I used a simple grammar and standard parser generator. It’s been nice to have the grammar as we’ve made tweaks over the years to change various behaviours and add features.

vrighter 2y ago

first time I tried it, I asked it to find bugs in a piece of very well tested C code. It introduced an off-by-one error by miscounting the number of arguments in an sprintf call, breaking the program. And then proceeded to fail to find that bug that it introduced.

jonas21 2y ago

I think the difference comes down to interacting with it like IDE autocomplete vs. interacting with it like a colleague. It sounds like you're doing the former -- and yeah, it can make mistakes that autocomplete wouldn't or generate code that's wrong or overly complex. On the other hand, I've found that if you treat it more like a colleague, it works wonderfully. Ask it to do something, then read the code and ask follow-up questions. If you see something that's wrong or just seems off, tell it, and ask it to fix it. If you don't understand something, ask for an explanation. I've found that this process generates great code that I often understand better than if I had written it from scratch, and in a fraction of the time. It also sounds like you're asking it to do basic tasks that you already know how to do. I find that it's most useful in tackling things that I don't know how to do. It'll already have read all of the documentation and know the right way to call whatever APIs, etc, and -- this is key -- you can have a conversation with it to clear up anything that's confusing. This takes a big shift in mindset if you've been using IDEs all your life and have expectations of LLMs being a fancy autocomplete. And you really have to unlearn a lot of stuff to get the most out of them.

LinearEntropy 2y ago
I'm in the same boat as the person you're responding to. I really don't understand how to get anything helpful out of ChatGPT, or more than anything basic out of Claude. > I've found that if you treat it more like a colleague, it works wonderfully. This is what I've been trying to do. I don't use LLM code completion tools. I'll ask anything from how to do something "basicish" with html & css, and it'll always output something that doesn't work as expected. Question it and I'll get into a loop of the same response code, regardless of how I explain that it isn't correct. On the other end of the scale, I'll ask about an architectural or design decision. I'll often get a response that is in the realm of what I'd expect. When drilling down and asking specifics however, the responses really start to fall apart. I inevitably end up in the loop of asking if an alternative is [more performant/best practice/the language idiomatic way] and getting the "Sorry, you're correct" response. The longer I stay in that loop, the more it contradicts itself, and the less cohesive the answers get. I _wish_ I could get the results from LLMs that so many people seem to. It just doesn't happen for me.

CSMastermind 2y ago

I mean if you're getting no value out of ChatGPT I'd love to have a session seeing how you use it.