8 ms·

Seriously, are any of you "increasing output" with AI? I get like 10 good lines of code a day from Copilot. I don't even try ChatGPT anymore, as its only usefu

Seriously, are any of you "increasing output" with AI? I get like 10 good lines of code a day from Copilot. I don't even try ChatGPT anymore, as its only useful if you're writing boilerplate. Everything thats actually hard about my job requires the full context of the rest of the codebase, within repo and without, and architectural knowledge of the problems we're trying to solve. I read another post here yesterday that said "90% of my skillset just became useless" in regards to AI. I don't understand how? Do people spend 90% of their time writing blank HTML templates or empty react components? Maybe people are just extrapolating. "90% of my skillset will become useless soon". I don't know.

peoplearepeople 4y ago
There must be some people out there with skillsets so low/shallow that these AI tools really do seem like a productivity boost
- sandspar 4y ago
  What a titillating thought, how much more skillful and deep we are than all those people!
bobobob420 4y ago
Yes for unit tests lol
mnky9800n 4y ago
yes, this is my experience as well. but any time i mention it, it seems like people don't want to hear that the AI revolution is really just "slightly better intellisense some of the time."
bmelton 4y ago
I'm not a content creator, but while I would never rely solely on AI to write content that I expected to be useful, I do use it for drumming up ideas and have found that (at least for me) its most effective utility is in generating outlines. When I'm writing things informally (like HN posts) rarely allows me to do so clearly and linearly, and I am constantly either leaving out entire paragraphs that I just assumed I'd already written or am over-responding to things because I keep seeing blanks and just keep filling them. ChatGPT (and presumably all the copywriting tools that rely on it) is much better than I am at giving me a checklist of touchpoints to hit.
tristor 4y ago
Their company writes blog spam used for "SEO" for scam products. It's completely reasonable to expect ChatGPT to increase output, because the prompt could be as simple as "Give me a 2500 word blog post in an optimistic tone of voice about human muscle development, and reference the sources used on Wikipedia. During the article link to X at least 6 times." and voila, one customer's scam product down. Repeat 50x.
whstl 4y ago
I kinda did, I guess. I use ChatGPT to generate backend classes or Typescript types from JSON data, to generate unit tests from files, sometimes it also does some browser tests. I also write some parsers so it helps extending syntax a bit (but it's not too good at that). It also helps extracting localization keys, at detecting un-localized keys in HTML templates. If a project has a lot of boilerplate, then it's definitely more than 10 lines per day. But not exactly "good lines". :)
dpkirchner 4y ago
I think I am. I don't know how much of my time is spent writing boilerplate but it feels like a lot. Reducing that time is a big win. Recently I've successfully used ChatGPT to generate: * TS declaration files for existing JS * test fixtures that cover boundary conditions * a regular expression that includes backtracking and optional named capture groups The last one was pretty neat. I gave it sample data and it gave me a regular expression along with JS code I could copy into a node REPL for testing. It didn't get the task right immediately but we were able to iterate until we got the right solution. My regex-fu is weak when it comes to backtracking so this saved me a lot of time.
DonHopkins 4y ago
I've actually found it useful for "decreasing output", or distilling uncharted mountains of my own bullshit down into neatly organized buckets of bullshit. The stuff I want to write about, I've usually written about many times in different contexts over many years. I can stuff all that into GPT-3, and it will summarize and organize it into a nice outline that I can edit, populate, and elaborate myself by talking or typing. So it's all my stuff, but GPT-3 has more of a subtractive and organizational role than a generative role. It does a pretty good job converting lots of redundant huge walls of text, flame war campaigns, and axe grinding diatribes, into concise outlines, introductions, summaries, lists, comparisons, etc. I'm using it to organize, remix, and synthesize lots of papers, articles, postings, emails, conversations, notes, transcripts of video demos and talks, and other things I've written into a coherent 8 module "Vaster Class in Game Design". (Like a Master Class, but Vaster!)
humandeveloper 4y ago
Same for me. I primarily work on mobile applications. ChatGPT is only good for generic tasks that can be explained without additional context which is by far the easiest part of my job.
dkrich 4y ago
No. To me the entire generative AI play is all part of a repetitive cycle. First people are apt to believe anything and get burned in a bubble. Then they are skeptical of everything and get left out of the next actual sea change. Then a new development comes along and they are trained to believe it will turn into a sea change also. I'm saying this because so many people who don't seem to have a great track record of being of the forefront of sea changes are making all sorts of wild predictions about what this will bring. Many also seem to want to believe others will get caught up in a bubble and looking for opportunities to exploit that for personal gain. I realize this is a bit of a rant but it's the lens I've come to view it through based on personal experience. Generative AI seems to be the first useable case for AI but the holy grail- full automation and predictive capabilities still seem quite a ways off because they are much harder problems to solve.
elicash 4y ago
To your first question, I am. And higher quality, not just quantity. As far as code, it's admittedly not the main part of my job and not my biggest strength, so yeah I need the help. I ask it to solve small problems, not write the entire thing. For my writing, most recently I used it to incorporate social pressure into membership emails (peer influence and social proof). On your last point, my first job in politics was creating the MySpace theme for a party committee; I used to create stuff in Flash; most of the toolsets I've used for my work no longer exist. The list of useless skills I possess is endless and my useless skills are way above 90%. You just have to learn new stuff and it's fine.
- pathartl 4y ago
  Agreed, it's good in small chunks. Sometimes I'll feed it a prompt to generate new code to double check my approach is valid. Sometimes it's shown I am the better architect, other times it's made me aware of parts of an API that I did not know about. Ironically, it's helped me the most in writing recursive code.
JauntyHatAngle 4y ago
I work in devops, and while chat gpt can write some starting terraform code or suggest aws architecture to some degree, the reality is my job is more about figuring out all the hairy details and how to smooth them out or add to them, in an existing world with a domain knowledge required. Chatgpt isn't even remotely useful for the money making part of my work. If all I did was write boilerplate pipelines I'd be worth diddly squat.
achenatx 4y ago
my productivity has increased tremendously with chatgpt For example, I was getting an error that a method x was missing on var of type MyObject. In debugging mode I could see that they type was MyObject and the attribute x was there. I paste the small snippet of code and the error and chapgpt immediately identified that the method call was actually returning a collection. I just didnt notice that the variable was a collection of type MyObject and not an actual MyObject. When Im looking at other people's code I can paste it in and chatgpt will give an opinion on what is happening with the code. I was creating UI components from scratch the other day and needed a word wrapping algorithm. ChatGPT was able to write one that didnt quite work, but was most of the way there. When I looking at unfamiliar APIs it is much faster to ask chatgpt than to do google searches. Even for things like how many police officers did austin have in 2008 are much faster with chatgpt. Ill ask for the reference (always broken) but enough of the reference is correct that I can google and get to the reference. Lately chat gpt has been disconnecting forcing me to reconnect if I go idle for more than about 5 minutes, that has been annoying.
kraftman 4y ago
What I'm finding it most useful for is grokking some new subject rapidly, so the output multiplier is directly related to how well i know the subject im working on. At work, I know the tech stack pretty well so I'm not finding it that valuable, i use it to remind myself of bits and peices here and there. In my free time ive been working in some areas and frameworks i dont know at all and the output increase really is crazy. i was playing around with pixijs and very rapidly had a basic UI cobbled together that would have taken me at least 4-5x longer if id had to read through different docs, blogs, etc. its making me much more willing to try new frameworks and ideas out that i wouldnt have before because of how easily i can get over blockers in my concept of how something works.
- MattPalmer1086 4y ago
  That is exactly what I have found to be most useful. Not saying there arent other good uses, but boy, does it accelerate trying out new things.
surgical_fire 4y ago
> Seriously, are any of you "increasing output" with AI? Depends on what I'm doing. Professionally, I'm finding it very good for a couple of things: - Exploring frameworks and libraries for languages I'm already familiar with - Writing documentation for code that I've written (I don't use this at my workplace to avoid leaking internal code, but I use it as a way to write small tutorials based on side projects I do on my own) For writing code I find it very unreliable, even for boilerplate.
Turing_Machine 4y ago
I found it really helpful for porting an extension I'd written for Atom (now unsupported/defunct) to VS Code. Now, if I were writing VS Code extensions on a daily basis, I'd want to spend the time to learn all the details, but (ideally) I'll never have to write one again. This is the only one I ever had to write for Atom, so it's a decent bet. And yeah, most of the code was boilerplate-- but still ChatGPT saved me from having to learn how to write that boilerplate. At least a day of time, I'd reckon. Possibly several days. The task was to recognize the Scheme expression (atom or pair) preceding the current caret position, sign it using a password hash, send it to a remote Scheme REPL for evaluation, then insert and select the result of the evaluation in the editor. ChatGPT essentially wrote everything for me, except the part that parsed out the Scheme expression. It wrote a really horrible regex for that. Fortunately I already had a proper parser implemented in the previous Atom extension, which dropped right in.
danShumway 4y ago
I think there's a very narrow window where I'm getting useful results out of it, enough that I'm not completely writing it off as a way to help out with some productive work. I do expect I'll make use of it occasionally while programming in the future. But all of the times I've used it, it hasn't been an extremely clear productivity boost (certainly not at the x2/x10 level), and I do need to babysit it way more than people are suggesting. I go back and forth in my opinion on it. There are some topics where I get useful sessions, there are some topics where it's a mess. The problem is I don't know in advance which are which. It seems to be most useful in scenarios where I either need some extra motivation to get over the initial hump of starting a task (making an task feel more approachable is I think a great use-case), or where I know just enough about a task to know if it's generating nonsense, but not enough to be able to knock out the same results in less time myself. The other problem is that in areas where I'm not immediately familiar with what's going on, I sometimes have to spend a lot of time debugging its code. And reading and debugging other people's code is (to me) a lot harder than writing code myself. So there's an inherent slowdown there where I have to check its work as if I am reading a junior programmer's code and checking it for errors. Code reviews are annoying and take a lot of mental effort to do well. You can ask GPT-4 to explain itself, but there's no way to know without actually stepping through the code if its explanation is correct. I also worry that it's railroading me a bit. I had it prototype an algorithm for me and debugged it and helped it expand the algorithm and get it working with the results I wanted. Only afterwards did I realize there was a much simpler way to get the result I wanted without invoking the algorithm at all. If I wasn't in a conversational loop with an AI focused on getting its code to work, I might have noticed that earlier. But my brain was only focused on debugging the code for that task, it wasn't focused on going back and rethinking the algorithm and cutting out the useless bits. BUT... sometimes it seems helpful with prototyping? So it's not all bad. I don't necessarily want to dismiss it entirely. My take is potentially useful, and for some people who are particularly suited for this kind of rubber-duck peer-coding style of work, maybe a big increase in productivity? But certainly not universally useful for every programmer, and I can still see the seams. GPT-4 is significantly better than GPT-3, but it's not as good as people have hyped it to me, and I think people are vastly over-estimating its capabilities. I went in skeptical, but even with that attitude from what people had told me beforehand I still expected GPT-4 to be better than it is. Am I worried this is going to steal my job? No. That's honestly a kind of laughable idea to me after seeing the kind of code it writes and how unreliable it is. ---- It's hard to put this into words, but it also kind of hints to me at a much better system, and I can see problems here around the auto-regressive nature of the LLM and the need for alignment training and around the whole structure of the thing and the slowness of generating text over and over, and the sort of forced linearity of the chat session that makes me think that maybe GPT-4 is just kind of hinting at a debugging/prototyping methodology that would be a massive improvement to productivity but that doesn't necessarily involve LLMs or involves them to a much lesser degree. It's in some ways a very clumsy/brutalist approach to simulating a feedback repl and accessing documentation, and because it's clumsy/brutalist it over-emphasizes the natural language aspect. And I'm curious to see if future projects can identify what specifically is helpful here and deliver that without all of the inherent downsides of needing to use a conversational model. That's all ignoring the privacy/data concerns, but they're a separate problem.
Balgair 4y ago
Yes, by ~20x. Copy-pasting a comment of mine from a few days ago: We decided at work to run a little experiment with GPT3 to see if/how-much it was 'worth it'. Since baseball is back and most of us are fans, we decided to write a baseball simulator. We each had a Friday afternoon to write one up. Half of us got to use the free GPT3, and half had just regular googling. After the jam, we'd compare notes at the bar and see what the difference, if any, was. Holy cow, was there ever a difference. Those without GPT3 got pretty far. Got the balls and strikes and bases and 9 innings. Most got extra innings down. One even tried the integration with ERA and batting stats in the probabilities of an event occurring but was unable to get it done. The GPT3 group was estimated to be 2 weeks worth of work ahead of the googling group. Turns out, there is a whole python library for baseball simulations and statistics. The googling group didn't find that, but GPT3 just prompted it outright on the first query for everyone using it. This group got the basics of the game done in ~30 minutes. Managed to get integration with actual MLB statistics. Built somewhat real physics simulators of balls in play and distances, adjusted for temperature and altitude. Not all of them at once, but a lot of really great stuff. Aside: Did you know that MLB publishes, in real time, all 6 degrees of freedom for a ball, from where it leaves a pitchers hand to where a catcher/batter interacts with it? They put out the spin rates in three axes! Wild stuff. Our conclusions were that it's totally 'worth it' and is a ~20x multiplier in coding speed. It spits out a lot of really bad code, but it gets the skeletons out very quickly and just rockets you to the crux of the problems. For example: it gave out a lot of jibberish code with the python baseball library; like trying to pass a date into a function that only takes in names. But it gives you the correct functions. Easy enough to go and figure out the documentation on that function. Like I said, it's a ~20x multiplier for our little experiment. Action Items for management: Pay whatever you have to and let us use it all the time.
- lisasays 4y ago
  You used GPT to crank out a bunch of boilerplate. Not to create (or even make meaningful changes to) an actual, working system. With actual users. There is huge chasm of difference between the two.
  - sandspar 4y ago
ryanmcbride 4y ago
Last time I tried it I ended up in an infinite loop of: me: TypeError: sodium.crypto_secretbox_key is not a function when using sodium 3.0.5 chatgpt: I apologize for the confusion. sodium.crypto_secretbox_key was removed from the sodium package in version 3.0.0. Instead, you can use sodium.crypto_secretbox_keygen() to generate a secret key. me: now I get 'TypeError: sodium.crypto_secretbox_keygen is not a function 3.0.5' chat:gpt I apologize for the confusion. sodium.crypto_secretbox_keygen was removed from the sodium package in version 3.0.0. Instead, you can use sodium.crypto_secretbox_key to generate a secret key. Back and forth. And that was when I was like "welp guess I gotta read some docs now" which isn't a big deal, the interesting part for me is that it's basically hallucinating about what features were added or removed in a given library version, which really seems like the sort of thing it aught to be pretty good with.
thiht 4y ago
Same for me, I find Copilot to be a huge waste of time overall. It has a tendency to complete fields that don’t even exist, doesn’t check types, and it’s slow as a cow. It also comes in place of perfectly good deterministic intellisense when it has no idea what I want to write. It’s helpful for autocompleting pure algorithms but that’s about it. And even then, I feel like I waste time reviewing its output and making sure it does what it says it does. I can write these algorithms myself slower but more reliably. It’s really not worth its price.

peoplearepeople 4y ago

There must be some people out there with skillsets so low/shallow that these AI tools really do seem like a productivity boost

sandspar 4y ago
What a titillating thought, how much more skillful and deep we are than all those people!

bobobob420 4y ago

Yes for unit tests lol

mnky9800n 4y ago

yes, this is my experience as well. but any time i mention it, it seems like people don't want to hear that the AI revolution is really just "slightly better intellisense some of the time."

bmelton 4y ago

I'm not a content creator, but while I would never rely solely on AI to write content that I expected to be useful, I do use it for drumming up ideas and have found that (at least for me) its most effective utility is in generating outlines. When I'm writing things informally (like HN posts) rarely allows me to do so clearly and linearly, and I am constantly either leaving out entire paragraphs that I just assumed I'd already written or am over-responding to things because I keep seeing blanks and just keep filling them. ChatGPT (and presumably all the copywriting tools that rely on it) is much better than I am at giving me a checklist of touchpoints to hit.

tristor 4y ago

Their company writes blog spam used for "SEO" for scam products. It's completely reasonable to expect ChatGPT to increase output, because the prompt could be as simple as "Give me a 2500 word blog post in an optimistic tone of voice about human muscle development, and reference the sources used on Wikipedia. During the article link to X at least 6 times." and voila, one customer's scam product down. Repeat 50x.

whstl 4y ago

I kinda did, I guess. I use ChatGPT to generate backend classes or Typescript types from JSON data, to generate unit tests from files, sometimes it also does some browser tests. I also write some parsers so it helps extending syntax a bit (but it's not too good at that). It also helps extracting localization keys, at detecting un-localized keys in HTML templates. If a project has a lot of boilerplate, then it's definitely more than 10 lines per day. But not exactly "good lines". :)

dpkirchner 4y ago

I think I am. I don't know how much of my time is spent writing boilerplate but it feels like a lot. Reducing that time is a big win. Recently I've successfully used ChatGPT to generate: * TS declaration files for existing JS * test fixtures that cover boundary conditions * a regular expression that includes backtracking and optional named capture groups The last one was pretty neat. I gave it sample data and it gave me a regular expression along with JS code I could copy into a node REPL for testing. It didn't get the task right immediately but we were able to iterate until we got the right solution. My regex-fu is weak when it comes to backtracking so this saved me a lot of time.

DonHopkins 4y ago

I've actually found it useful for "decreasing output", or distilling uncharted mountains of my own bullshit down into neatly organized buckets of bullshit. The stuff I want to write about, I've usually written about many times in different contexts over many years. I can stuff all that into GPT-3, and it will summarize and organize it into a nice outline that I can edit, populate, and elaborate myself by talking or typing. So it's all my stuff, but GPT-3 has more of a subtractive and organizational role than a generative role. It does a pretty good job converting lots of redundant huge walls of text, flame war campaigns, and axe grinding diatribes, into concise outlines, introductions, summaries, lists, comparisons, etc. I'm using it to organize, remix, and synthesize lots of papers, articles, postings, emails, conversations, notes, transcripts of video demos and talks, and other things I've written into a coherent 8 module "Vaster Class in Game Design". (Like a Master Class, but Vaster!)

humandeveloper 4y ago

Same for me. I primarily work on mobile applications. ChatGPT is only good for generic tasks that can be explained without additional context which is by far the easiest part of my job.

dkrich 4y ago

No. To me the entire generative AI play is all part of a repetitive cycle. First people are apt to believe anything and get burned in a bubble. Then they are skeptical of everything and get left out of the next actual sea change. Then a new development comes along and they are trained to believe it will turn into a sea change also. I'm saying this because so many people who don't seem to have a great track record of being of the forefront of sea changes are making all sorts of wild predictions about what this will bring. Many also seem to want to believe others will get caught up in a bubble and looking for opportunities to exploit that for personal gain. I realize this is a bit of a rant but it's the lens I've come to view it through based on personal experience. Generative AI seems to be the first useable case for AI but the holy grail- full automation and predictive capabilities still seem quite a ways off because they are much harder problems to solve.

elicash 4y ago

To your first question, I am. And higher quality, not just quantity. As far as code, it's admittedly not the main part of my job and not my biggest strength, so yeah I need the help. I ask it to solve small problems, not write the entire thing. For my writing, most recently I used it to incorporate social pressure into membership emails (peer influence and social proof). On your last point, my first job in politics was creating the MySpace theme for a party committee; I used to create stuff in Flash; most of the toolsets I've used for my work no longer exist. The list of useless skills I possess is endless and my useless skills are way above 90%. You just have to learn new stuff and it's fine.

pathartl 4y ago
Agreed, it's good in small chunks. Sometimes I'll feed it a prompt to generate new code to double check my approach is valid. Sometimes it's shown I am the better architect, other times it's made me aware of parts of an API that I did not know about. Ironically, it's helped me the most in writing recursive code.

JauntyHatAngle 4y ago

I work in devops, and while chat gpt can write some starting terraform code or suggest aws architecture to some degree, the reality is my job is more about figuring out all the hairy details and how to smooth them out or add to them, in an existing world with a domain knowledge required. Chatgpt isn't even remotely useful for the money making part of my work. If all I did was write boilerplate pipelines I'd be worth diddly squat.

achenatx 4y ago

my productivity has increased tremendously with chatgpt For example, I was getting an error that a method x was missing on var of type MyObject. In debugging mode I could see that they type was MyObject and the attribute x was there. I paste the small snippet of code and the error and chapgpt immediately identified that the method call was actually returning a collection. I just didnt notice that the variable was a collection of type MyObject and not an actual MyObject. When Im looking at other people's code I can paste it in and chatgpt will give an opinion on what is happening with the code. I was creating UI components from scratch the other day and needed a word wrapping algorithm. ChatGPT was able to write one that didnt quite work, but was most of the way there. When I looking at unfamiliar APIs it is much faster to ask chatgpt than to do google searches. Even for things like how many police officers did austin have in 2008 are much faster with chatgpt. Ill ask for the reference (always broken) but enough of the reference is correct that I can google and get to the reference. Lately chat gpt has been disconnecting forcing me to reconnect if I go idle for more than about 5 minutes, that has been annoying.

kraftman 4y ago

What I'm finding it most useful for is grokking some new subject rapidly, so the output multiplier is directly related to how well i know the subject im working on. At work, I know the tech stack pretty well so I'm not finding it that valuable, i use it to remind myself of bits and peices here and there. In my free time ive been working in some areas and frameworks i dont know at all and the output increase really is crazy. i was playing around with pixijs and very rapidly had a basic UI cobbled together that would have taken me at least 4-5x longer if id had to read through different docs, blogs, etc. its making me much more willing to try new frameworks and ideas out that i wouldnt have before because of how easily i can get over blockers in my concept of how something works.

MattPalmer1086 4y ago
That is exactly what I have found to be most useful. Not saying there arent other good uses, but boy, does it accelerate trying out new things.

surgical_fire 4y ago

> Seriously, are any of you "increasing output" with AI? Depends on what I'm doing. Professionally, I'm finding it very good for a couple of things: - Exploring frameworks and libraries for languages I'm already familiar with - Writing documentation for code that I've written (I don't use this at my workplace to avoid leaking internal code, but I use it as a way to write small tutorials based on side projects I do on my own) For writing code I find it very unreliable, even for boilerplate.

Turing_Machine 4y ago

I found it really helpful for porting an extension I'd written for Atom (now unsupported/defunct) to VS Code. Now, if I were writing VS Code extensions on a daily basis, I'd want to spend the time to learn all the details, but (ideally) I'll never have to write one again. This is the only one I ever had to write for Atom, so it's a decent bet. And yeah, most of the code was boilerplate-- but still ChatGPT saved me from having to learn how to write that boilerplate. At least a day of time, I'd reckon. Possibly several days. The task was to recognize the Scheme expression (atom or pair) preceding the current caret position, sign it using a password hash, send it to a remote Scheme REPL for evaluation, then insert and select the result of the evaluation in the editor. ChatGPT essentially wrote everything for me, except the part that parsed out the Scheme expression. It wrote a really horrible regex for that. Fortunately I already had a proper parser implemented in the previous Atom extension, which dropped right in.

danShumway 4y ago

I think there's a very narrow window where I'm getting useful results out of it, enough that I'm not completely writing it off as a way to help out with some productive work. I do expect I'll make use of it occasionally while programming in the future. But all of the times I've used it, it hasn't been an extremely clear productivity boost (certainly not at the x2/x10 level), and I do need to babysit it way more than people are suggesting. I go back and forth in my opinion on it. There are some topics where I get useful sessions, there are some topics where it's a mess. The problem is I don't know in advance which are which. It seems to be most useful in scenarios where I either need some extra motivation to get over the initial hump of starting a task (making an task feel more approachable is I think a great use-case), or where I know just enough about a task to know if it's generating nonsense, but not enough to be able to knock out the same results in less time myself. The other problem is that in areas where I'm not immediately familiar with what's going on, I sometimes have to spend a lot of time debugging its code. And reading and debugging other people's code is (to me) a lot harder than writing code myself. So there's an inherent slowdown there where I have to check its work as if I am reading a junior programmer's code and checking it for errors. Code reviews are annoying and take a lot of mental effort to do well. You can ask GPT-4 to explain itself, but there's no way to know without actually stepping through the code if its explanation is correct. I also worry that it's railroading me a bit. I had it prototype an algorithm for me and debugged it and helped it expand the algorithm and get it working with the results I wanted. Only afterwards did I realize there was a much simpler way to get the result I wanted without invoking the algorithm at all. If I wasn't in a conversational loop with an AI focused on getting its code to work, I might have noticed that earlier. But my brain was only focused on debugging the code for that task, it wasn't focused on going back and rethinking the algorithm and cutting out the useless bits. BUT... sometimes it seems helpful with prototyping? So it's not all bad. I don't necessarily want to dismiss it entirely. My take is potentially useful, and for some people who are particularly suited for this kind of rubber-duck peer-coding style of work, maybe a big increase in productivity? But certainly not universally useful for every programmer, and I can still see the seams. GPT-4 is significantly better than GPT-3, but it's not as good as people have hyped it to me, and I think people are vastly over-estimating its capabilities. I went in skeptical, but even with that attitude from what people had told me beforehand I still expected GPT-4 to be better than it is. Am I worried this is going to steal my job? No. That's honestly a kind of laughable idea to me after seeing the kind of code it writes and how unreliable it is. ---- It's hard to put this into words, but it also kind of hints to me at a much better system, and I can see problems here around the auto-regressive nature of the LLM and the need for alignment training and around the whole structure of the thing and the slowness of generating text over and over, and the sort of forced linearity of the chat session that makes me think that maybe GPT-4 is just kind of hinting at a debugging/prototyping methodology that would be a massive improvement to productivity but that doesn't necessarily involve LLMs or involves them to a much lesser degree. It's in some ways a very clumsy/brutalist approach to simulating a feedback repl and accessing documentation, and because it's clumsy/brutalist it over-emphasizes the natural language aspect. And I'm curious to see if future projects can identify what specifically is helpful here and deliver that without all of the inherent downsides of needing to use a conversational model. That's all ignoring the privacy/data concerns, but they're a separate problem.

Balgair 4y ago

Yes, by ~20x. Copy-pasting a comment of mine from a few days ago: We decided at work to run a little experiment with GPT3 to see if/how-much it was 'worth it'. Since baseball is back and most of us are fans, we decided to write a baseball simulator. We each had a Friday afternoon to write one up. Half of us got to use the free GPT3, and half had just regular googling. After the jam, we'd compare notes at the bar and see what the difference, if any, was. Holy cow, was there ever a difference. Those without GPT3 got pretty far. Got the balls and strikes and bases and 9 innings. Most got extra innings down. One even tried the integration with ERA and batting stats in the probabilities of an event occurring but was unable to get it done. The GPT3 group was estimated to be 2 weeks worth of work ahead of the googling group. Turns out, there is a whole python library for baseball simulations and statistics. The googling group didn't find that, but GPT3 just prompted it outright on the first query for everyone using it. This group got the basics of the game done in ~30 minutes. Managed to get integration with actual MLB statistics. Built somewhat real physics simulators of balls in play and distances, adjusted for temperature and altitude. Not all of them at once, but a lot of really great stuff. Aside: Did you know that MLB publishes, in real time, all 6 degrees of freedom for a ball, from where it leaves a pitchers hand to where a catcher/batter interacts with it? They put out the spin rates in three axes! Wild stuff. Our conclusions were that it's totally 'worth it' and is a ~20x multiplier in coding speed. It spits out a lot of really bad code, but it gets the skeletons out very quickly and just rockets you to the crux of the problems. For example: it gave out a lot of jibberish code with the python baseball library; like trying to pass a date into a function that only takes in names. But it gives you the correct functions. Easy enough to go and figure out the documentation on that function. Like I said, it's a ~20x multiplier for our little experiment. Action Items for management: Pay whatever you have to and let us use it all the time.

lisasays 4y ago
You used GPT to crank out a bunch of boilerplate. Not to create (or even make meaningful changes to) an actual, working system. With actual users. There is huge chasm of difference between the two.
- sandspar 4y ago

ryanmcbride 4y ago

Last time I tried it I ended up in an infinite loop of: me: TypeError: sodium.crypto_secretbox_key is not a function when using sodium 3.0.5 chatgpt: I apologize for the confusion. sodium.crypto_secretbox_key was removed from the sodium package in version 3.0.0. Instead, you can use sodium.crypto_secretbox_keygen() to generate a secret key. me: now I get 'TypeError: sodium.crypto_secretbox_keygen is not a function 3.0.5' chat:gpt I apologize for the confusion. sodium.crypto_secretbox_keygen was removed from the sodium package in version 3.0.0. Instead, you can use sodium.crypto_secretbox_key to generate a secret key. Back and forth. And that was when I was like "welp guess I gotta read some docs now" which isn't a big deal, the interesting part for me is that it's basically hallucinating about what features were added or removed in a given library version, which really seems like the sort of thing it aught to be pretty good with.

thiht 4y ago

Same for me, I find Copilot to be a huge waste of time overall. It has a tendency to complete fields that don’t even exist, doesn’t check types, and it’s slow as a cow. It also comes in place of perfectly good deterministic intellisense when it has no idea what I want to write. It’s helpful for autocompleting pure algorithms but that’s about it. And even then, I feel like I waste time reviewing its output and making sure it does what it says it does. I can write these algorithms myself slower but more reliably. It’s really not worth its price.