7 ms·

I use the same programming prompt on both Bard and ChatGPT. "write a function in java to decode a rfc 3588 diameter header" They both need a lot of hand-holdi

I use the same programming prompt on both Bard and ChatGPT. "write a function in java to decode a rfc 3588 diameter header" They both need a lot of hand-holding. Too much to make it worth using them. Things they get wrong... 1) They are confident about the structure, but are 100% wrong. 2) They get Java types wrong, and try to put unsigned data into the same size signed type (8 bits into a byte, 24 bits into an int). 3) They marshal data into ints in a way that is correct for C code, but wrong for Java code (missing the 0xFF mask) 4) They forget previous instructions and keep re-introducing bugs. 5) ChatGPT kept thinking that version only had information in bits 6-8. If you know _anything_ about what you're trying to do, it's a waste of time to use a LLM. If the code has anything interesting, a LLM will write broken code. On the plus side, Bard was quoting sources, complete with license. That led to my first pull request to hopefully fix Bard's upstream source, and let me understand what (byte[0]&0x77) << 16 does, why it works and why byte[0]<<16 doesn't. But then, the LLMs didn't help with that, my need to fix it did. http://blog.jason.pollock.ca/2023/04/trying-to-write-code-with-bard-and.html http://blog.jason.pollock.ca/2023/04/trying-to-write-code-wi...

gwoolhurme 3y ago
That has been a bit of my experience with LLMs in general. If you know what you are doing before hand it's not too much of a boost, in fact it can be a bit frustrating to wrestle with it to do what you want when you can do it faster. I can see where it is amazing tooling though. Especially for a new project where you don't really have much of an idea or need a working demo.
- worrycue 3y ago
  But if you don’t know what you are doing, you have to worry about its hallucinations slipping pass you.
  - gwoolhurme 3y ago
    Really good point too, though for the most part I like to think if I audit the code well enough it will work? That said the other day I used it for some string manipulation based on powers of 10, and the code looked correct as per my audit of it. However in reality when I ran the code it was really bad. You might say that I my self would produce buggy code the first time around, that is true... but I also find the LLM equally frustrating because for me (and only me, a sample size of 1 calm down people)? I find I can iterate faster over an idea by hand vs using an LLM for a lot of things.
  - throwaway675309 3y ago
    eh, there is always somebody who inevitably frets on about these hallucinations, but I find that most of the people who are super worried about these aren't actually using the LLMs the way that they're intended. For example, I needed a GitHub action that would automatically check out build and deploy a node project to a VPS. I also needed it to pipe out the secrets into a local environment variable (.env) and copy that over to the VPS as well. GPT4 got it in one try. Its this kind of DevOps boring mundane crap that GPT excels at eliminating. Even though I had very little familiarity with the syntax for GH action yaml, I was able to read it and reason out that it was correct. Could I have sifted through the documentation, or waited through 50 stack overflow pages? Sure, but this was a hell of a lot more convenient.
famouswaffles 3y ago
What exactly is the point of using two models, both far from state of the art and saying, "LLMs are not useful". I mean, what's the thought process here ?
- kyleyeats 3y ago
  You'll see this in the workplace too. It's very hard to explain to the not-most-productive people that the most productive people on the team are also usually making the most mistakes. They'll always be happy to just sit back and criticize the mistakes. It's the only way they can be better.
- jpollock 3y ago
  Sorry, my comment wasn't LLMs in general, it was this specific pair of LLMs at this specific time. I've been trying the same prompt since ChatGPT was released, and I keep revisiting it. They are getting better at writing code, Bard's addition of sourcing information is very important to me. I do this so I can see if it will help _me_ in what I'm trying to do, and where the failure modes are. Their failure modes are very sneaky and very dangerous for the kind of programming I do. I would use them to maybe give me a first pass at some code in a language I have zero familiarity with, and only to solve general problems. I'm thinking throw-away, one-off code where the operational range is known up front and fully middle of the road. I'm thinking reports. I might try to get it to write something useful in R, I have no knowledge of that. I can see them as really handy refactoring tools. They take instruction pretty well. They just need to be stable in the code they produce - they keep reintroducing bugs you've instructed them to fix. Changing to an edit stream against code instead of re-generating the code might be a fix?
agileAlligator 3y ago
Try using https://www.phind.com https://www.phind.com on expert mode. You will probably have better results.
- jpollock 3y ago
  Thanks for the tip, phind is much better. Still has the "won't fit in an int" problem, but it did use byte & 0x77 to get them into ints. Amazingly, it got the field correct and in the right order! It did not reference a source, but that seems to be a Bard feature right now.
  - jpollock 3y ago
    phind needed two additional prompts to get to correct code. It needed: 1) write a function in java to decode a rfc 3588 diameter header 2) 32 bit values will not fit into a java int, please fix the code. 3) 24 bit values will not fit into a java int, please fix the code. I think I'm going to have to come up with a more complicated problem! Pretty impressively, this is what it produced: public class DiameterHeader { private int version; private long messageLength; private int commandFlags; private long commandCode; private long applicationId; private long hopByHopId; private long endToEndId; public DiameterHeader(byte[] data) { if (data == null || data.length != 20) { throw new IllegalArgumentException("Invalid Diameter header data"); } version = data[0] & 0xFF; messageLength = ((long) data[1] & 0xFF) << 16 | ((long) data[2] & 0xFF) << 8 | ((long) data[3] & 0xFF); commandFlags = data[4] & 0xFF; commandCode = ((long) data[5] & 0xFF) << 16 | ((long) data[6] & 0xFF) << 8 | ((long) data[7] & 0xFF); applicationId = (((long) data[8] & 0xFF) << 24) | (((long) data[9] & 0xFF) << 16) | (((long) data[10] & 0xFF) << 8) | ((long) data[11] & 0xFF); hopByHopId = (((long) data[12] & 0xFF) << 24) | (((long) data[13] & 0xFF) << 16) | (((long) data[14] & 0xFF) << 8) | ((long) data[15] & 0xFF); endToEndId = (((long) data[16] & 0xFF) << 24) | (((long) data[17] & 0xFF) << 16) | (((long) data[18] & 0xFF) << 8) | ((long) data[19] & 0xFF); } // Getters for the fields }
  - agileAlligator 3y ago
    If you have expert mode on it should reference sources most of the time.
bumbledraven 3y ago
Try GPT-4.
rrrrrrrrrrrryan 3y ago
Have you tried GPT-4? Here's the response I got: https://shareg.pt/Q0ZcWCX https://shareg.pt/Q0ZcWCX Is it any better? (Your prompt is related to a completely different domain than the one I work in.)
- jpollock 3y ago
  Nifty, thanks! The code has the standard "you can't fit 24 bit or 32 bit unsigned values in an int" bug. I get the feeling this is a common bug. Probably insufficient testing in the hobby implementations used as training data. My reading of "int length = first4Bytes & 0xFFFFFF" is that it will truncate values incorrectly. I'm pretty sure that "-1 & 0xFFFFFFF != 0xFFFFFFF". It goes (-1 & 0xFFFFFFF) is a long, which is then downcast back to int, truncating it? commandCode is wrong, GPT-4 is using 28 bits instead of 24. I _think_ the command code flag decoding is wrong, but it's been a while since I've done bit fiddling. The spec says Request is bit 0, but the code has it in 0x80 - bit 8? That's much improved from ChatGPT and Bard. The use of the ByteBuffer avoids many of the problems. I've seen that in some GitHub Diameter decoders. The RFC is here: https://www.ietf.org/rfc/rfc3588.txt https://www.ietf.org/rfc/rfc3588.txt - Page 31 for the header.
  - davidjones332 3y ago
    You forgot anyone can just run your example in ChatGPT and post a screenshot showing you're talking out of your ass.
- davidjones332 3y ago
  The guy who was trying to trash ChatGPT (gpt4) forgot that anyone can just check his example and post a screenshot I think.

gwoolhurme 3y ago

That has been a bit of my experience with LLMs in general. If you know what you are doing before hand it's not too much of a boost, in fact it can be a bit frustrating to wrestle with it to do what you want when you can do it faster. I can see where it is amazing tooling though. Especially for a new project where you don't really have much of an idea or need a working demo.

worrycue 3y ago
But if you don’t know what you are doing, you have to worry about its hallucinations slipping pass you.
- gwoolhurme 3y ago
  Really good point too, though for the most part I like to think if I audit the code well enough it will work? That said the other day I used it for some string manipulation based on powers of 10, and the code looked correct as per my audit of it. However in reality when I ran the code it was really bad. You might say that I my self would produce buggy code the first time around, that is true... but I also find the LLM equally frustrating because for me (and only me, a sample size of 1 calm down people)? I find I can iterate faster over an idea by hand vs using an LLM for a lot of things.
- throwaway675309 3y ago
  eh, there is always somebody who inevitably frets on about these hallucinations, but I find that most of the people who are super worried about these aren't actually using the LLMs the way that they're intended. For example, I needed a GitHub action that would automatically check out build and deploy a node project to a VPS. I also needed it to pipe out the secrets into a local environment variable (.env) and copy that over to the VPS as well. GPT4 got it in one try. Its this kind of DevOps boring mundane crap that GPT excels at eliminating. Even though I had very little familiarity with the syntax for GH action yaml, I was able to read it and reason out that it was correct. Could I have sifted through the documentation, or waited through 50 stack overflow pages? Sure, but this was a hell of a lot more convenient.

famouswaffles 3y ago

What exactly is the point of using two models, both far from state of the art and saying, "LLMs are not useful". I mean, what's the thought process here ?

kyleyeats 3y ago
You'll see this in the workplace too. It's very hard to explain to the not-most-productive people that the most productive people on the team are also usually making the most mistakes. They'll always be happy to just sit back and criticize the mistakes. It's the only way they can be better.
jpollock 3y ago
Sorry, my comment wasn't LLMs in general, it was this specific pair of LLMs at this specific time. I've been trying the same prompt since ChatGPT was released, and I keep revisiting it. They are getting better at writing code, Bard's addition of sourcing information is very important to me. I do this so I can see if it will help _me_ in what I'm trying to do, and where the failure modes are. Their failure modes are very sneaky and very dangerous for the kind of programming I do. I would use them to maybe give me a first pass at some code in a language I have zero familiarity with, and only to solve general problems. I'm thinking throw-away, one-off code where the operational range is known up front and fully middle of the road. I'm thinking reports. I might try to get it to write something useful in R, I have no knowledge of that. I can see them as really handy refactoring tools. They take instruction pretty well. They just need to be stable in the code they produce - they keep reintroducing bugs you've instructed them to fix. Changing to an edit stream against code instead of re-generating the code might be a fix?

agileAlligator 3y ago

Try using https://www.phind.com https://www.phind.com on expert mode. You will probably have better results.

jpollock 3y ago
Thanks for the tip, phind is much better. Still has the "won't fit in an int" problem, but it did use byte & 0x77 to get them into ints. Amazingly, it got the field correct and in the right order! It did not reference a source, but that seems to be a Bard feature right now.
- jpollock 3y ago
  phind needed two additional prompts to get to correct code. It needed: 1) write a function in java to decode a rfc 3588 diameter header 2) 32 bit values will not fit into a java int, please fix the code. 3) 24 bit values will not fit into a java int, please fix the code. I think I'm going to have to come up with a more complicated problem! Pretty impressively, this is what it produced: public class DiameterHeader { private int version; private long messageLength; private int commandFlags; private long commandCode; private long applicationId; private long hopByHopId; private long endToEndId; public DiameterHeader(byte[] data) { if (data == null || data.length != 20) { throw new IllegalArgumentException("Invalid Diameter header data"); } version = data[0] & 0xFF; messageLength = ((long) data[1] & 0xFF) << 16 | ((long) data[2] & 0xFF) << 8 | ((long) data[3] & 0xFF); commandFlags = data[4] & 0xFF; commandCode = ((long) data[5] & 0xFF) << 16 | ((long) data[6] & 0xFF) << 8 | ((long) data[7] & 0xFF); applicationId = (((long) data[8] & 0xFF) << 24) | (((long) data[9] & 0xFF) << 16) | (((long) data[10] & 0xFF) << 8) | ((long) data[11] & 0xFF); hopByHopId = (((long) data[12] & 0xFF) << 24) | (((long) data[13] & 0xFF) << 16) | (((long) data[14] & 0xFF) << 8) | ((long) data[15] & 0xFF); endToEndId = (((long) data[16] & 0xFF) << 24) | (((long) data[17] & 0xFF) << 16) | (((long) data[18] & 0xFF) << 8) | ((long) data[19] & 0xFF); } // Getters for the fields }
- agileAlligator 3y ago
  If you have expert mode on it should reference sources most of the time.

bumbledraven 3y ago

Try GPT-4.

rrrrrrrrrrrryan 3y ago

Have you tried GPT-4? Here's the response I got: https://shareg.pt/Q0ZcWCX https://shareg.pt/Q0ZcWCX Is it any better? (Your prompt is related to a completely different domain than the one I work in.)

jpollock 3y ago
Nifty, thanks! The code has the standard "you can't fit 24 bit or 32 bit unsigned values in an int" bug. I get the feeling this is a common bug. Probably insufficient testing in the hobby implementations used as training data. My reading of "int length = first4Bytes & 0xFFFFFF" is that it will truncate values incorrectly. I'm pretty sure that "-1 & 0xFFFFFFF != 0xFFFFFFF". It goes (-1 & 0xFFFFFFF) is a long, which is then downcast back to int, truncating it? commandCode is wrong, GPT-4 is using 28 bits instead of 24. I _think_ the command code flag decoding is wrong, but it's been a while since I've done bit fiddling. The spec says Request is bit 0, but the code has it in 0x80 - bit 8? That's much improved from ChatGPT and Bard. The use of the ByteBuffer avoids many of the problems. I've seen that in some GitHub Diameter decoders. The RFC is here: https://www.ietf.org/rfc/rfc3588.txt https://www.ietf.org/rfc/rfc3588.txt - Page 31 for the header.
- davidjones332 3y ago
  You forgot anyone can just run your example in ChatGPT and post a screenshot showing you're talking out of your ass.
davidjones332 3y ago
The guy who was trying to trash ChatGPT (gpt4) forgot that anyone can just check his example and post a screenshot I think.