4 ms·

Watch out, I'm doing devops too and I've caught chatgpt on such obviously stupid behaviors it hurts to even think about it. It's not just hallucinations problem

Watch out, I'm doing devops too and I've caught chatgpt on such obviously stupid behaviors it hurts to even think about it. It's not just hallucinations problem (edge cases or doing unusual stuff). It seems to give answers that are 100% incorrect and when told so says "of course you're right, here is the right answer". The only stuff I'd use it for is if I already know exactly how to write the script and I'm just using it to type it for me quickly because I don't remember if aws_instance is the correct spelling or aws-instance in terraform...

scarface_74 3y ago
Exactly. But with code, it’s easy enough to prove correctness just by running it. That being said, the one bug I find consistently with ChatGPT is that with the AWS APIs, all list type methods pagínate and you have to account for that. Python/boto3 have built in paginators and it doesn’t. This an insidious bug because things will work correctly in a dev account with only few resources. But will fail in hard to debug ways in production. What’s even worse is that ChatGPT “knows” the pattern and will correct itself once you say something like “This won’t work with more than 50 roles/ec2 instances, etc”
mullingitover 3y ago
My trick is to get second and third opinions - sometimes ChatGPT gets it wrong, but bard is right, or 3.5 is right when 4 is wrong. So I just copy the same question to all available chatbots and compare. Asking them to provide sources for the answers is also a good way to keep them honest.
- scarface_74 3y ago
  With code in particular the only thing I use it for, the source of truth is running the code and testing for corner cases. I usually know what the right answer is. It can just get to it faster. I may not know always know the correct API or CloudFormation/CDK/Terraform syntax. But if it gets it wrong, I can read the docs and correct it. Providing sources doesn’t usually help. ChatGPT consistently makes up sources.
  - mullingitover 3y ago
    Vanilla ChatGPT can hallucinate sources, but with the web browsing plugin in Plus it'll produce real links. Bard and Bing AI browse and produce accurate links right out of the box. Certainly with code the proof is in the pudding, but most recently my problem was "I need to create AWS monitors in Datadog to alert when a region is down." ChatGPT was hopeless but bard was able to point me to the exact doc explaining how to set it up.

scarface_74 3y ago

Exactly. But with code, it’s easy enough to prove correctness just by running it. That being said, the one bug I find consistently with ChatGPT is that with the AWS APIs, all list type methods pagínate and you have to account for that. Python/boto3 have built in paginators and it doesn’t. This an insidious bug because things will work correctly in a dev account with only few resources. But will fail in hard to debug ways in production. What’s even worse is that ChatGPT “knows” the pattern and will correct itself once you say something like “This won’t work with more than 50 roles/ec2 instances, etc”

mullingitover 3y ago

My trick is to get second and third opinions - sometimes ChatGPT gets it wrong, but bard is right, or 3.5 is right when 4 is wrong. So I just copy the same question to all available chatbots and compare. Asking them to provide sources for the answers is also a good way to keep them honest.

scarface_74 3y ago
With code in particular the only thing I use it for, the source of truth is running the code and testing for corner cases. I usually know what the right answer is. It can just get to it faster. I may not know always know the correct API or CloudFormation/CDK/Terraform syntax. But if it gets it wrong, I can read the docs and correct it. Providing sources doesn’t usually help. ChatGPT consistently makes up sources.
- mullingitover 3y ago
  Vanilla ChatGPT can hallucinate sources, but with the web browsing plugin in Plus it'll produce real links. Bard and Bing AI browse and produce accurate links right out of the box. Certainly with code the proof is in the pudding, but most recently my problem was "I need to create AWS monitors in Datadog to alert when a region is down." ChatGPT was hopeless but bard was able to point me to the exact doc explaining how to set it up.