4 ms·
Call this a cautionary tale but I tried using ChatGPT yesterday to help me with some work. I was converting code in a an old lambda authorizer and it used a lib
by _fat_santa 4y ago
Call this a cautionary tale but I tried using ChatGPT yesterday to help me with some work. I was converting code in a an old lambda authorizer and it used a library called "jose" for managing JWTs, JWKs, JWEs, etc. Long story short the existing lambda used version 1.x and I wanted to use version 4.x so I wrote a prompt for ChatGPT and got it to convert the code.
At first I was amazed that it could figure it out, until I started to scrutinize the code and realized it was entirely incorrect. It didn't produce a correct answer, just garbage that looks like a correct answer.
I assume ChatGPT will be better for more "broad" topics with more training data but in this scenario it totally fell on it's face.
- zamnos 4y agoI'll second that. I tested ChatGPT and asked it how I could export my tailscale config and it told me to use the tailscale export command, which all well and good - except it doesn't exist. (You want /var/lib/tailscale/tailscaled.state) Hard to keep using it after that. asked ChatGPT how to export my tailscale config and it told me Whether you say ChatGPT is lying or say it's spewing bullshit or a hallucination, it's just plain wrong.
- simonw 4y agoPart of the challenge of using ChatGPT is developing instincts about whether it will do well on a specific question. Tailscale is too new: ChatGPT's training is unlikely to know much if anything about it. I wouldn't be surprised if Bing could handle this much better, since it has the ability to search for current documentation.