3 ms·

This massively underestimate what current LLMs can do. Yesterday, I was able to create a 600 lines script in 20 minutes or so that essentially setup a Cloudflar

by csomar 2y ago

This massively underestimate what current LLMs can do. Yesterday, I was able to create a 600 lines script in 20 minutes or so that essentially setup a Cloudflare worker bindings (KV, Queues, Hyperdrive, etc...). The complexity is very low and debuggu-ability is easy. Reading this infra. code is fast. However, if I was to do this manually, it would have taken me a full day reading through the docs and trying the implementation back and forth for each binding I am connecting to. Claude 3.5 did it from the first shot.

jason_zig 2y ago
You still have to become a domain expert to debug it though?
- hombre_fatal 2y ago
  Not when it's code that was only hard to write because you needed to know the right incantations to pipe data between different services. Now you see the incantations that mostly work and the job of transforming it is easy. Java's Bouncy Castle crypto library is a good example of this. The thing you're trying to do might be simple, but to do it, you might need to instantiate 8+ Java classes. It doesn't mean it's complex to read or hard to debug.
  - ConspiracyFact 2y ago
    > The thing you're trying to do might be simple, but to do it, you might need to instantiate 8+ Java classes. It doesn't mean it's complex to read or hard to debug. I’m skeptical that code that needs to instantiate eight separate classes will remain easy to debug in the general case.
  - kuschku 2y ago
    LLMs give you a lot of false confidence, just because something looks right doesn't mean it is. Especially with cryptography you should NEVER use LLMs. Read the docs, write down some notes, and make sure you properly understand everything before you use it. You need to really think it through before you end up leaking user data or worse.
kuschku 2y ago
And 2 months later I get asked to debug code like yours when it doesn't work for a customer and have to spend days or weeks digging into your code before I notice the LLM took some shortcuts that work most of the time, but are ever so slightly broken in edge cases, followed by me having to rebuild it all from scratch. I literally just spent a full week on such a project. Respectfully, fuck people who don't read the docs/spec.
- roland35 2y ago
  Maybe it's easier to verify a working program with the docs rather than try to build something from scratch? I agree it's a bad idea to fire off AI code without any sort of understanding though!!
- csomar 2y ago
  Respectfully, I don't understand what your problem is. First, you don't have to debug any code. That's your personal choice. Second, whether LLM are used or not is irrelevant. The original developer is the party that decides to check and double-check his work and whether he extensively verified what the code is doing (and maybe tightly tested it). LLMs are not making this worse or better.
  - kuschku 2y ago
    First: > First, you don't have to debug any code. That's your personal choice. To quote the Simpsons: > Money can be exchanged for goods and services Second: > LLMs are not making this worse or better. The bottleneck is never "writing code", it's "thinking". LLMs can solve one, but not the other. They allow you to produce more thought (and give you false confidence), but the code has less thought put into it, making it worse.
danny_codes 2y ago
Agree it's good for boilerplate, provided the thing you want to do is extremely basic / just setup. Once you need something slightly more complex it seems to break down rather quickly. Claude 3.5 is pretty good at giving you the right hints though. If you aren't familiar with a library it's definitely faster than grepping through docs. If you are an expert in a library than it's pretty useless.

by csomar 2y ago

jason_zig 2y ago

You still have to become a domain expert to debug it though?

hombre_fatal 2y ago
Not when it's code that was only hard to write because you needed to know the right incantations to pipe data between different services. Now you see the incantations that mostly work and the job of transforming it is easy. Java's Bouncy Castle crypto library is a good example of this. The thing you're trying to do might be simple, but to do it, you might need to instantiate 8+ Java classes. It doesn't mean it's complex to read or hard to debug.
- ConspiracyFact 2y ago
  > The thing you're trying to do might be simple, but to do it, you might need to instantiate 8+ Java classes. It doesn't mean it's complex to read or hard to debug. I’m skeptical that code that needs to instantiate eight separate classes will remain easy to debug in the general case.
- kuschku 2y ago
  LLMs give you a lot of false confidence, just because something looks right doesn't mean it is. Especially with cryptography you should NEVER use LLMs. Read the docs, write down some notes, and make sure you properly understand everything before you use it. You need to really think it through before you end up leaking user data or worse.

kuschku 2y ago

And 2 months later I get asked to debug code like yours when it doesn't work for a customer and have to spend days or weeks digging into your code before I notice the LLM took some shortcuts that work most of the time, but are ever so slightly broken in edge cases, followed by me having to rebuild it all from scratch. I literally just spent a full week on such a project. Respectfully, fuck people who don't read the docs/spec.

roland35 2y ago
Maybe it's easier to verify a working program with the docs rather than try to build something from scratch? I agree it's a bad idea to fire off AI code without any sort of understanding though!!
csomar 2y ago
Respectfully, I don't understand what your problem is. First, you don't have to debug any code. That's your personal choice. Second, whether LLM are used or not is irrelevant. The original developer is the party that decides to check and double-check his work and whether he extensively verified what the code is doing (and maybe tightly tested it). LLMs are not making this worse or better.
- kuschku 2y ago
  First: > First, you don't have to debug any code. That's your personal choice. To quote the Simpsons: > Money can be exchanged for goods and services Second: > LLMs are not making this worse or better. The bottleneck is never "writing code", it's "thinking". LLMs can solve one, but not the other. They allow you to produce more thought (and give you false confidence), but the code has less thought put into it, making it worse.

danny_codes 2y ago

Agree it's good for boilerplate, provided the thing you want to do is extremely basic / just setup. Once you need something slightly more complex it seems to break down rather quickly. Claude 3.5 is pretty good at giving you the right hints though. If you aren't familiar with a library it's definitely faster than grepping through docs. If you are an expert in a library than it's pretty useless.