3 ms·

I don’t share the sentiment that it’s “very good”. It’s strongly mediocre at basic tasks and regularly fails at anything complex.

by klysm 2y ago

glzone1 2y ago
I'm getting what I consider very good results. Yes, I have to iterate it and talk to it, spec very clearly, but if you know pretty precisely what you want I've found it very reliable. I usually try to actually provide AWS resource IDs / VPC id's, bucket names and everything and at least with AWS / Python stuff it is good (yes, some corners still but the trend is very good). My prompts aren't short but it spell out very clearly what I want. To process 2 million XML files let's say stored in S3, using an SQS queue ZZZ that will pass in S3 object ID's in this format, store results back to RDS Postgresql here. Use Loguru or whatever for logging. I provide sample XML files, sample results, schema for database (DDL) etc etc. Then you can iterate on this - deploy using docker to ECS, use fargate, then do some scale out when queue depth is deep, go to asyncio or multi-process or multi-threading etc etc and just build. It goes quick.
- klysm 2y ago
  That doesn’t sound like an inherently complex programming task - just simple mindless munging
simonw 2y ago
Have we been using the same models? Claude 3.5 Sonnet writes me excellent code in a variety of different languages. I have a bunch of examples here: https://simonwillison.net/tags/claude-3-5-sonnet/ https://simonwillison.net/tags/claude-3-5-sonnet/
- klysm 2y ago
  Yes we have, but probably for different problems.
- alecco 2y ago
  For Python it's bliss. For non-std C++ it's hit-and-miss. For CUDA C++ it's terrible.
  - justaj 2y ago
    For POSIX sh it generates kind of ugly and unnecessarily complex scripts.
    - Two4 2y ago
      "POSIX sh it" hehehe - my inner 5 year old is too pleased right now.
personjerry 2y ago
That's a failure of imagination. If I can architect and describe my interfaces and have Claude write the current component, then I can repeat with every other component and write minimal code / corrections. At the same time, I've minimized complexity in my codebase. It's a win-win, and honestly probably the next paradigm of software engineering.
- klysm 2y ago
  Depends on the problem at hand. Some code has high essential complexity. Claude doesn’t do well with math heavy work in my experience

I don’t share the sentiment that it’s “very good”. It’s strongly mediocre at basic tasks and regularly fails at anything complex.

by klysm 2y ago

I don’t share the sentiment that it’s “very good”. It’s strongly mediocre at basic tasks and regularly fails at anything complex.

glzone1 2y ago

I'm getting what I consider very good results. Yes, I have to iterate it and talk to it, spec very clearly, but if you know pretty precisely what you want I've found it very reliable. I usually try to actually provide AWS resource IDs / VPC id's, bucket names and everything and at least with AWS / Python stuff it is good (yes, some corners still but the trend is very good). My prompts aren't short but it spell out very clearly what I want. To process 2 million XML files let's say stored in S3, using an SQS queue ZZZ that will pass in S3 object ID's in this format, store results back to RDS Postgresql here. Use Loguru or whatever for logging. I provide sample XML files, sample results, schema for database (DDL) etc etc. Then you can iterate on this - deploy using docker to ECS, use fargate, then do some scale out when queue depth is deep, go to asyncio or multi-process or multi-threading etc etc and just build. It goes quick.

klysm 2y ago
That doesn’t sound like an inherently complex programming task - just simple mindless munging

simonw 2y ago

Have we been using the same models? Claude 3.5 Sonnet writes me excellent code in a variety of different languages. I have a bunch of examples here: https://simonwillison.net/tags/claude-3-5-sonnet/ https://simonwillison.net/tags/claude-3-5-sonnet/

klysm 2y ago
Yes we have, but probably for different problems.
alecco 2y ago
For Python it's bliss. For non-std C++ it's hit-and-miss. For CUDA C++ it's terrible.
- justaj 2y ago
  For POSIX sh it generates kind of ugly and unnecessarily complex scripts.
  - Two4 2y ago
    "POSIX sh it" hehehe - my inner 5 year old is too pleased right now.

personjerry 2y ago

That's a failure of imagination. If I can architect and describe my interfaces and have Claude write the current component, then I can repeat with every other component and write minimal code / corrections. At the same time, I've minimized complexity in my codebase. It's a win-win, and honestly probably the next paradigm of software engineering.

klysm 2y ago
Depends on the problem at hand. Some code has high essential complexity. Claude doesn’t do well with math heavy work in my experience