11 ms·

A couple weeks ago I was working on a parser that needed to handle a new file format that was a large change from existing formats. I wanted some test inputs, b

A couple weeks ago I was working on a parser that needed to handle a new file format that was a large change from existing formats. I wanted some test inputs, both valid and invalid cases. I had the codebase of a tool chain that I knew could generate valid files, some public documentation about the new file format, and my parser codebase. A good problem to throw at AI, I thought. I handed the tools to a SOTA model and asked it to generate me some files. Garbage. Some edits to the prompts and I still get garbage. Okay, that's pretty hard to generate a binary with complex internal structure directly. Let's ask it to tell me how to make the toolchain generate these for me. It gives me back all sorts of CLI examples. None work. I keep telling it what output I am getting and how it differs from what I want. Over and over it fails. I finally reach out to somebody on the toolchain team and they tell me how to do it. Great, now I can generate some valid files. Let's try to generate some invalid ones to test error paths. I've got a file. I've got the spec. I ask the LLM to modify the file to break the spec in a single way each time and tell me which part of the spec it broke each time. Doesn't work. Okay. I ask it to write me a python program that does this. Works a little bit, but not consistently and I need to inspect each output carefully. Finally I throw my files into a coverage guided fuzzer corpus and over a short period of time it's generated inputs that have excellent branch coverage for me. What would effective have looked like to you in this situation?

wordofx 1y ago
Edit: also thanks for actually replying with a great comment cos usually the replies are not even worth entertaining and only proving my point. This is the classic example of “I want it to do everything. But it didn’t do what I wanted. So obviously it’s not helpful.” It doesn’t solve /all/ problems. And some models are better than others at certain tasks. You see people talk about PRDs and they say “I got Claude to create a PRD and it sucked” but you sit them down with o3 and generate the same PRD and they are like “oh wow this is actually pretty decent”. But difficult to help over HN. As a sort of related example. Back in May I had a requirement to ingest files from a new POS system we didn’t currently support. The exports we got are CSV but based on the first char decides the first type of line that needs to be parsed and how many commas will be in that data set line. I used o3 and explained everything I could have how the file worked and how it should parse it into a generic list etc. got it to generate a basic PRD with steps and assertions along the way. I then fed this into cursor using Claude sonnet 4 with the csv files asking it to look at the files and the PRD and asked it if there was anything that didn’t make sense. Then asked it to begin implementing the steps 1 by 1 and letting me check before moving onto the next step. Couple of times it misunderstood and did things slightly wrong but I just corrected it or asked Claude to correct it. But it essentially wrote code. Wrote test. Verified. I verified. It moved on. Typically in the past these tasks take a few days to implement and roll out. This whole thing took about 1 hour. The code is in the style of the other parsers with the exception that it optimised some parts and despite being a bit more complicated runs faster than some of our older parsers. While most of my usage is around programming. We also use AI for marketing, customer leads, I have scheduled tasks to give me summaries of tickets from the support system to know what should be prioritised for the day. So even tho AI doesn’t solve all programming issues we get value in almost all aspects of the business.
- UncleMeat 1y ago
  The claim was that I don't know how to use AI effectively. This felt like a reasonable thing for it to do. What did I do wrong? Why do you feel the need to denigrate me in your post?
  - wordofx 1y ago
    Sigh. I thought you were being genuine. Sorry. Your post perfectly explained that you don’t know how to use AI. You’re at the stage where you hear stories. You try it. Don’t get the expected result on first attempt. Then decide it sucks. You give no information on the models you’re using. Nothing about the prompts or process. It’s like you’re asking a junior to go build a house. So he goes off and uses bricks to create 4 walls. And you turn around and get angry because he didn’t use wood. So he uses wood and you get angry cos he didn’t add windows, roof, door. I’m sorry that you picked up a career where you struggle to use new tools and technologies.
    - UncleMeat 1y ago
      I am being genuine. Then you decided that I was saying "I want it to do everything. But it didn’t do what I wanted. So obviously it’s not helpful" when that's clearly not correct. Now you are shitting on my for being in software? Unbelievable.

wordofx 1y ago

Edit: also thanks for actually replying with a great comment cos usually the replies are not even worth entertaining and only proving my point. This is the classic example of “I want it to do everything. But it didn’t do what I wanted. So obviously it’s not helpful.” It doesn’t solve /all/ problems. And some models are better than others at certain tasks. You see people talk about PRDs and they say “I got Claude to create a PRD and it sucked” but you sit them down with o3 and generate the same PRD and they are like “oh wow this is actually pretty decent”. But difficult to help over HN. As a sort of related example. Back in May I had a requirement to ingest files from a new POS system we didn’t currently support. The exports we got are CSV but based on the first char decides the first type of line that needs to be parsed and how many commas will be in that data set line. I used o3 and explained everything I could have how the file worked and how it should parse it into a generic list etc. got it to generate a basic PRD with steps and assertions along the way. I then fed this into cursor using Claude sonnet 4 with the csv files asking it to look at the files and the PRD and asked it if there was anything that didn’t make sense. Then asked it to begin implementing the steps 1 by 1 and letting me check before moving onto the next step. Couple of times it misunderstood and did things slightly wrong but I just corrected it or asked Claude to correct it. But it essentially wrote code. Wrote test. Verified. I verified. It moved on. Typically in the past these tasks take a few days to implement and roll out. This whole thing took about 1 hour. The code is in the style of the other parsers with the exception that it optimised some parts and despite being a bit more complicated runs faster than some of our older parsers. While most of my usage is around programming. We also use AI for marketing, customer leads, I have scheduled tasks to give me summaries of tickets from the support system to know what should be prioritised for the day. So even tho AI doesn’t solve all programming issues we get value in almost all aspects of the business.

UncleMeat 1y ago
The claim was that I don't know how to use AI effectively. This felt like a reasonable thing for it to do. What did I do wrong? Why do you feel the need to denigrate me in your post?
- wordofx 1y ago
  Sigh. I thought you were being genuine. Sorry. Your post perfectly explained that you don’t know how to use AI. You’re at the stage where you hear stories. You try it. Don’t get the expected result on first attempt. Then decide it sucks. You give no information on the models you’re using. Nothing about the prompts or process. It’s like you’re asking a junior to go build a house. So he goes off and uses bricks to create 4 walls. And you turn around and get angry because he didn’t use wood. So he uses wood and you get angry cos he didn’t add windows, roof, door. I’m sorry that you picked up a career where you struggle to use new tools and technologies.
  - UncleMeat 1y ago
    I am being genuine. Then you decided that I was saying "I want it to do everything. But it didn’t do what I wanted. So obviously it’s not helpful" when that's clearly not correct. Now you are shitting on my for being in software? Unbelievable.