4 ms·

I've been playing around with data extraction using OpenAI's structured outputs feature last week. Perhaps this is due to me still being fairly new to this worl

I've been playing around with data extraction using OpenAI's structured outputs feature last week. Perhaps this is due to me still being fairly new to this world, but man, has it been an exercise in frustration. My general takeaway so far is that LLMs are fantastic for creative writing, and are a much harder beast to tame when you're looking for exact, reproducible answers. Even when those answers should be the most obvious ones. My engineering brain does not appreciate all the prompt engineering hacks, the hopeful JSON schema tweaking, the "let's try this other random thing just to see if it works" approach to making progress. The worst has been getting to an OK point with one of the fields, only to find out later that giving directions for another field blew away the earlier progress. I'm sure I must be missing something since there are products that are doing this, but so far I have little to show for hours spent other than growing knowledge of LLM quirks and subtle behaviors.

mattmanser 2y ago
It's a really interesting comment, but I would also have loved the context of what you're trying to do?
- romanhn 2y ago
  Without going into specifics, trying to classify a body of freeform text (or rather, many such bodies) along a bunch of dimensions. Imagine a fiction story that you're trying to answer questions about such as "what is the genre?" and "is the protagonist male or female, adult or child?", etc, where I'm instructing the LLM to return values out of a specific range only (aka, enums). OpenAI's new-ish structured outputs feature enforces answers to a user-specified JSON schema, which is awesome, but I'm finding that the actual answers it comes up with are sub-par. There are all kinds of hacks out there like creating additional fields to let it reason out the answers before constraining them to an enum elsewhere, but I have yet to make it work well for me. There are other approaches I haven't yet tried that require much more effort or cost (more expensive models, fine-tuning custom models). It feels like it should work, so the random poking and prodding continues...
  - mattmanser 2y ago
    Interesting, thanks for sharing. By coincidence I had been wondering how hard something like that would be to do the other day.

mattmanser 2y ago

It's a really interesting comment, but I would also have loved the context of what you're trying to do?

romanhn 2y ago
Without going into specifics, trying to classify a body of freeform text (or rather, many such bodies) along a bunch of dimensions. Imagine a fiction story that you're trying to answer questions about such as "what is the genre?" and "is the protagonist male or female, adult or child?", etc, where I'm instructing the LLM to return values out of a specific range only (aka, enums). OpenAI's new-ish structured outputs feature enforces answers to a user-specified JSON schema, which is awesome, but I'm finding that the actual answers it comes up with are sub-par. There are all kinds of hacks out there like creating additional fields to let it reason out the answers before constraining them to an enum elsewhere, but I have yet to make it work well for me. There are other approaches I haven't yet tried that require much more effort or cost (more expensive models, fine-tuning custom models). It feels like it should work, so the random poking and prodding continues...
- mattmanser 2y ago
  Interesting, thanks for sharing. By coincidence I had been wondering how hard something like that would be to do the other day.