4 ms·
… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?
by jncfhnb 2y ago
… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?
- janpieterz 2y agoHow would I do this reliably? Eg give me 10 different values, all in one prompt for performance reasons? Might not need JSON but whatever format it outputs, it needs to be reliable.
- jncfhnb 2y agoDon’t do it all in one prompt.
- janpieterz 2y agoRight, but now I’m basically running a huge performance hit, need to parallelize my queries etc. I was parsing a document recently, 10-ish questions for 1 document, would make things expensive. Might be what’s needed but not ideal.
- jncfhnb 2y agoLLM performance is a function of the number of tokens, not queries
- esafak 2y agoIf the LLM doesn't output data that conforms to a schema, you can't reliably parse it, so you're back to square one.
- jncfhnb 2y agoIt’s significantly easier to output an integer than a JSON with a key value structure where the value is an integer and everything else is exactly as desired
- esafak 2y agoThat's because you've dumbed down the problem. If it was just about outputting one integer, there would be nothing to discuss. Now add a bunch more fields, add some nesting and other constraints into it...
- jncfhnb 2y agoThe more complexity you add the less likely the LLM is to give you a valid response in one shot. It’s still going to be easier to get the LLM to supply values to a fixed scheme than to get the LLM to give the answers and the scheme
- neverokay 2y agoIs there a general model that got fine tuned on these json schema/output pairs? Seems like it would be universally useful.