3 ms·
For those wondering about the use case, this is very useful when enabling streaming for structured output in LLM responses, such as JSON responses. For my local
by syx 1y ago
For those wondering about the use case, this is very useful when enabling streaming for structured output in LLM responses, such as JSON responses. For my local Raspberry Pi agent I needed something performant, I've been using streaming-json-js [1], but development appears to have been a bit dormant over the past year. I'll definitely take a look at your jsonriver and see how it compares!
[1] https://github.com/karminski/streaming-json-js https://github.com/karminski/streaming-json-js
- cjonas 1y agoParticularly for REACT style agents that use a "final" tool call to end the run.
- rokkamokka 1y agoFor LLMs I recommend just doing NDJSON, that is, newline delimited json. It's much simpler to implement
- rictic 1y agoDo any LLMs support constrained generation of newline delimited json? Or have you found that they're generally reliable enough that you don't need to do constrained sampling?
- sprobertson 1y agonot for the standard hosted APIs using structured output or function calling, best you can get is an array
- stevage 1y agoI love NDJSON in general. I use it a lot for spatial data processing (GDAL calls it GeoJsonSeq).