3 ms·
Does this describe the "streaming HTTP" to which you refer https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88 https://gist.github.com/CMCDragonkai/6bfad
by 1vuio0pswjnm7 3y ago
Does this describe the "streaming HTTP" to which you refer
https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88 https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88
If not, is there an example of "streaming HTTP" you could provide that illustrates the limitation
- dumbo-octopus 3y agoYes, that is that I mean. All the modern "ChatGPT" style API's use this, so if you're building anything that invokes them you can choose between buffering the entire response and modifying/relaying it once complete, or building up a toolchain of streaming-capable utilities. Of all the http-client applications I tested (Curl, Postman, Insomnia, Bruno), somewhat hilariously Curl has the best support. It will output all 'Transfer-Encoding: chunked' with line-by-line buffering, whereas Postman only supports responses precisely following the `Content-Type: text/event-stream` format (strictly less powerful than curl, as this format requires newlines in between events, and a bunch of overhead on top of that). The others buffer the entire response before displaying anything. The `Content-Type: text/event-stream` format is fine enough, but I personally prefer to just plainly chunk the responses and let the client choose whether to buffer them into memory and operate on the entire value at once, or interpret them live as the chunks come in. With tools like gjp-4-gpt (Gradual JSON Parser 4 GPT's) you can even interpret partial JSON objects live as they come in and display complex/nested data structures in a generative manner.
- deleted 3y ago[deleted]
- 1vuio0pswjnm7 3y agoWhat if instead of JSON, i.e., strings of unknown length, used something more like netstrings. https://cr.yp.to/proto/netstrings.txt https://cr.yp.to/proto/netstrings.txt Personally I use a lame but effective simple 85.9 KiB static binary filter, a small C program, that removes the chunk sizes so the response is ready for use by other programs, e.g., in a pipe. Buffer is set at 8 KiB. Is there a way to experiment with one of these streaming JSON GPT APIs non-interactively by just sending an HTTP request, without need for a third party program, an account, use of a Javascript engine, etc.
- dumbo-octopus 3y agoThe unknown length isn't much of a problem for me in practice: GPT's are slow enough that getting a large chunk is almost impossible. I like the idea of the C filter, but in the end you're just piping the data to the program, why add the middle step? Is it to protect against too-large chunks in some way? I don't know a public API that returns JSON slowly, but you could simulate it by just taking a JSON string, splitting it into 3-5 char chunks, and sending each of those in a `Transfer-Encoding: Chunked` response at ~100ms intervals. Actually, now that I look at the underlying mechanism behind `Transfer-Encoding: Chunked`, it looks like it's already basically the same as the netstrings. What I'm referring to is the (variable length) contents of the netstring/chunk being sequential slices of a JSON object.
- 1vuio0pswjnm7 3y ago"I like the idea of the C filter, but in the end you're just piping the data to the program, why add the middle step?" Only for the flexibility to use more programs. Otherwise every program I use to process HTTP responses needs to be able to accomodate chunked transfer encoding. Plus only a minority of sites send chunked responses. Instead, have one program that does one thing: remove chunked transfer encoding. IIUC, what you want is uniform chunk sizes where you know the size before you send the request. GPTs sound annoying if they are so slow that they only output a few characters every ~100ms..
- dumbo-octopus 3y agoAh I see, I'm working a bit further up the stack from you so the JS runtime handles making the transfer encoding of the response more or less irrelevant, for any response with any encoding you can access `response.text()` and get the entire contents when they're ready, or do something like `for await (const chunk of response.body) { ... }` to work with the pieces as they roll in. > IIUC, what you want is uniform chunk sizes where you know the size before you send the request. I don't think so... I don't really want anything! Just a GUI that displays existing HTTP chunked responses as they come. > GPTs sound annoying if they are so slow that they only output a few characters every ~100ms.. That's perhaps an exaggeration, but in general the speed and "smartness" of the model are inversely correlated.