3 ms·
Hey there! While I was doing some LLM project with OpenAI's API, I stepped with the challenge of streaming OpenAI's API responses to the browser so that respon
by GabrielMusat 3y ago
Hey there!
While I was doing some LLM project with OpenAI's API, I stepped with the challenge of streaming OpenAI's API responses to the browser so that responses appear progressively on the user's screen.
This was a real challenge as my backend was deployed in AWS lambda, which does not support streaming. I was also using a backend framework that did not support streaming anyways... I thought, it would be awesome if OpenAI offered some pre-signed URLs the same way AWS S3 does with their buckets.
Then I came up with this idea, a gateway that sits in front of any API, that listens to requests that provide a valid cryptographic signature, and if authentic
and not expired, redirecting the requests to the specified destination. There is a sequence diagram explaining this in the Readme.md https://github.com/gabotechs/signway https://github.com/gabotechs/signway.
It would be awesome to get feedback from the community!