4 ms·

You can exploit this and build your own stream of streams for interesting use cases: https://chrlschn.dev/blog/2024/05/need-for-speed-llms-beyond-openai-w-dotne

You can exploit this and build your own stream of streams for interesting use cases: https://chrlschn.dev/blog/2024/05/need-for-speed-llms-beyond-openai-w-dotnet-sse-channels-llama3-fireworks-ai/ https://chrlschn.dev/blog/2024/05/need-for-speed-llms-beyond... (screen in action: https://chrlschn.dev/img/need-for-speed/generation-example.gif https://chrlschn.dev/img/need-for-speed/generation-example.g...) Most interesting is combining this with web components and having GPT directly output streams with small web components.

jackmpcollins 2y ago
That gif is really cool! I built a Python package magentic [0] which similarly parses the LLM streamed output and allows it to be used before it is finished being generated. There are plenty of use cases / prompts that can be refactored into a "generate list, then generate for each item" pattern to take advantage of this speedup from concurrent generation. [0] https://magentic.dev/streaming/#object-streaming https://magentic.dev/streaming/#object-streaming