3 ms·
Not x86_64's Streaming SIMD Extensions, but Server-sent events [1]. SSE and AVX are probably not that bad at /handling/ LLM tokens... [1] https://en.wikipedia
by wyldfire 10mo ago
Not x86_64's Streaming SIMD Extensions, but Server-sent events [1]. SSE and AVX are probably not that bad at /handling/ LLM tokens...
[1] https://en.wikipedia.org/wiki/Server-sent_events https://en.wikipedia.org/wiki/Server-sent_events
- FuckButtons 10mo agoSse is ancient and limited to 128 bit wide registers without any native 16 bit float ops or 8/4 bit ints. It’s definitely going to be very slow.
- wyldfire 10mo agoAnd yet it's an improvement over the scalar core.