4 ms·
Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.
by rao-v 2mo ago
Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.
- cmrdporcupine 2mo agoYou're right. I'm getting ~33tok/sec w/ dflash on it, even bursts up to 60tok/sec, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based) That's pretty respectable. Still working on optimizing and cleaning up before I push it.
- cmrdporcupine 2mo agohttps://github.com/rdaum/eider/ https://github.com/rdaum/eider/ fwiw