3 ms·
Wow, an 80% reduction in size for DeepSeek-R1 is just amazing! It's fantastic to see such large models becoming more accessible to those of us who don't have ac
by mtrovo 2y ago
Wow, an 80% reduction in size for DeepSeek-R1 is just amazing! It's fantastic to see such large models becoming more accessible to those of us who don't have access to top-tier hardware. This kind of optimization opens up so many possibilities for experimenting at home.
I'm impressed by the 140 tokens per second speed with the 1.58-bit quantization running on dual H100s. That kind of performance makes the model practical for small or mid sized shops to use it for local applications. This is a huge win for people working on agents that require low latency that only local models could support.
- danielhanchen 2y agoI was pleasantly surprised by 140 tokens/s as well! I literally thought I did something wrong but it was real!
- paradite 2y agoBtw completely off topic, but your comment triggered the internal classification in my brain, and it looks like AI-generated. Not accusing you anything. Could be that you happen to write in a way similar to LLMs. Could be that we are influenced by LLM writing styles and are writing more and more like LLMs. Could be that the difference between LLM generated content and human-generated content is getting smaller and harder to tell.
- j_bum 2y ago+1 my LLM spidy senses were tingling. It’s the exclamation point in the first paragraph, the concise and consistent sentence structure, and the lack of colloquial tone. OP, no worries if you’re real. I often read my own messages or writing and worry that people will think I’m an LLM too.
- jasonjmcghee 2y ago"This kind of optimization opens up so many possibilities" was what triggered me.
- mtrovo 2y agohaha you got me. I'm real person using LLM to proofread the stuff I write. English is not my native language and I'm trying to improve my written vocabulary a little bit. Sorry if it reads a little bit too off.
- paradite 2y agoHaha no worries. This is a perfectly valid use case of LLM. I'm happy that the comment sounds very professional and to the point.
- deleted 2y ago[deleted]
- ahmeneeroe-v2 2y agoVery funny, I didn't mentally jump to LLM, but the language was so lifeless that I stopped reading. Amazing that OP confirmed you're correct (and good use of LLM @OP).