4 ms·
I use the Qwens as a vision model for my DeepSeek V4 Flashes to handle. But the Qwens run on old RTX A6000 Ampere. Does anyone know if there's any news about IN
by arjie 2mo ago
I use the Qwens as a vision model for my DeepSeek V4 Flashes to handle. But the Qwens run on old RTX A6000 Ampere. Does anyone know if there's any news about INT4/AWQ quants for the RTX A6000?
- ericd 2mo agoWas recently thinking about doing something similar, do you basically just have the qwens describe what they see for the flashes? Was considering adding a LoRa/vision head to Flash, but seems like it could take a while to get it right. If DSv4 Flash was multimodal, I’d probably be done model shopping for a while
- arjie 2mo agoSame, with a multimodal DSv4 Flash I would just stop paying attention to things. Very smart, and at 260 tok/s it's too fast to care about anything else. If you ever graft something like that I would love to hear about it. Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.
- ericd 2mo agoYeah, I might just replicate what you're doing. Main issue right now is just finding spare vram to actually run another model in parallel... And yeah, if I train up a vision adapter somehow, I'll try to put it up/post about it, seems like we're getting the killer apps for local LLMs right now, where it's just feasible enough if you're enthusiastic enough to be a bit economically irrational, and just useful enough to sort of rationalize.