3 ms·
Very curious what hardware you're running this on!
by indoordin0saur 4mo ago
Very curious what hardware you're running this on!
- hypfer 4mo agoThe same 24GB VRAM RTX 4090 I bought to play Cyberpunk 2077 with. Works perfectly fine in llama.cpp throwing 70+t/s at me with 128k q8 K/V context when using the IQ4_NL quant + MTP at q4 MTP K/V. Also leaving this here because you might find it useful: https://hypfer.github.io/will-it-fit-llama-cpp/ https://hypfer.github.io/will-it-fit-llama-cpp/
- cdelsolar 4mo agoWhat did you call me?
- indoordin0saur 4mo agoNice! Do you do anything with that compute when you're not actively using it? Is the crypto-mining hobby still worth it? I've also wondered if such expensive hardware can be rented back out to offset cost. Looks like these cards are going for as much as $4k nowadays.
- all2 4mo agoThere are services where you can hook your card up and rent it out to other users. I don't know what any of them are called, but they do exist.
- dghlsakjg 4mo agoSalad.com is one. (I’m unaffiliated, just happened to come across it this week while looking for a cheap option)
- hypfer 4mo agoI've paid ~2k€ in 2023. Since I'm usually sitting next to it, I'm only using it when I want to use it. It can get quite loud and warm. Crypto (to my knowledge at least) moved away from GPU mining. I guess you could maybe rent out GPU compute, but - being in germany - it's not worth the legal hassle. You could of course always commit tax fraud, though I wouldn't recommend that.
- esseph 4mo ago> I've also wondered if such expensive hardware can be rented back out to offset cost. Massive legal liability. Not worth it.
- deleted 4mo ago[deleted]
- Rzor 4mo agoCan you fix MTP-GEMMA-4-26B-A4B-IT? It says the weights are 0.5 GB in size. edit: nvm, I'm confusing models.