3 ms·

As an autodidactic, I use a similar approach, curious who else is similar? - my box "sm3llslik3s0ld3r" is a love hate passion project, but surpringly similar i

As an autodidactic, I use a similar approach, curious who else is similar? - my box "sm3llslik3s0ld3r" is a love hate passion project, but surpringly similar in hardware to yours! I originally made "sm3lly" (my box) after screwing up and leaving a high priced azure gpu cloud instance running on an unrestricted pay-as-you-go billing scope, that was filtered from my view (separate az-ad).. This was for a almost a week before I caught it resulting in a $7k usd type 3 fun "surprise" hit to my personal r&d budget. I suspect its a small niche group: those who are doing ai on our own desktop hardware though. I assume most ai persons don't have the systems expertise to do/manage this or they work for a company/school which grants them the resources and don't bring "work" home with them. Fwiw I also bought all "sm3lly" hardware on taobao from shenzhen directly, so it's not identical to yours but I was picking up pre-loved bitcoin mining GPus on the very cheap side after chinas crypto purge.

fxtentacle 5y ago
I'm currently waiting for the Ethereum vote to flood the market with used GPUs and then hopefully new ones will become more affordable, too. And yes, I've also been burned multiple times by runaway AWS costs. Especially since you keep paying for CPU time for instances that are shut down but not fully deleted. That always struck me as odd and most other cloud providers handle that differently. And yes, I would also assume that most AI persons don't have the systems expertise, which is kinda sad. But then again, most AI tutorials are buggy as hell, presumably for the same reason. For example https://huggingface.co/blog/wav2vec2-with-ngram https://huggingface.co/blog/wav2vec2-with-ngram (yes, the people building the Huggingface AI toolkit) says: "The 5-gram correctly includes a "Unknown" or <unk>, as well as a begin-of-sentence, <s> token, but no end-of-sentence, </s> token. This sadly has to be corrected currently after the build." And then follows a cringe-worthy script to "fix" the missing </s> by rewriting a 100+ GB file in python. But actually, they are just using KenLM the wrong way. Once you put each sentence onto its own line - like the manual says - it produces the </s> just fine. And now I wonder if I still want to trust their python packages to do the right thing...
- mherrmann 5y ago
  Re unexpected AWS costs: This happened to me too. I hate AWS for it. The best solution I've found is to set up a "Budget" in the AWS console that alerts me by email when my account spends more than I expect. Just last week this caught a case early that would have likely resulted in an unnecessary 4-figure bill.

fxtentacle 5y ago

I'm currently waiting for the Ethereum vote to flood the market with used GPUs and then hopefully new ones will become more affordable, too. And yes, I've also been burned multiple times by runaway AWS costs. Especially since you keep paying for CPU time for instances that are shut down but not fully deleted. That always struck me as odd and most other cloud providers handle that differently. And yes, I would also assume that most AI persons don't have the systems expertise, which is kinda sad. But then again, most AI tutorials are buggy as hell, presumably for the same reason. For example https://huggingface.co/blog/wav2vec2-with-ngram https://huggingface.co/blog/wav2vec2-with-ngram (yes, the people building the Huggingface AI toolkit) says: "The 5-gram correctly includes a "Unknown" or <unk>, as well as a begin-of-sentence, <s> token, but no end-of-sentence, </s> token. This sadly has to be corrected currently after the build." And then follows a cringe-worthy script to "fix" the missing </s> by rewriting a 100+ GB file in python. But actually, they are just using KenLM the wrong way. Once you put each sentence onto its own line - like the manual says - it produces the </s> just fine. And now I wonder if I still want to trust their python packages to do the right thing...

mherrmann 5y ago
Re unexpected AWS costs: This happened to me too. I hate AWS for it. The best solution I've found is to set up a "Budget" in the AWS console that alerts me by email when my account spends more than I expect. Just last week this caught a case early that would have likely resulted in an unnecessary 4-figure bill.