4 ms·
Most large farms don't have the ability to properly tune cards for best ROI. At scale, it is too hard because each card is a snowflake in terms of performance.
by latchkey 4y ago
Most large farms don't have the ability to properly tune cards for best ROI. At scale, it is too hard because each card is a snowflake in terms of performance. In order to tune, you're basically pushing the card to the limits of underclock/undervolt... which means the card crashes. Therefore, most large farms actually undertuned, for stability, which is the least abusive on the cards.
If the card crashes due to tuning, the machine crashes, and the rest of the cards are taken offline until the machine reboots. In order to change settings, you have to reboot the entire machine too. It is a pain in the ass to find optimal tuning at scale.
There is no publicly available software to 'autotune' cards at scale. It took me a long time to first learn how to tune cards (by hand tuning thousands of cards and getting a feeling for things) and then once I figured that out, I built software to do this. It required changes in the AMD driver to get it to output in the kernel messages, which card was crashing. It required updating the mining software itself. It was a nutty insane project, but it worked well. We hit our target hashrate / power usage.
- hailwren 4y agoWhat are you doing now? Did you shut the project down entirely?
- latchkey 4y agoSome is shut down, some is mining other coins. Kind of a wait and see time right now.
- arcticbull 4y agoThe planet thanks you for shutting down. Don't worry, I'd say the same for any tobacco executive.
- latchkey 4y agoOur power is unused hydro, in a geographic location full of other data centers for all the large tech companies. This comment is just your continued trolling with your anti-crypto trope. Please try another tactic. Update: I got downvoted to -4 on this one after getting several upvotes. Makes me think that articbull has multiple accounts and is just gaming the system.
- mattwilsonn888 4y agoNot only have I debated this person multiple times, but on a search about settlement a HN thread came up and he had the top comment citing the infamous Mora et al study which makes the naive assumption that required hash for a proof of work network scales linearly with transaction volume, rather than block volume (which come at a constant rate no matter what the total hash is). This person is a successful troll, and a pathetic thinker.
- arcticbull 4y agoThere's no such thing. That power should be connected to the grid instead of wasted or put to productive use, not to mention all those GPUs that are about to head for the landfill after a long life of guessing random numbers and warming up the outside. Wasting the power for money disincentivizes storage and grid connection.
- latchkey 4y agoYou are hilarious, your crypto hatred has clouded your judgement. Since you have so much karma here, you don't mind saying silly things and losing points left and right. Transmission of power is expensive and lossy. These data centers are in a very remote location that isn't heavily populated. That's why they are there to begin with! The power would absolutely go to waste otherwise. That's also why we picked that area... power was cheap because it wasn't being used. Our GPUs are 'new old stock'. They were already produced, sitting in warehouses not doing anything, and would have gone to waste anyway. Update: Similar with the downvotes...
- arcticbull 4y ago> You are hilarious, your crypto hatred has clouded your judgement. Since you have so much karma here, you don't mind saying silly things and losing points left and right. First question, where do you think I got most of the karma? Sentiment here is broadly against crypto and has been for years (although it comes and goes). The pro side is loud, but enjoys far less support than you might imagine. > Transmission of power is expensive and lossy. No, it isn't. Transmission of power is super efficient. The EIA says the US lost only 5% of its electricity to T&D over the 2016-2020 period. [1] Expensive it may be, but that's not a reason not to do it. A single long power line from Boston to LA would only see ~25% loss from one end to the other. The whole Ohm's law thing (P=IV), we use transformers that are 99%+ efficient to raise the voltage to ~345kV and big fat wires to reduce current and resistance and hence transmission line losses. > That's also why we picked that area... power was cheap because it wasn't being used. It should be used to do something productive, not generate e-waste. > Our GPUs are 'new old stock'. They were already produced, sitting in warehouses not doing anything, and would have gone to waste anyway. Or, prices would have gone down and more people would have bought them. I can't see 3060s getting scrapped due to lack of demand when they were going for $850 a pop. [1] https://www.eia.gov/tools/faqs/faq.php?id=105&t=3 https://www.eia.gov/tools/faqs/faq.php?id=105&t=3
- superjan 4y agoI don’t particularly like crypto but I don’t think we need to rehash the same debate again. Isn’t it interesting to hear what this guy is going to do? Passing judgement does not improve the conversation.
- Kerbonut 4y agoConsidering the current state of PoW mining, it doesn’t seem to me like there is much advantage to holding secrets on auto tuning and diagnostics… (correct me if I’m wrong), would you consider releasing any of that work to the community? Auto tuning and diagnostic information would be incredibly useful for almost everyone.
- latchkey 4y agoMy knowledge is very specific to just these cards. The only 'secret', which isn't a secret, was that I realized that since this is hardware, we know the maximum settings (hash/watt) that should be possible. Therefore, I set the cards to that best setting and then tune down from there. This is the opposite train of thinking than most other miners. Most people think, let's start low and then tune up from there to make them 'faster'. The cards crash when they can't handle the settings, so it turns out that tuning down is a better way to tune since they stop crashing when they are stable... and thus don't need tuning from there. There are 3 different sets of 'knobs' to tweak, so I had to build an algo to adjust the knobs in the right order to tune things down. I just had the concept of 'current -> next settings'. Temperature and power fluctuations can make the cards crash too... so by always tuning down, you're always heading towards more stability instead of instability. Since neither of those could be controlled, machines would reboot randomly all the time. The software I built was a golang daemon that ran on each machine and watched for these crashes and modified the tuning of each card individually. The daemon is pretty cool as it is effectively a task runner. I had different tasks to configure and monitor the machines as well. The machines are all independent, idempotent and self-healing workers. Reliably distributing the software to 20k different workers, is a fun challenge. There are a ton of unit tests, so that helped a lot. If I have the energy, I may rip out the tasks from the daemon, turn it into a library and open source that. It is kind of a fun project that could be useful for others trying to manage large scale individual workers. Tasks could easily be 'apt install' or monitoring utilities. I even bundled node_exporter into the binary, so that we could monitor the machines with prometheus.
- Ruphin 4y agoWith Ethereum no longer in the game the total USD issuance for GPU mining probably dropped over 80%. Is your strategy to just wait out for miners to capitulate until mining is profitable again? Are you not worried that this will crater the value of your hardware?
- latchkey 4y agoWe bootstrapped the company on mining, knowing full well that it would eventually end, and our business model goes beyond just mining ETH. That was another reason to go with the RX4/5 series... fastest route to ROI.
- Melatonic 4y agoYou should really pivot and start selling that software. Package it up with a nice pretty bow and wrapper and it could be a very lucrative product.