3 ms·

To be clear, it was a CPU fault that doesn't occur at all when running e.g. stress-ng, but only (as far as I know) when running our particular production worklo

by derefr 3y ago

To be clear, it was a CPU fault that doesn't occur at all when running e.g. stress-ng, but only (as far as I know) when running our particular production workload. And only after several hours of running our production workload. But then, once it's known to be provokeable for a given machine, it's extremely reliable to trigger it again — in that it seems to take the same number of executed instructions that utilize the faulty part of the die, since power on. (I.e. if I run a workload that's 50% AES-NI and 50% something else, then it takes exactly twice as long to fault as if the workload was 100% AES-NI.) And it isn't provoked any more quickly, by having just provoked it and then running the same workload again — i.e. there's no temporal locality to it. Which would make both "environmental conditions" and "CPU is overheating / overvolting" much less likely as contributing factors. > There have been enough of them in private hands for long enough that if there were widespread issues they would be well-known. Our setup is likely a bit unusual. These machines that experienced the faults, have every available PCIe lane (other than the few given to the NIC) dedicated to NVMe; where we've got the NVMe sticks stuck together in extremely-wide software RAID0 (meaning that every disk read fans in as many almost-precisely-parallel PCIe packets contending for bus time to DMA their way back into the kernel BIO buffers.) On top of this, we then have every core saturated with parallel CPU-bottlenecked activity, with a heavy focus on these AES-NI instructions; and a high level of rapid allocation/dellocation of multi-GB per-client working arenas, contending against a very large and very hot disk page cache, for a working set that's far, far larger than memory. I'll put it like this: some of these machines are "real-time OLAP" DB (Postgres) servers. And under load, our PG transactions sit in WAIT_LWLOCK waiting to start up, because they're actually (according to our profiling) contending over acquiring the global in-memory pg_locks table in order to write their per-table READ_SHARED locks there (in turn because they're dealing with wide joins across N tables in M schemas where each table has hundreds of partitions and the query is an aggregate so no constraint-exclusion can be used. Our pg_locks Prometheus metrics look crazy.) Imagine the TLB havoc going on, as those forked-off heavy-workload query workers also all fight to memory-map the same huge set of backing table heap files. It's to the point that if we don't either terminate our long-lived client connections (even when not idle), or restart our PG servers at least once a month, we actually see per-backend resource leaks that eventually cause PG to get OOMed! The machines that aren't DB servers, meanwhile — but are still set up the same on an OS level — are blockchain nodes, running https://github.com/ledgerwatch/erigon https://github.com/ledgerwatch/erigon, which likes to do its syncing work in big batches: download N blocks, then execute N blocks, then index N blocks. The part that reliably causes the faults is "hashing N blocks", for sufficiently large values of N that you only ever really hit during a backfill sync, not live sync. In neither case would I expect many others to have hit on just the right combination of load to end up with the same problems. (Which is why I don't really believe that whatever problem AMD might have seen, is related to this one. This seems more like a single-batch production error than anything, where OVH happened to acquire multiple CPUs from that single batch.) --- > It's possible that AMD didn't order enough capacity from TSMC to meet demand, and couldn't get more during the COVID supply chain issues. Yes, but that doesn't explain why they weren't able to ramp up production at any point in the last four years. Even now, there are still likely some smaller hosts that would like to buy EPYC 7xxxs at more-affordable prices, if AMD would make them. You need an additional factor to explain this lack of ramp-up post-COVID; and to explain why the cloud providers never started receiving more 7xxxs (which they would normally do, to satisfy legacy clients who want to replicate their exact setup across more AZs/regions.) Server CPUs don't normally have 2-year purchase commitments! It's normally more like 6! Sure, maybe Zen4c was super-marketable to the clouds' customers and saved them a bunch of OpEx — so they negotiated with AMD to drop all their existing spend commitments on 7xxx parts purchases in favor of committing to 9xxx parts purchases. But why would AMD agree to that, without anything the clouds could hold over their head to force them into it? It would mean shutting down many of the 7xxx production lines early, translating to the CapEx for those production lines not getting paid off! Being able to pay off the production lines is why CPU vendors negotiate these long purchase commitments in the first place! And if the clouds are replacing capacity, then where are all those used CPUs going? Take notice that the OP article isn't talking about a used CPU, but a "new server" — namely (I think) this one: https://www.newegg.com/tyan-s8030gm4ne-2t-supports-amd-epyc-7002-7003-series-processor/p/N82E16813151354 https://www.newegg.com/tyan-s8030gm4ne-2t-supports-amd-epyc-... This server was never in an IaaS datacenter. This is a motherboard straight from the motherboard vendor, with an EPYC 7C13 prepopulated into it. This isn't the sort of thing you get when a cloud resells. This is the sort of thing you get when a cloud (or other hosting provider) stops buying unexpectedly — and upstream suppliers/manufacturers/integrators are left holding the bag, of preconfigured-to-spec hardware they no longer have a pre-committed buyer for.

AnthonyMouse 3y ago
> if I run a workload that's 50% AES-NI and 50% something else, then it takes exactly twice as long to fault as if the workload was 100% AES-NI. If you mean it takes twice as long on average, that's consistent with marginal hardware, where the fault is more sensitive to occurring during AES instructions so the more you execute the higher the probability. If you mean it always takes exactly twice as long, that sounds more like a software issue, where there is some counter and when it rolls over its behavior changes. In theory this could be microcode/firmware/library rather than your code, but then it's likely that someone else would have noticed by now. > And it isn't provoked any more quickly, by having just provoked it and then running the same workload again — i.e. there's no temporal locality to it. Which would make both "environmental conditions" and "CPU is overheating / overvolting" much less likely as contributing factors. That's if the workload is inducing higher power consumption or higher temperatures, rather than the voltage or temperature constantly being out of spec but only marginally, so there is at all times a probability of random errors, to which certain types of instructions are more susceptible. > Imagine the TLB havoc going on, as those forked-off heavy-workload query workers also all fight to memory-map the same huge set of backing table heap files. So now there are two possibilities. One, your workload is really unusual and is triggering a rare hardware bug nobody else hits. But then nobody else is worried about or even knows about it, so it wouldn't be affecting the market. Two, it's on the heavy side but not so much that other people don't hit it too, and then the issue would be public. It's not very plausible that the problem could be known to all cloud providers but not the general public. > Yes, but that doesn't explain why they weren't able to ramp up production at any point in the last four years. Even now, there are still likely some smaller hosts that would like to buy EPYC 7xxxs at more-affordable prices, if AMD would make them. They don't always lower the prices of the old models very much, they just keep making them for anyone who wants to keep buying them because they want uniformity. But then anyone who didn't buy a lot of them before isn't going to buy a lot of them now instead of just buying the new ones. The prices come down on the used market from supply and demand, as the people buying new ones and selling the old ones, and then for new stock that retailers already own and want to get it off the shelves to make room for the new. But that doesn't mean you could find new stock of the old model in volume for the lower price. Which is why the people who want that contract for it ahead of time. > But why would AMD agree to that, without anything the clouds could hold over their head to force them into it? Because they're just as happy to make more of the newer models as the older ones, if the production capacity is available. Also, they could just be charging more for them. "Oh, you want 10,000 units for $2500 each instead of the contract which says you have to buy 10,000 units for $2000 each? Okay then." > It would mean shutting down many of the 7xxx production lines early, translating to the CapEx for those production lines not getting paid off! AMD doesn't have production lines, they use TSMC. TSMC, in turn, would just sell the capacity to someone else. > And if the clouds are replacing capacity, then where are all those used CPUs going? The original problem was that they couldn't make enough of them. Also, the cloud providers generally replace their oldest servers. Zen2/Zen3 isn't all that old. They'll be installing Zen4 and taking out Zen1 or five and ten year old Xeons. Which are all over the place on eBay.

by derefr 3y ago

AnthonyMouse 3y ago

> if I run a workload that's 50% AES-NI and 50% something else, then it takes exactly twice as long to fault as if the workload was 100% AES-NI. If you mean it takes twice as long on average, that's consistent with marginal hardware, where the fault is more sensitive to occurring during AES instructions so the more you execute the higher the probability. If you mean it always takes exactly twice as long, that sounds more like a software issue, where there is some counter and when it rolls over its behavior changes. In theory this could be microcode/firmware/library rather than your code, but then it's likely that someone else would have noticed by now. > And it isn't provoked any more quickly, by having just provoked it and then running the same workload again — i.e. there's no temporal locality to it. Which would make both "environmental conditions" and "CPU is overheating / overvolting" much less likely as contributing factors. That's if the workload is inducing higher power consumption or higher temperatures, rather than the voltage or temperature constantly being out of spec but only marginally, so there is at all times a probability of random errors, to which certain types of instructions are more susceptible. > Imagine the TLB havoc going on, as those forked-off heavy-workload query workers also all fight to memory-map the same huge set of backing table heap files. So now there are two possibilities. One, your workload is really unusual and is triggering a rare hardware bug nobody else hits. But then nobody else is worried about or even knows about it, so it wouldn't be affecting the market. Two, it's on the heavy side but not so much that other people don't hit it too, and then the issue would be public. It's not very plausible that the problem could be known to all cloud providers but not the general public. > Yes, but that doesn't explain why they weren't able to ramp up production at any point in the last four years. Even now, there are still likely some smaller hosts that would like to buy EPYC 7xxxs at more-affordable prices, if AMD would make them. They don't always lower the prices of the old models very much, they just keep making them for anyone who wants to keep buying them because they want uniformity. But then anyone who didn't buy a lot of them before isn't going to buy a lot of them now instead of just buying the new ones. The prices come down on the used market from supply and demand, as the people buying new ones and selling the old ones, and then for new stock that retailers already own and want to get it off the shelves to make room for the new. But that doesn't mean you could find new stock of the old model in volume for the lower price. Which is why the people who want that contract for it ahead of time. > But why would AMD agree to that, without anything the clouds could hold over their head to force them into it? Because they're just as happy to make more of the newer models as the older ones, if the production capacity is available. Also, they could just be charging more for them. "Oh, you want 10,000 units for $2500 each instead of the contract which says you have to buy 10,000 units for $2000 each? Okay then." > It would mean shutting down many of the 7xxx production lines early, translating to the CapEx for those production lines not getting paid off! AMD doesn't have production lines, they use TSMC. TSMC, in turn, would just sell the capacity to someone else. > And if the clouds are replacing capacity, then where are all those used CPUs going? The original problem was that they couldn't make enough of them. Also, the cloud providers generally replace their oldest servers. Zen2/Zen3 isn't all that old. They'll be installing Zen4 and taking out Zen1 or five and ten year old Xeons. Which are all over the place on eBay.