3 ms·

Some maybe-interesting observations about my experiences over the last few years, as someone who uses both cloud-provisioned (GCP N2D) and dedicated-server (e.g

by derefr 3y ago

Some maybe-interesting observations about my experiences over the last few years, as someone who uses both cloud-provisioned (GCP N2D) and dedicated-server (e.g. OVH HGR-HCI-class) AMD EPYC-based machines at $work. • GCP N2D instances always had a strict per-AZ allocation quota. This allocation quota has not increased over time. And when we asked to have it bumped up, it was the only time a quota-increase request of ours has ever been denied. • When OVH was offering their HGR-HCI-6 machine type (2x EPYC 7532), we provisioned a few of them. The first few, leased ~2 years back, each took a few days to provision — presumably, OVH doesn't buy these expensive CPUs until a customer asks for a machine to be stood up with one in them. More recently, though (~6mo ago), for the same machine type, they gave us a provisioning lead time of more than a month, due to supply difficulties for the CPU. • These chips were buggy! Again on OVH, when allocating these HGR-HCI-6 machines, we were allocated two separate machines that ended up having CPU faults. (Symptoms: random reboots after an hour or two of heavily utilizing the native AES-NI instructions; and spurious "PCI-e link training errors" in talking to the network card and/or NVMe drives.) They were replaced quickly, but I've never seen this kind of CPU fault on a hardware-managed system before or since. • Just a month ago, the high-end dedicated-server hosters (OVH, but also Hetzner and so forth) seem to have removed all their SKUs that use 2nd- and 3rd-gen EPYC 7xxx CPUs. (Except for one baseline SKU on OVH, which is probably there because they have a big pile of them.) Everything suddenly switched over to 4th-gen 9xxx EPYCs just a month or two ago. It might just be that availability of these 9xxx EPYCs is finally reaching levels where these providers think they can meet demand with them — but everyone switching over simultaneously, and dropping their old SKUs at the same time? • GCP recently launched the storage-optimized Z3 instance type. They chose to build this instance type on an Intel platform (Sapphire Rapids.) That's even though AMD EPYCs have had enough PCIe lanes to deliver equivalent performance to this Z3 platform — ignoring the "Titanium offload" part, which isn't CPU-platform-specific — for years. (In fact, the need for a huge pool of fast NVMe is in part why we switched some of our base load from GCP over to those OVH HGR-HCI-6 instances — which satisfied our needs quite well.) GCP could in theory have launched something akin to this instance type, with the same 36TiB storage pool size (but PCIe 4.0 speeds rather than 5.0) three years ago, using EPYC 7xxxs. Cusomers have been asking for something like that for years now — wondering why GCP instances are all limited to 8.8TiB of local NVMe. (We actually asked them ourselves, back then, where "the instance type with more local NVMe" was. They gave a very handwave-y response, which in retrospect, may have been a "we're trying, but it's not looking good for delivering this at scale right now" response.) These points all lead me to believe that something weird happened with the EPYC 7xxx rollout. Supply didn't grow to meet demand over time. And then, suddenly, after the seeming EOL of these chips — but long before cloud providers would normally cycle them out — we're seeing 7xxxs ending up on the open market, in enough bulk to make them affordable? Bizarre. --- My own vague hypothesis at this point, is that at some point during the generation, AMD discovered a fatal flaw in the silicon of the entire EPYC 7xxx platform. Maybe it was the hardware crypto instructions, like I saw. Or maybe it was some capability more specific to cloud-computing customers (SEV-SNP?) that turned out to not work right (which would make more sense given that Threadrippers didn't see the same problems.) So the big cloud customers immediately halted their purchase orders (keeping only what they had already installed so as to not disturb existing customer workloads); and AMD responded by scaling down production. This resulted in two things: a supply shock of AMD EPYC-based machines/VMs that lasted for a while; but also, negotiated settlements with the cloud vendors, where AMD was now obligated to fulfill existing POs for 7xxx parts with 9xxxs, as they ramped up production of those. Which is why 9xxxs have taken so long (2 years!) to make it onto the open market: the lines have been dedicated to fulfilling not just 9xxx bulk purchase-orders, but also 7xxx purchase-orders. (And which is why the switchover to 9xxx among smaller players is so immediate: such a switchover has been on every hosting company's roadmap for a long time now, having been repeatedly delayed by supply issues due to the huge volume of 9xxx parts required to satisfy the clouds' backlogged demand. They've had a stock of 9xxx-compatible motherboards + memory + PSUs + chassis just sitting there for months/years now, waiting for 9xxx CPUs to slot into them.) Perhaps we're seeing these cloud-customer 7Cxx parts on the open market now, because the clouds have finally received enough 9xxxs to satisfy their actual demand for 9xxxs, and their backlogged demand for 7xxxs; and the clouds are now finally at the point where they can replace their initial actual (faulty / feature-disabled) 7xxx parts they were sent with 9xxxs, selling off the 7xxx parts. My guess is that, now that they have "fixed" AMD chips in place, we'll soon see the cloud providers heavily hyping up some particular AMD-silicon-enabled feature that they had been starting to market four years ago, but then went radio-silent on. ("Confidential computing", maybe.) --- I'd love to hear what someone with more insider knowledge thinks is happening here.

AnthonyMouse 3y ago
CPU faults on individual machines aren't that rare. The machine has a dodgy power supply that almost works but has voltage drop under load etc. Sometimes this can be caused by environmental factors. The rack is positioned poorly and has thermal issues, the UPS is supplying bad power etc. Then you can see issues with multiple machines, or replace the machine without fixing the issue. Vendors often put machines for the same customer in the same rack for various reasons, e.g. because they might send a lot of traffic to each other and put less load on their network if connected to the same switch, but then if there is a problem in that rack it affects more of your machines. The Epyc 7000 series was popular. There have been enough of them in private hands for long enough that if there were widespread issues they would be well-known. It's possible that AMD didn't order enough capacity from TSMC to meet demand, and couldn't get more during the COVID supply chain issues. For the 9000 series they learned from their mistake, or there is otherwise more fab capacity available now, so customers can get them. Meanwhile cloud providers really like Zen4c because they can sell "cores" that cost less and use less power, so they're buying it and replacing their existing hardware as they tend to do regardless. That is typically how they expand their business: If you add more servers you need more real estate and power and cooling. If you replace older servers with faster ones, you don't.
- derefr 3y ago
  To be clear, it was a CPU fault that doesn't occur at all when running e.g. stress-ng, but only (as far as I know) when running our particular production workload. And only after several hours of running our production workload. But then, once it's known to be provokeable for a given machine, it's extremely reliable to trigger it again — in that it seems to take the same number of executed instructions that utilize the faulty part of the die, since power on. (I.e. if I run a workload that's 50% AES-NI and 50% something else, then it takes exactly twice as long to fault as if the workload was 100% AES-NI.) And it isn't provoked any more quickly, by having just provoked it and then running the same workload again — i.e. there's no temporal locality to it. Which would make both "environmental conditions" and "CPU is overheating / overvolting" much less likely as contributing factors. > There have been enough of them in private hands for long enough that if there were widespread issues they would be well-known. Our setup is likely a bit unusual. These machines that experienced the faults, have every available PCIe lane (other than the few given to the NIC) dedicated to NVMe; where we've got the NVMe sticks stuck together in extremely-wide software RAID0 (meaning that every disk read fans in as many almost-precisely-parallel PCIe packets contending for bus time to DMA their way back into the kernel BIO buffers.) On top of this, we then have every core saturated with parallel CPU-bottlenecked activity, with a heavy focus on these AES-NI instructions; and a high level of rapid allocation/dellocation of multi-GB per-client working arenas, contending against a very large and very hot disk page cache, for a working set that's far, far larger than memory. I'll put it like this: some of these machines are "real-time OLAP" DB (Postgres) servers. And under load, our PG transactions sit in WAIT_LWLOCK waiting to start up, because they're actually (according to our profiling) contending over acquiring the global in-memory pg_locks table in order to write their per-table READ_SHARED locks there (in turn because they're dealing with wide joins across N tables in M schemas where each table has hundreds of partitions and the query is an aggregate so no constraint-exclusion can be used. Our pg_locks Prometheus metrics look crazy.) Imagine the TLB havoc going on, as those forked-off heavy-workload query workers also all fight to memory-map the same huge set of backing table heap files. It's to the point that if we don't either terminate our long-lived client connections (even when not idle), or restart our PG servers at least once a month, we actually see per-backend resource leaks that eventually cause PG to get OOMed! The machines that aren't DB servers, meanwhile — but are still set up the same on an OS level — are blockchain nodes, running https://github.com/ledgerwatch/erigon https://github.com/ledgerwatch/erigon, which likes to do its syncing work in big batches: download N blocks, then execute N blocks, then index N blocks. The part that reliably causes the faults is "hashing N blocks", for sufficiently large values of N that you only ever really hit during a backfill sync, not live sync. In neither case would I expect many others to have hit on just the right combination of load to end up with the same problems. (Which is why I don't really believe that whatever problem AMD might have seen, is related to this one. This seems more like a single-batch production error than anything, where OVH happened to acquire multiple CPUs from that single batch.) --- > It's possible that AMD didn't order enough capacity from TSMC to meet demand, and couldn't get more during the COVID supply chain issues. Yes, but that doesn't explain why they weren't able to ramp up production at any point in the last four years. Even now, there are still likely some smaller hosts that would like to buy EPYC 7xxxs at more-affordable prices, if AMD would make them. You need an additional factor to explain this lack of ramp-up post-COVID; and to explain why the cloud providers never started receiving more 7xxxs (which they would normally do, to satisfy legacy clients who want to replicate their exact setup across more AZs/regions.) Server CPUs don't normally have 2-year purchase commitments! It's normally more like 6! Sure, maybe Zen4c was super-marketable to the clouds' customers and saved them a bunch of OpEx — so they negotiated with AMD to drop all their existing spend commitments on 7xxx parts purchases in favor of committing to 9xxx parts purchases. But why would AMD agree to that, without anything the clouds could hold over their head to force them into it? It would mean shutting down many of the 7xxx production lines early, translating to the CapEx for those production lines not getting paid off! Being able to pay off the production lines is why CPU vendors negotiate these long purchase commitments in the first place! And if the clouds are replacing capacity, then where are all those used CPUs going? Take notice that the OP article isn't talking about a used CPU, but a "new server" — namely (I think) this one: https://www.newegg.com/tyan-s8030gm4ne-2t-supports-amd-epyc-7002-7003-series-processor/p/N82E16813151354 https://www.newegg.com/tyan-s8030gm4ne-2t-supports-amd-epyc-... This server was never in an IaaS datacenter. This is a motherboard straight from the motherboard vendor, with an EPYC 7C13 prepopulated into it. This isn't the sort of thing you get when a cloud resells. This is the sort of thing you get when a cloud (or other hosting provider) stops buying unexpectedly — and upstream suppliers/manufacturers/integrators are left holding the bag, of preconfigured-to-spec hardware they no longer have a pre-committed buyer for.

by derefr 3y ago

AnthonyMouse 3y ago

CPU faults on individual machines aren't that rare. The machine has a dodgy power supply that almost works but has voltage drop under load etc. Sometimes this can be caused by environmental factors. The rack is positioned poorly and has thermal issues, the UPS is supplying bad power etc. Then you can see issues with multiple machines, or replace the machine without fixing the issue. Vendors often put machines for the same customer in the same rack for various reasons, e.g. because they might send a lot of traffic to each other and put less load on their network if connected to the same switch, but then if there is a problem in that rack it affects more of your machines. The Epyc 7000 series was popular. There have been enough of them in private hands for long enough that if there were widespread issues they would be well-known. It's possible that AMD didn't order enough capacity from TSMC to meet demand, and couldn't get more during the COVID supply chain issues. For the 9000 series they learned from their mistake, or there is otherwise more fab capacity available now, so customers can get them. Meanwhile cloud providers really like Zen4c because they can sell "cores" that cost less and use less power, so they're buying it and replacing their existing hardware as they tend to do regardless. That is typically how they expand their business: If you add more servers you need more real estate and power and cooling. If you replace older servers with faster ones, you don't.

derefr 3y ago
To be clear, it was a CPU fault that doesn't occur at all when running e.g. stress-ng, but only (as far as I know) when running our particular production workload. And only after several hours of running our production workload. But then, once it's known to be provokeable for a given machine, it's extremely reliable to trigger it again — in that it seems to take the same number of executed instructions that utilize the faulty part of the die, since power on. (I.e. if I run a workload that's 50% AES-NI and 50% something else, then it takes exactly twice as long to fault as if the workload was 100% AES-NI.) And it isn't provoked any more quickly, by having just provoked it and then running the same workload again — i.e. there's no temporal locality to it. Which would make both "environmental conditions" and "CPU is overheating / overvolting" much less likely as contributing factors. > There have been enough of them in private hands for long enough that if there were widespread issues they would be well-known. Our setup is likely a bit unusual. These machines that experienced the faults, have every available PCIe lane (other than the few given to the NIC) dedicated to NVMe; where we've got the NVMe sticks stuck together in extremely-wide software RAID0 (meaning that every disk read fans in as many almost-precisely-parallel PCIe packets contending for bus time to DMA their way back into the kernel BIO buffers.) On top of this, we then have every core saturated with parallel CPU-bottlenecked activity, with a heavy focus on these AES-NI instructions; and a high level of rapid allocation/dellocation of multi-GB per-client working arenas, contending against a very large and very hot disk page cache, for a working set that's far, far larger than memory. I'll put it like this: some of these machines are "real-time OLAP" DB (Postgres) servers. And under load, our PG transactions sit in WAIT_LWLOCK waiting to start up, because they're actually (according to our profiling) contending over acquiring the global in-memory pg_locks table in order to write their per-table READ_SHARED locks there (in turn because they're dealing with wide joins across N tables in M schemas where each table has hundreds of partitions and the query is an aggregate so no constraint-exclusion can be used. Our pg_locks Prometheus metrics look crazy.) Imagine the TLB havoc going on, as those forked-off heavy-workload query workers also all fight to memory-map the same huge set of backing table heap files. It's to the point that if we don't either terminate our long-lived client connections (even when not idle), or restart our PG servers at least once a month, we actually see per-backend resource leaks that eventually cause PG to get OOMed! The machines that aren't DB servers, meanwhile — but are still set up the same on an OS level — are blockchain nodes, running https://github.com/ledgerwatch/erigon https://github.com/ledgerwatch/erigon, which likes to do its syncing work in big batches: download N blocks, then execute N blocks, then index N blocks. The part that reliably causes the faults is "hashing N blocks", for sufficiently large values of N that you only ever really hit during a backfill sync, not live sync. In neither case would I expect many others to have hit on just the right combination of load to end up with the same problems. (Which is why I don't really believe that whatever problem AMD might have seen, is related to this one. This seems more like a single-batch production error than anything, where OVH happened to acquire multiple CPUs from that single batch.) --- > It's possible that AMD didn't order enough capacity from TSMC to meet demand, and couldn't get more during the COVID supply chain issues. Yes, but that doesn't explain why they weren't able to ramp up production at any point in the last four years. Even now, there are still likely some smaller hosts that would like to buy EPYC 7xxxs at more-affordable prices, if AMD would make them. You need an additional factor to explain this lack of ramp-up post-COVID; and to explain why the cloud providers never started receiving more 7xxxs (which they would normally do, to satisfy legacy clients who want to replicate their exact setup across more AZs/regions.) Server CPUs don't normally have 2-year purchase commitments! It's normally more like 6! Sure, maybe Zen4c was super-marketable to the clouds' customers and saved them a bunch of OpEx — so they negotiated with AMD to drop all their existing spend commitments on 7xxx parts purchases in favor of committing to 9xxx parts purchases. But why would AMD agree to that, without anything the clouds could hold over their head to force them into it? It would mean shutting down many of the 7xxx production lines early, translating to the CapEx for those production lines not getting paid off! Being able to pay off the production lines is why CPU vendors negotiate these long purchase commitments in the first place! And if the clouds are replacing capacity, then where are all those used CPUs going? Take notice that the OP article isn't talking about a used CPU, but a "new server" — namely (I think) this one: https://www.newegg.com/tyan-s8030gm4ne-2t-supports-amd-epyc-7002-7003-series-processor/p/N82E16813151354 https://www.newegg.com/tyan-s8030gm4ne-2t-supports-amd-epyc-... This server was never in an IaaS datacenter. This is a motherboard straight from the motherboard vendor, with an EPYC 7C13 prepopulated into it. This isn't the sort of thing you get when a cloud resells. This is the sort of thing you get when a cloud (or other hosting provider) stops buying unexpectedly — and upstream suppliers/manufacturers/integrators are left holding the bag, of preconfigured-to-spec hardware they no longer have a pre-committed buyer for.