6 ms·

Has there ever been a precedent for this? When there were major bugs in Intel CPU's (or drives, or RAM, or motherboards) did the likes of Amazon and Google inve

Has there ever been a precedent for this? When there were major bugs in Intel CPU's (or drives, or RAM, or motherboards) did the likes of Amazon and Google invest in diversification? And has it affected stock prices meaningfully? My guess is that they'll see this as just another one off issue that can be fixed with software, then move on. For a large enterprise, monoculture that works is actually better than diversification. When you think about your own workstation, it's not a big deal to build an Intel or AMD system. But when you buy 100k motherboards and spend the time adjusting your tooling to those, from packaging to power, to cooling, to support, to OS code, etc. and then you on a whim decide to get another 100k motherboards of a different architecture, you spend a non-trivial amount of time and money to support those as well. Again, if AMD provides better hardware, it's absolutely worth it. But I personally wouldn't do it based on this bug. I don't own shares of either AMD or Intel.

revelation 9y ago
I checked the stock of Intel during the FDIV bug (1994/1995) where they had to go as far as recalling the affected processors at a cost of $500M in January 1995 and there was basically zero effect. By the end of 1995 the stock had actually pretty much doubled in value..
- sitkack 9y ago
  I personally think FDIV made Intel money, it told the world how important Intel was. It wasn't just the calculator sitting on some trader's desk. It ran the stock market and the stock market responded.
uhhhhhhh 9y ago
When you're building out your DC, its a function of cost relative to performance/power use. I could see new setups may look at AMD over Intel, especially if they're running workloads impacted by the software fix, at least for the current generation of CPU's, maybe even the next 1-2 that are in the pipeline. When you're scaling up/maintaining your DC, you're much more likely to be looking for single sku, like for like products that allow similar tooling, knowledge base, experience etc... Like you said, monoculture has its benefits in some situations. Personally even with this bug I'd be very hesitant to switch. Our previous tests between them for very specific workloads showed our best cost/performance was with Intel over successive generations, and the scaling/tip over points were different. We have a combination of experience and knowledge around the existing arch and how our applications and workloads interact with it that involved a number of pain points that I'm not sure its worth it to re-experience with another arch. On the other hand, those running on non-bare metal, cloud based, auto-scaling/automated solutions that have a wider tolerance for individual app performance, are probably in a situation where they care less about this, but at the same time have little to no say in the arch they run on, that decision is left to the cloud providers they use. just my 2 cents anyways.
zrm 9y ago
> For a large enterprise, monoculture that works is actually better than diversification. But that's the problem. Nothing works 100% of the time, which is why monoculture is bad. When there is a bug that affects 20% of your systems, you can continue operating at 80% capacity, which at a reasonable level of reserve/redundancy means you're still entirely up. With a monoculture the bug affects everything and you're entirely down. > But when you buy 100k motherboards and spend the time adjusting your tooling to those, from packaging to power, to cooling, to support, to OS code, etc. and then you on a whim decide to get another 100k motherboards of a different architecture, you spend a non-trivial amount of time and money to support those as well. This is why hardware abstraction is a thing. It's almost always less expensive to support diverse hardware from the beginning than to wait until after the market shifts. Eventually the day comes to switch from 68K to PowerPC, or PowerPC to Intel, or Intel to ARM, or ARM to whatever else. Because eventually you save/gain a zillion dollars by switching and it "only" costs three quarters of a zillion to switch. But it would have cost a tenth that much to have supported diverse hardware from the start, and then the transition is only a matter of using more of the now-superior hardware rather than being stuck on the now-inferior hardware for potentially years while everything is rearchitected from scratch.
- IgorPartola 9y ago
  > This is why hardware abstraction is a thing. This is the mistake in your argument. Motherboards, CPUs, RAM chips, GPUs are analog and physical objects. For AWS to switch a DC from one mobo to another just to find out that this one draws 5% more power and their standard backup generator can't handle it, which now starts a chain reaction of upgrades is going to incur real world costs. Costs that can't be amortized by writing some code to make the motherboards look the same. This is basically the A/B testing/one-armed bandit problem. How much do you spend time exploring alternatives vs how much time to you reap the benefits of the fact that all your hardware is exactly the same and best of breed, as based on your testing? > When there is a bug that affects 20% of your systems, you can continue operating at 80% capacity, which at a reasonable level of reserve/redundancy means you're still entirely up. With a monoculture the bug affects everything and you're entirely down. These situations simply don't lose enough money to make up for the gains of a monoculture. Think about it this way. You probably run or at least know someone who runs SaaS products. Do you/they use five different cloud providers in equal measure to make sure you have diversity if one has an issue? Do you/they use five different software stacks in case there is a remote exploit for RoR and PHP holds things up? Do you/they buy groceries at five different grocery stores in case one of them has an e. coli outbreak that the other four don't? The answer to all that is now, because no matter how you try to abstract these things, there is a meaningful difference between PHP vs Django vs RoR vs Express vs .NET and between AWS and GC and Azure, that it would cost you a lot more, not just in billing but in engineering effort to support. Another example: chances are you've at some point built a RAID array. Did you put different size and performance drives from different manufacturers into it or did you buy N of the same drive type to ensure even performance? If so, why? Put another way, how much more are you willing to pay on your AWS bill to ensure they are running a mix of ARM, AMD, PowerPC, and Intel chips? Because my guess is that it won't be in the range of 1-2%.
shaklee3 9y ago
Google has stated that they would move to the power9 if the performance claimes live up to the hype. That has yet to be seen.
- kev009 9y ago
  POWER has been better performing than intel since basically 1990, though intel's tick/tock cadence and trading blows in fab tech have kept things interesting. That shouldn't be surprising since POWER is ultra focused on the high end and intel is fending off attacks from the low end and never had good long term thinking on the high end. The reason every server isn't POWER is: ecosystem. For any random company, switching archs for anything less than a multiple factor gain is a daunting multi-generation proposition. For a hyperscaler like Google the bar is a lot lower but you need a compliant vendor that will do a lot of the long haul platform work. IBM's been trying to establish that for many years and is just about to pull it off. Supply chain is also important, hyperscalers have come to expect buying and building systems a certain way and IBM will now just sell chips or even the IP for you to fab yourself. And of course the total cost calculus: capex, and opex in the form of TDP, support burden. Google _will_ be using P9 for GPU servers internally. The inflection point for them was I/O and memory bandwidth. So, paradigm shift was what was needed to turn a juggernaut.. and that is what adding a bunch of accelerators to your platform is. Intel has no good solution there.
  - shaklee3 9y ago
    Right. I think HPC will be the first to take on POWER9 since it has some huge advantages with CAPI and PCIe v4. Outside of that, it will take some run time to convince the larger cloud providers it's useful. I believe POWER9 has the ability to be either big or little endian as well, so that helps for compatibility issues, and it's just a matter of whether your application can compile.

revelation 9y ago

I checked the stock of Intel during the FDIV bug (1994/1995) where they had to go as far as recalling the affected processors at a cost of $500M in January 1995 and there was basically zero effect. By the end of 1995 the stock had actually pretty much doubled in value..

sitkack 9y ago
I personally think FDIV made Intel money, it told the world how important Intel was. It wasn't just the calculator sitting on some trader's desk. It ran the stock market and the stock market responded.

uhhhhhhh 9y ago

When you're building out your DC, its a function of cost relative to performance/power use. I could see new setups may look at AMD over Intel, especially if they're running workloads impacted by the software fix, at least for the current generation of CPU's, maybe even the next 1-2 that are in the pipeline. When you're scaling up/maintaining your DC, you're much more likely to be looking for single sku, like for like products that allow similar tooling, knowledge base, experience etc... Like you said, monoculture has its benefits in some situations. Personally even with this bug I'd be very hesitant to switch. Our previous tests between them for very specific workloads showed our best cost/performance was with Intel over successive generations, and the scaling/tip over points were different. We have a combination of experience and knowledge around the existing arch and how our applications and workloads interact with it that involved a number of pain points that I'm not sure its worth it to re-experience with another arch. On the other hand, those running on non-bare metal, cloud based, auto-scaling/automated solutions that have a wider tolerance for individual app performance, are probably in a situation where they care less about this, but at the same time have little to no say in the arch they run on, that decision is left to the cloud providers they use. just my 2 cents anyways.

zrm 9y ago

> For a large enterprise, monoculture that works is actually better than diversification. But that's the problem. Nothing works 100% of the time, which is why monoculture is bad. When there is a bug that affects 20% of your systems, you can continue operating at 80% capacity, which at a reasonable level of reserve/redundancy means you're still entirely up. With a monoculture the bug affects everything and you're entirely down. > But when you buy 100k motherboards and spend the time adjusting your tooling to those, from packaging to power, to cooling, to support, to OS code, etc. and then you on a whim decide to get another 100k motherboards of a different architecture, you spend a non-trivial amount of time and money to support those as well. This is why hardware abstraction is a thing. It's almost always less expensive to support diverse hardware from the beginning than to wait until after the market shifts. Eventually the day comes to switch from 68K to PowerPC, or PowerPC to Intel, or Intel to ARM, or ARM to whatever else. Because eventually you save/gain a zillion dollars by switching and it "only" costs three quarters of a zillion to switch. But it would have cost a tenth that much to have supported diverse hardware from the start, and then the transition is only a matter of using more of the now-superior hardware rather than being stuck on the now-inferior hardware for potentially years while everything is rearchitected from scratch.

IgorPartola 9y ago
> This is why hardware abstraction is a thing. This is the mistake in your argument. Motherboards, CPUs, RAM chips, GPUs are analog and physical objects. For AWS to switch a DC from one mobo to another just to find out that this one draws 5% more power and their standard backup generator can't handle it, which now starts a chain reaction of upgrades is going to incur real world costs. Costs that can't be amortized by writing some code to make the motherboards look the same. This is basically the A/B testing/one-armed bandit problem. How much do you spend time exploring alternatives vs how much time to you reap the benefits of the fact that all your hardware is exactly the same and best of breed, as based on your testing? > When there is a bug that affects 20% of your systems, you can continue operating at 80% capacity, which at a reasonable level of reserve/redundancy means you're still entirely up. With a monoculture the bug affects everything and you're entirely down. These situations simply don't lose enough money to make up for the gains of a monoculture. Think about it this way. You probably run or at least know someone who runs SaaS products. Do you/they use five different cloud providers in equal measure to make sure you have diversity if one has an issue? Do you/they use five different software stacks in case there is a remote exploit for RoR and PHP holds things up? Do you/they buy groceries at five different grocery stores in case one of them has an e. coli outbreak that the other four don't? The answer to all that is now, because no matter how you try to abstract these things, there is a meaningful difference between PHP vs Django vs RoR vs Express vs .NET and between AWS and GC and Azure, that it would cost you a lot more, not just in billing but in engineering effort to support. Another example: chances are you've at some point built a RAID array. Did you put different size and performance drives from different manufacturers into it or did you buy N of the same drive type to ensure even performance? If so, why? Put another way, how much more are you willing to pay on your AWS bill to ensure they are running a mix of ARM, AMD, PowerPC, and Intel chips? Because my guess is that it won't be in the range of 1-2%.

shaklee3 9y ago

Google has stated that they would move to the power9 if the performance claimes live up to the hype. That has yet to be seen.

kev009 9y ago
POWER has been better performing than intel since basically 1990, though intel's tick/tock cadence and trading blows in fab tech have kept things interesting. That shouldn't be surprising since POWER is ultra focused on the high end and intel is fending off attacks from the low end and never had good long term thinking on the high end. The reason every server isn't POWER is: ecosystem. For any random company, switching archs for anything less than a multiple factor gain is a daunting multi-generation proposition. For a hyperscaler like Google the bar is a lot lower but you need a compliant vendor that will do a lot of the long haul platform work. IBM's been trying to establish that for many years and is just about to pull it off. Supply chain is also important, hyperscalers have come to expect buying and building systems a certain way and IBM will now just sell chips or even the IP for you to fab yourself. And of course the total cost calculus: capex, and opex in the form of TDP, support burden. Google _will_ be using P9 for GPU servers internally. The inflection point for them was I/O and memory bandwidth. So, paradigm shift was what was needed to turn a juggernaut.. and that is what adding a bunch of accelerators to your platform is. Intel has no good solution there.
- shaklee3 9y ago
  Right. I think HPC will be the first to take on POWER9 since it has some huge advantages with CAPI and PCIe v4. Outside of that, it will take some run time to convince the larger cloud providers it's useful. I believe POWER9 has the ability to be either big or little endian as well, so that helps for compatibility issues, and it's just a matter of whether your application can compile.