5 ms·

I used to love working with E10k/E15k boxes. I was a performance engineer for a telco software provider, and it was so much fun squeezing every single thing out

I used to love working with E10k/E15k boxes. I was a performance engineer for a telco software provider, and it was so much fun squeezing every single thing out of the big iron. It’s a bit sad that nobody gives a shit about performance any more. They just provision more cloud hardware. I saved telcos millions upon millions in my early career. I’d jump straight into it again if a job came up, so much fun.

mlyle 1y ago
> It’s a bit sad that nobody gives a shit about performance any more. They just provision more cloud hardware. It's hard to get as excited about performance when the typical family sedan has >250HP. Or when a Raspberry Pi 5 can outrun a maxxed-E10k on almost everything. ...(yah, less RAM, but you need fewer client connections when you can get rid of them quickly enough).
kstrauser 1y ago
My experience was a bit different. I first saw a Starfire when we were deploying a bunch of Linux servers in the DC. The Sun machine was brilliant, fast, enormous, and far more expensive per unit of work than these little x86 boxes we were carting in. The Starfire started at around $800K. Our Linux servers started at around $1K. The Sun box was not 800x faster at anything than a single x86 box. It was an impressive example of what I considered the wrong road. I think history backs me on this one. > It’s a bit sad that nobody gives a shit about performance any more. Everyone gives a shit about performance at some point, but the answer is horizontal scaling. You can’t vertically scale a single machine to run a FAANG. At a certain vertical scale, it starts to look a helluva lot like horizontal scaling (“how many CPUs for this container? How many drives?”), except in a single box with finite and small limits.
- trollied 1y ago
  I don’t disagree. But most also don’t give a shit and then scale horizontally endlessly, and spend too much money, to deal with their crappy code. As a dev it isn’t your problem if the company you work for just happily provisions and sucks it up.
  - kstrauser 1y ago
    That’s a thing, to be sure. The calculus gets a little complicated when that developer’s pay is far more than the EC2 bill. There’s a spectrum with a small shop wasting $1000 a year hosting inefficient code, and Google-scale where SRE teams would love to put “saved .3% on our cloud bill!” on their annual review.
  - rjsw 1y ago
    > ... to deal with their crappy code written in an interpreted language.
  - mshook 1y ago
    That's my experience as well in 2 different companies where we went from 2 E15K to 2 E25K because it was "cheaper" than rewriting who knows how much code, for how long and at what cost At the other one, we jumped from 2 25k to 2 M9000-64 for the same reasons...
- znpy 1y ago
  > Everyone gives a shit about performance at some point, but the answer is horizontal scaling. You can’t vertically scale a single machine to run a FAANG. You might be surprised about how many companies think they're FAANG (but aren't) though.
  - kstrauser 1y ago
    That’s a whole other story, to be sure! “We absolutely must have multi-region simultaneous writes capable of supporting 300,000,000 simultaneous users!” “Your company sells door knobs and has 47 customers. Throw it on PostgreSQL and call it solved.”
    - znpy 1y ago
      Yup. And if you reach 94 users, just get a larger machine for PostgreSQL and call it a day, if you really think you need it.
- axiolite 1y ago
  > The Sun box was not 800x faster at anything than a single x86 box. You don't buy enterprise gear because it's economical for bulk number-crunching... You buy enterprise gear when you have a critical SPOF application (typically the database) that has to be super-reliable, or that requires greater resources than you can get in commodity boxes. RAS is an expensive proposition. Commodity servers often don't have it, or have much less of it than enterprise gear. Proprietary Unix systems offered RAS as a major selling point. IBM mainframes still have a strong market today. It wasn't until the late 2000's when x86 went to 64-bit, so if your application wanted to gobble more than 2GB/4GB of RAM, you had to go with something proprietary. It was even more recently that the world collectively put a huge amount of effort in, and figured out how to parallelize a large amount of number-crunching problems that were previously limited to single-threaded. There have been many situations like these through the history of computing... Going commodity is always cheaper, but if you have needs commodity systems don't meet, you pay the premium for proprietary systems that do.
  - kstrauser 1y ago
    First, yes, everything you said is true. And especially when you’re supporting an older application designed around such SPOFs, you need those to be bulletproof. That’s completely reasonable. That said, a fair chunk of my work since the 90s has been in building systems that try to avoid SPOFs in the first place. Can we use sharded databases such that upgrading one doesn’t take the others down? Shared-nothing backend servers? M-to-N meshes so we’re not shoving everything through a single load balancer or switch? Redundant data centers? The list goes on. I don’t think that approach is inherently better than what you described. Each has its own tradeoffs and there’s a time and place for both of them. I absolutely did see a lot of Big Iron companies marketing their giant boxes as the “real, proven” alternative to a small cluster of LAMP servers, though. I don’t blame them for wanting to be big players in that market, too, but that wasn’t a good reason to use them (unless you already had their stuff installed and wanted to add a web service next to your existing programs). I wouldn’t run a bank on an EC2 instance, but neither would I ever buy a mainframe to host Wordpress at any scale.
  - jeffbee 1y ago
    You didn't need imaginary 64-bit PCs because a rack full of smaller 64-bit SPARC systems would have been much cheaper than a single E10k. Something that large in a single system was only necessary for people with irreducible memory requirements, ie not delivering mail.
lokar 1y ago
In the end that approach to very high scale and reliability was a dead end. It’s much better and cheaper to solve these problems in software using cheap computers and fast networks.
- trollied 1y ago
  Less cheap computers is still a thing. Entirely missing the point.
  - lokar 1y ago
    A lot of the examples here are things like running a large email service. Doing that with this kind of hardware makes no sense.
    - Henchman21 1y ago
      It might make no sense today, but it made loads of sense back then. One cannot apply modern circumstances backwards in time.
- chasil 1y ago
  If you have applications that run (and rely) on z/OS, this kind a machine makes sense. The e10k didn't have applications like that. Just about everything you could do on it could be made to work on commodity x86 with Linux (after some years, for 64-bit).
  - linksnapzz 1y ago
    Yes, it absolutely did. Tuxedo for transaction processing, for one.
    - chasil 1y ago
      Tuxedo ran on many other vendors operating systems and architectures. It was portable, developed by AT&T. How many other platforms will run IMS?
amiga386 1y ago
I used to work for a telco equipment provider around the time everyone was replacing PDH with SONET. Telcos were gagging to buy our stuff, the main reason being basic hardware advances. Telephone Exchanges / Central Offices have to be in the centre of the lines they serve, meaning some very expensive real estate, and datacenter-level HVAC in the middle of cities is very, very expensive. They loved nothing more than to replace old 1980s switches with ones that took up a quarter to a tenth of the floorspace, used less than half the electricity, and had fabrics that could switch fibre optics directly.

mlyle 1y ago

> It’s a bit sad that nobody gives a shit about performance any more. They just provision more cloud hardware. It's hard to get as excited about performance when the typical family sedan has >250HP. Or when a Raspberry Pi 5 can outrun a maxxed-E10k on almost everything. ...(yah, less RAM, but you need fewer client connections when you can get rid of them quickly enough).

kstrauser 1y ago

My experience was a bit different. I first saw a Starfire when we were deploying a bunch of Linux servers in the DC. The Sun machine was brilliant, fast, enormous, and far more expensive per unit of work than these little x86 boxes we were carting in. The Starfire started at around $800K. Our Linux servers started at around $1K. The Sun box was not 800x faster at anything than a single x86 box. It was an impressive example of what I considered the wrong road. I think history backs me on this one. > It’s a bit sad that nobody gives a shit about performance any more. Everyone gives a shit about performance at some point, but the answer is horizontal scaling. You can’t vertically scale a single machine to run a FAANG. At a certain vertical scale, it starts to look a helluva lot like horizontal scaling (“how many CPUs for this container? How many drives?”), except in a single box with finite and small limits.

trollied 1y ago
I don’t disagree. But most also don’t give a shit and then scale horizontally endlessly, and spend too much money, to deal with their crappy code. As a dev it isn’t your problem if the company you work for just happily provisions and sucks it up.
- kstrauser 1y ago
  That’s a thing, to be sure. The calculus gets a little complicated when that developer’s pay is far more than the EC2 bill. There’s a spectrum with a small shop wasting $1000 a year hosting inefficient code, and Google-scale where SRE teams would love to put “saved .3% on our cloud bill!” on their annual review.
- rjsw 1y ago
  > ... to deal with their crappy code written in an interpreted language.
- mshook 1y ago
  That's my experience as well in 2 different companies where we went from 2 E15K to 2 E25K because it was "cheaper" than rewriting who knows how much code, for how long and at what cost At the other one, we jumped from 2 25k to 2 M9000-64 for the same reasons...
znpy 1y ago
> Everyone gives a shit about performance at some point, but the answer is horizontal scaling. You can’t vertically scale a single machine to run a FAANG. You might be surprised about how many companies think they're FAANG (but aren't) though.
- kstrauser 1y ago
  That’s a whole other story, to be sure! “We absolutely must have multi-region simultaneous writes capable of supporting 300,000,000 simultaneous users!” “Your company sells door knobs and has 47 customers. Throw it on PostgreSQL and call it solved.”
  - znpy 1y ago
    Yup. And if you reach 94 users, just get a larger machine for PostgreSQL and call it a day, if you really think you need it.
axiolite 1y ago
> The Sun box was not 800x faster at anything than a single x86 box. You don't buy enterprise gear because it's economical for bulk number-crunching... You buy enterprise gear when you have a critical SPOF application (typically the database) that has to be super-reliable, or that requires greater resources than you can get in commodity boxes. RAS is an expensive proposition. Commodity servers often don't have it, or have much less of it than enterprise gear. Proprietary Unix systems offered RAS as a major selling point. IBM mainframes still have a strong market today. It wasn't until the late 2000's when x86 went to 64-bit, so if your application wanted to gobble more than 2GB/4GB of RAM, you had to go with something proprietary. It was even more recently that the world collectively put a huge amount of effort in, and figured out how to parallelize a large amount of number-crunching problems that were previously limited to single-threaded. There have been many situations like these through the history of computing... Going commodity is always cheaper, but if you have needs commodity systems don't meet, you pay the premium for proprietary systems that do.
- kstrauser 1y ago
  First, yes, everything you said is true. And especially when you’re supporting an older application designed around such SPOFs, you need those to be bulletproof. That’s completely reasonable. That said, a fair chunk of my work since the 90s has been in building systems that try to avoid SPOFs in the first place. Can we use sharded databases such that upgrading one doesn’t take the others down? Shared-nothing backend servers? M-to-N meshes so we’re not shoving everything through a single load balancer or switch? Redundant data centers? The list goes on. I don’t think that approach is inherently better than what you described. Each has its own tradeoffs and there’s a time and place for both of them. I absolutely did see a lot of Big Iron companies marketing their giant boxes as the “real, proven” alternative to a small cluster of LAMP servers, though. I don’t blame them for wanting to be big players in that market, too, but that wasn’t a good reason to use them (unless you already had their stuff installed and wanted to add a web service next to your existing programs). I wouldn’t run a bank on an EC2 instance, but neither would I ever buy a mainframe to host Wordpress at any scale.
- jeffbee 1y ago
  You didn't need imaginary 64-bit PCs because a rack full of smaller 64-bit SPARC systems would have been much cheaper than a single E10k. Something that large in a single system was only necessary for people with irreducible memory requirements, ie not delivering mail.

lokar 1y ago

In the end that approach to very high scale and reliability was a dead end. It’s much better and cheaper to solve these problems in software using cheap computers and fast networks.

trollied 1y ago
Less cheap computers is still a thing. Entirely missing the point.
- lokar 1y ago
  A lot of the examples here are things like running a large email service. Doing that with this kind of hardware makes no sense.
  - Henchman21 1y ago
    It might make no sense today, but it made loads of sense back then. One cannot apply modern circumstances backwards in time.
chasil 1y ago
If you have applications that run (and rely) on z/OS, this kind a machine makes sense. The e10k didn't have applications like that. Just about everything you could do on it could be made to work on commodity x86 with Linux (after some years, for 64-bit).
- linksnapzz 1y ago
  Yes, it absolutely did. Tuxedo for transaction processing, for one.
  - chasil 1y ago
    Tuxedo ran on many other vendors operating systems and architectures. It was portable, developed by AT&T. How many other platforms will run IMS?

amiga386 1y ago

I used to work for a telco equipment provider around the time everyone was replacing PDH with SONET. Telcos were gagging to buy our stuff, the main reason being basic hardware advances. Telephone Exchanges / Central Offices have to be in the centre of the lines they serve, meaning some very expensive real estate, and datacenter-level HVAC in the middle of cities is very, very expensive. They loved nothing more than to replace old 1980s switches with ones that took up a quarter to a tenth of the floorspace, used less than half the electricity, and had fabrics that could switch fibre optics directly.