3 ms·

I see we’re doing this again. I last worked on whole-DC designs around 1999 with Sun, Intel, and of course… Mellanox since we were to use RDMA/InfiniBand. The p

I see we’re doing this again. I last worked on whole-DC designs around 1999 with Sun, Intel, and of course… Mellanox since we were to use RDMA/InfiniBand. The plan was to embed InfiniBand semiconductors on disks, RAM (individual sticks or trays of sticks) and CPUs. There was to be an orchestration/provisioning layer allowing you to reserve a composable machine (CPU(s), RAM, storage, other IO, etc) at will, and some kind of supervisory layer was to ensure that these things were as local as possible to one-another in any given rack, aisle, floor.

Traubenfuchs 4y ago
So you were pretty much working on a more hardware based predecessor of kubernetes? Why did it not work out?
- SgtBastard 4y ago
  It wasn’t written in Go </snark> I’d also love to hear more about the concept.
  - sacnoradhq 4y ago
    Go peaked in 2017. Rust is the new hotness. 3 years from now, it will be something else.
- bobleeswagger 4y ago
  > Why did it not work out? Software won, it seems.
  - KaiserPro 4y ago
    kinda, It was cheaper to use fibrechannel/ethernet and plain old sparc/intel machines This was about the time that grid engines promised mainframe computing without the cost. Only that it required people to understand how to parallelise their workloads. That and modify programmes to run in on many machines at once.
- KaiserPro 4y ago
  > Why did it not work out? 1) cost 2) software support 3) cost
- al2o3cr 4y ago
  Some big problems that jump out: * Infiniband hardware didn't get cheaper as fast as RAM + disk * Infiniband hardware didn't get faster as quickly as locally-attached RAM + disk
- nickdothutton 4y ago
  I think a few things combined to torpedo it. Firstly we were hit with the dotcom crash. Secondly Sun’s future rapidly looked less bright, and they were the main server partner in the gang. Thirdly there wasn’t (yet) the “Datacenter OS/Orchestration” layer. It didn’t exist and was the part that would allow us (I think worlds largest ISP and shared/dedicated hosting provider at the time) to carve up the data center on-demand. Remember this is ‘99 so there was no VMware server for another ~2 years. Finally, x86 rack servers got “good enough” and that low cost Pee-Cee architecture took over (with more than a little help from Linux). Yes it was hard to orchestrate. Yes it lead to sprawl (before even VM sprawl) but the market didn’t care.
sacnoradhq 4y ago
Infiniband was mostly displaced by 10, 40, & 100GbE due to costs per port. With costs so high, adoption is reduced. The vendors would've been smart to give away one part of it. Our prod environment doesn't use it because millions of servers would be cost-prohibitive. Corp uses it here and there for experimental clusters.

Traubenfuchs 4y ago

So you were pretty much working on a more hardware based predecessor of kubernetes? Why did it not work out?

SgtBastard 4y ago
It wasn’t written in Go </snark> I’d also love to hear more about the concept.
- sacnoradhq 4y ago
  Go peaked in 2017. Rust is the new hotness. 3 years from now, it will be something else.
bobleeswagger 4y ago
> Why did it not work out? Software won, it seems.
- KaiserPro 4y ago
  kinda, It was cheaper to use fibrechannel/ethernet and plain old sparc/intel machines This was about the time that grid engines promised mainframe computing without the cost. Only that it required people to understand how to parallelise their workloads. That and modify programmes to run in on many machines at once.
KaiserPro 4y ago
> Why did it not work out? 1) cost 2) software support 3) cost
al2o3cr 4y ago
Some big problems that jump out: * Infiniband hardware didn't get cheaper as fast as RAM + disk * Infiniband hardware didn't get faster as quickly as locally-attached RAM + disk
nickdothutton 4y ago
I think a few things combined to torpedo it. Firstly we were hit with the dotcom crash. Secondly Sun’s future rapidly looked less bright, and they were the main server partner in the gang. Thirdly there wasn’t (yet) the “Datacenter OS/Orchestration” layer. It didn’t exist and was the part that would allow us (I think worlds largest ISP and shared/dedicated hosting provider at the time) to carve up the data center on-demand. Remember this is ‘99 so there was no VMware server for another ~2 years. Finally, x86 rack servers got “good enough” and that low cost Pee-Cee architecture took over (with more than a little help from Linux). Yes it was hard to orchestrate. Yes it lead to sprawl (before even VM sprawl) but the market didn’t care.

sacnoradhq 4y ago

Infiniband was mostly displaced by 10, 40, & 100GbE due to costs per port. With costs so high, adoption is reduced. The vendors would've been smart to give away one part of it. Our prod environment doesn't use it because millions of servers would be cost-prohibitive. Corp uses it here and there for experimental clusters.