5 ms·

They aren't wrong but they are basically just digging up the Tandem architecture and adapting it to the current RoCE/FCoE enhanced ethernet we have today. Iron

by jpgvm 6y ago

They aren't wrong but they are basically just digging up the Tandem architecture and adapting it to the current RoCE/FCoE enhanced ethernet we have today. Ironically that is where most of our newer interconnects have their heritage. PCI-e is descendant from Infiniband which in turn is decendant from ServerNet which was developed at Tandem as the internal interconnect for their NonStop systems for the very usecases this paper describes. i.e the very ordered networking this relies on and the architectures it enables were invented ~25 years ago. Unfortunately the reality of today is we don't build data centres and especially not for very highly available systems. Instead we pay that latency penalty to build geographically redundant systems using cheap rented virtual hardware from 3rd parties. This is in part because of how the always on nature of the internet changed the availability requirements for most software (which is now delivered as SaaS) from business hours in 1 timezone to always-on everywhere in the world. It's no longer an excuse for a business critical application to be down because "DC lost power" or some variation of that. I'm old and grumpy though so whatever, everything old becomes new again eventually.

strictfp 6y ago
When working on networked control systems I realized that ordered networking is the reason why automation still uses proprietary protocols and interconnects such as Profibus and Profinet. If you want good guarantees you need determinism.
AceJohnny2 6y ago
that's funny, the tech lead at my last company was formerly from Tandem, and the architecture of our HA product reflected that :) It was a cool system, with architecture completely alien to the consumer products I work on nowadays, and I kinda miss QNX. Did you know a gregk?
nigwil_ 6y ago
"PCI-e is descendant from Infiniband which in turn is decendant from ServerNet ..." When you say PCIe descended from IB, at the signalling layer or a higher layer? Can you suggest a starting place to understand this lineage please, was it in the PCIe standards group discussions? I've not heard of this relationship before and I'm not having much luck finding the original discussion.
- rkagerer 6y ago
  Interesting, I hadn't heard that. I always just assumed it was a descendant of PCI and ISA. Here's a history video on busses from ISA to PCIe: https://youtu.be/51YEyPqvkTk https://youtu.be/51YEyPqvkTk
- p_l 6y ago
  You can see very early signs of PCI Express in intel presentations about 3GIO, which originally had a lot more external cabling in vision (which finally became common with thunderbolt, but with extra tunnel layer)
- w7 6y ago
  Yeah, that statement piqued my curiosity and I'd like a source too. I couldn't tell if it was meant there was a direct lineage or that work and concepts were done in tandem (ha) with the relevant standards bodies and engineers. Googling didn't yield much except InfiniBand's earlier genesis than PCIe, and Intel engineers being involved with InfiniBand and doing the initial work on PCIe.
- jpgvm 6y ago
  PCI-e's IB heritage is present in every aspect of the standard, it's electrically the same, signalling and encoding are the same etc. In the early days of all this there was a few competing standards that all had the same underlying architecture, serial, 8b/10b encoding, etc. The 2 big ones were NGIO and FutureIO. These two were eventually merged and was called ServerIO before being renamed to Infiniband. Infiniband was going to be the next big thing and a few companies set out to make the silicon, Mellanox, QLogic/Silverstorm, Topspin/Cisco among the main ones. Sun and Microsoft both committed to building the drivers to make this work. However pretty soon MS ran into issues with their drivers and eventually pulled the plug on shipping IB drivers in NT. This caused Cisco to drop IB and eventually doomed IB in the enterprise datacenter space and settled for Ethernet. Sun pretty much followed suit for standard enterprise data centers but kept working on IB drivers and IB switches for their HPC and storage units. You can still see it kicking around in stuff like Exadata. Intel is mainly responsible for making PCI-e happen. They had proposed one of the initial standards that became IB, NGIO. When NGIO was fused with FutureIO to create IB the same group renamed it to Arapahoe which was later renamed 3GIO before being renamed to PCI-e when standardized by the PCI SIG. Design wise little changed it just eschewed some of the switching complexity and focused on being an internal system interconnect. So in many ways it's not quite PCI-e is descendant from IB but rather PCI-e -is- IB lol.
foobiekr 6y ago
man, I agree with this so much. reading the tandem technical articles were very enlightening for me when I was a younger engineer.
jiggawatts 6y ago
I recently got to work on some HPE NonStop hardware for the local emergency number call centre. As you can imagine, availability was a critical requirement, because it truly is a 24/7 service and you can't have staff interrupted in a middle of a call when someone is dying. The actual final result was a shambles and couldn't remotely achieve the desired availability level. For example, mirrored or synchronized systems are more difficult to update by their very nature. They're also highly constrained physically, such as having to have physical proximity. They're so expensive that the org couldn't afford enough of them to provide dev/test/staging etc... environments. They were also already out-of-date and barely supported, because upgrading every 5 years was also too expensive. My $0.02 was that modern development practices of highly redundant but ordinary commodity servers would have been both far cheaper and far more robust in practice. Mind you, that specific case had a particularly schizophrenic architecture: They decided they wanted a Tier-1 data centre for availability, but their core application couldn't tolerate the latency increase. So they used Citrix to paper over this, but Citrix is complex and has many moving parts prone to failure. They also wanted security, so they tunnelled everything over the various WAN links. They would have been better off just leaving the servers in the same building as the call centre: no long distance links to break, no need for SSL gateways, no need for fibre rings or router failover, etc... My method would have been to have two call centres, each with local compute. Much simpler, fewer things to break, and truly highly available.
hinkley 6y ago
I joke about how we're just slowly rediscovering all of the tech developed for supercomputers in the late 80's to early 90's. Except it's not really a joke, is it? Time sharing and partitioning were big deals in Big Iron, and here we are piling little iron into heaps and having the same set of problems.
- bob1029 6y ago
  Don't forget the power of batch processing. Even today, you can dramatically speed up processing of work by putting things into batches. Speedups of 3-4 orders of magnitude can easily be realized, even if you hedge and do micro batches (10-1000 microseconds wide). To a consumer of one of these systems, you can make the batched processing occur so quickly as to seem real-time in nature, while still realizing most of the low-level CPU architecture benefits of doing things in reasonably-sized chunks. There are benefits to doing it the old-school way too (i.e. nightly). Giving the business ample time to respond to errors before EOD processing is a huge perk in many industries where the downstream consequences of errors cannot be reconciled. Real-time processing is wonderful, but there are still a lot of places where you don't really need it or where it causes more harm than good.
ignoramous 6y ago
> Unfortunately the reality of today is we don't build data centres and especially not for very highly available systems. Instead we pay that latency penalty to build geographically redundant systems using cheap rented virtual hardware from 3rd parties. Reminds me of AWS' contrasting approach (reliable delivery, out-of-order messages) in dealing with the latency penalty using specialized hardware and a proprietary transport protocol viz. SRD (Scalable Reliable Datagram): https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9167399 https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=916... > They aren't wrong but they are basically just digging up the Tandem architecture and adapting it to the current RoCE/FCoE enhanced ethernet we have today. Pat Helland, who worked on Tandem, published a paper on a novel way to avoid distributed transactions, which I believe is similar to Google Megastore's design (on top of BigTable) and seems (to my untrained eye) spiritually closer to Cloudflare's Durable Objects: https://queue.acm.org/detail.cfm?id=3025012 https://queue.acm.org/detail.cfm?id=3025012

by jpgvm 6y ago

strictfp 6y ago

When working on networked control systems I realized that ordered networking is the reason why automation still uses proprietary protocols and interconnects such as Profibus and Profinet. If you want good guarantees you need determinism.

AceJohnny2 6y ago

that's funny, the tech lead at my last company was formerly from Tandem, and the architecture of our HA product reflected that :) It was a cool system, with architecture completely alien to the consumer products I work on nowadays, and I kinda miss QNX. Did you know a gregk?

nigwil_ 6y ago

"PCI-e is descendant from Infiniband which in turn is decendant from ServerNet ..." When you say PCIe descended from IB, at the signalling layer or a higher layer? Can you suggest a starting place to understand this lineage please, was it in the PCIe standards group discussions? I've not heard of this relationship before and I'm not having much luck finding the original discussion.

rkagerer 6y ago
Interesting, I hadn't heard that. I always just assumed it was a descendant of PCI and ISA. Here's a history video on busses from ISA to PCIe: https://youtu.be/51YEyPqvkTk https://youtu.be/51YEyPqvkTk
p_l 6y ago
You can see very early signs of PCI Express in intel presentations about 3GIO, which originally had a lot more external cabling in vision (which finally became common with thunderbolt, but with extra tunnel layer)
w7 6y ago
Yeah, that statement piqued my curiosity and I'd like a source too. I couldn't tell if it was meant there was a direct lineage or that work and concepts were done in tandem (ha) with the relevant standards bodies and engineers. Googling didn't yield much except InfiniBand's earlier genesis than PCIe, and Intel engineers being involved with InfiniBand and doing the initial work on PCIe.
jpgvm 6y ago
PCI-e's IB heritage is present in every aspect of the standard, it's electrically the same, signalling and encoding are the same etc. In the early days of all this there was a few competing standards that all had the same underlying architecture, serial, 8b/10b encoding, etc. The 2 big ones were NGIO and FutureIO. These two were eventually merged and was called ServerIO before being renamed to Infiniband. Infiniband was going to be the next big thing and a few companies set out to make the silicon, Mellanox, QLogic/Silverstorm, Topspin/Cisco among the main ones. Sun and Microsoft both committed to building the drivers to make this work. However pretty soon MS ran into issues with their drivers and eventually pulled the plug on shipping IB drivers in NT. This caused Cisco to drop IB and eventually doomed IB in the enterprise datacenter space and settled for Ethernet. Sun pretty much followed suit for standard enterprise data centers but kept working on IB drivers and IB switches for their HPC and storage units. You can still see it kicking around in stuff like Exadata. Intel is mainly responsible for making PCI-e happen. They had proposed one of the initial standards that became IB, NGIO. When NGIO was fused with FutureIO to create IB the same group renamed it to Arapahoe which was later renamed 3GIO before being renamed to PCI-e when standardized by the PCI SIG. Design wise little changed it just eschewed some of the switching complexity and focused on being an internal system interconnect. So in many ways it's not quite PCI-e is descendant from IB but rather PCI-e -is- IB lol.

foobiekr 6y ago

man, I agree with this so much. reading the tandem technical articles were very enlightening for me when I was a younger engineer.

jiggawatts 6y ago

I recently got to work on some HPE NonStop hardware for the local emergency number call centre. As you can imagine, availability was a critical requirement, because it truly is a 24/7 service and you can't have staff interrupted in a middle of a call when someone is dying. The actual final result was a shambles and couldn't remotely achieve the desired availability level. For example, mirrored or synchronized systems are more difficult to update by their very nature. They're also highly constrained physically, such as having to have physical proximity. They're so expensive that the org couldn't afford enough of them to provide dev/test/staging etc... environments. They were also already out-of-date and barely supported, because upgrading every 5 years was also too expensive. My $0.02 was that modern development practices of highly redundant but ordinary commodity servers would have been both far cheaper and far more robust in practice. Mind you, that specific case had a particularly schizophrenic architecture: They decided they wanted a Tier-1 data centre for availability, but their core application couldn't tolerate the latency increase. So they used Citrix to paper over this, but Citrix is complex and has many moving parts prone to failure. They also wanted security, so they tunnelled everything over the various WAN links. They would have been better off just leaving the servers in the same building as the call centre: no long distance links to break, no need for SSL gateways, no need for fibre rings or router failover, etc... My method would have been to have two call centres, each with local compute. Much simpler, fewer things to break, and truly highly available.

hinkley 6y ago

I joke about how we're just slowly rediscovering all of the tech developed for supercomputers in the late 80's to early 90's. Except it's not really a joke, is it? Time sharing and partitioning were big deals in Big Iron, and here we are piling little iron into heaps and having the same set of problems.

bob1029 6y ago
Don't forget the power of batch processing. Even today, you can dramatically speed up processing of work by putting things into batches. Speedups of 3-4 orders of magnitude can easily be realized, even if you hedge and do micro batches (10-1000 microseconds wide). To a consumer of one of these systems, you can make the batched processing occur so quickly as to seem real-time in nature, while still realizing most of the low-level CPU architecture benefits of doing things in reasonably-sized chunks. There are benefits to doing it the old-school way too (i.e. nightly). Giving the business ample time to respond to errors before EOD processing is a huge perk in many industries where the downstream consequences of errors cannot be reconciled. Real-time processing is wonderful, but there are still a lot of places where you don't really need it or where it causes more harm than good.

ignoramous 6y ago

> Unfortunately the reality of today is we don't build data centres and especially not for very highly available systems. Instead we pay that latency penalty to build geographically redundant systems using cheap rented virtual hardware from 3rd parties. Reminds me of AWS' contrasting approach (reliable delivery, out-of-order messages) in dealing with the latency penalty using specialized hardware and a proprietary transport protocol viz. SRD (Scalable Reliable Datagram): https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9167399 https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=916... > They aren't wrong but they are basically just digging up the Tandem architecture and adapting it to the current RoCE/FCoE enhanced ethernet we have today. Pat Helland, who worked on Tandem, published a paper on a novel way to avoid distributed transactions, which I believe is similar to Google Megastore's design (on top of BigTable) and seems (to my untrained eye) spiritually closer to Cloudflare's Durable Objects: https://queue.acm.org/detail.cfm?id=3025012 https://queue.acm.org/detail.cfm?id=3025012