14 ms·
Reinventing the Network Stack for Compute-Intensive Applications
- zamadatix 7y ago8 dual port 100G NICs (available for a while)or 2 dual port 400 gigabit NICs (not yet available) would out-bandwidth the memory controller on an Epyc 7742; how are NICs such a bottleneck that they need to be increased 100x to keep up when DDR5 only doubles the bandwidth? I forget what the bandwidth of CPU cache is but I'm guessing it's not 10 terabit/second either.
- Sephr 7y agoHPC needs HBM or TCI memory, not DDR5. Systems using HBM[1] and TCI[2] can already push an aggregate bandwidth of 8 Tbps. 1. https://en.wikipedia.org/wiki/High_Bandwidth_Memory https://en.wikipedia.org/wiki/High_Bandwidth_Memory 2. https://www.hotchips.org/wp-content/uploads/hc_archives/hc26/HC26-11-day1-epub/HC26.11-3-Technology-epub/HC26.11.320-Chip_Stacking-Ditzel_Kuroda_Lee.-ThruChip-v24_final_submission.pdf https://www.hotchips.org/wp-content/uploads/hc_archives/hc26...
- zamadatix 7y agoIt's a good point that HBM has high aggregate bandwidth but I still don't think it makes sense to call it a 100x server/network boundary bottleneck when the already available NIC is faster than NVlink in the first place. What "server/network boundary" is in this case might not be the classical boundary though so maybe they also mean the same thing I'm saying just from a different perspective.
- petra 7y agoThe radeon VII already has 1TB/s memory bandwidth, using HBM2, with HBM2E offering almost double the bandwidth. Also if we're looking a bit forward, Intel recently demoed, with Ayar Labs, a 2.5D chip with a photonic chiplet that can do optical I/O at 1tbps/mm2.
- lmilcin 7y agoYou forget we need network performance to be wasted by badly designed software.
- mrb 7y ago«how are NICs such a bottleneck» Simple: when the packet data does not need to be processed by the CPU. For example a router forwarding network packets at 10 Tbit/s. The data can stay in the NIC cache as it is being forwarded. No PCIe/CPU/RAM bottleneck here. Also, EPYC Rome has 1.64 Tbps of RAM bandwidth today (eight DDR4-3200 channels). 10 Tbps is less than three doublings away. It's conceivable server CPUs can reach this bandwidth in 4-6 years.
- GhettoMaestro 7y agoYou still need to account for NIC-to-NIC packet transfers for scenarios where pkt arrives on Phy NIC A and needs to egress via Phy NIC B. Obviously there are better options than just PCIe transport these days.
- Cyph0n 7y agoThe ASR-9000 series can handle up to 3.2 Tbps of L3 traffic per linecard, but this is only achievable because of the dedicated routing ASICs.
- zamadatix 7y ago32 port 400G (12.8 terabit/s) 1u routers already exist today, the conversation is about the NIC at the server <-> network boundary not switching ASICs. The only reason you don't see 400G NICs in servers is the lack of the servers ability to put that much bandwidth over PCIe (the real bottleneck location).
- the8472 7y agoIn principle you can use P2P-DMA to shunt the bulk of the data to a specialized device (e.g. GPU, FPGA, storage) without it ever touching main memory or the CPU.
- jdc 7y agoThis is (sort of) a forget everything you know about ______ scenario. But I think the short answer is multi-die interconnects. https://www.anandtech.com/show/14211/intels-interconnected-future-chipslets-emib-foveros https://www.anandtech.com/show/14211/intels-interconnected-f... https://semiengineering.com/using-silicon-bridges-in-packages/ https://semiengineering.com/using-silicon-bridges-in-package...
- shaklee3 7y agoThis is what RDMA solves. you are only limited by the amount of pcie switches you stack up in your topology, and not at all by the processor anymore. all of your data is either handled directly by the nic, or offloaded to an accelerator card. Modern systems can support about 1.6Tbps (100G NICs). When pcie 4 comes out, this should double.
- blattimwind 7y ago> I forget what the bandwidth of CPU cache is but I'm guessing it's not 10 terabit/second either. L2/L3 have bandwidths around 1-1.5 TB/s these days. Which pretty much is 10 TBit/s ;)
- ChuckMcM 7y agoFrom the article -- “The true bottleneck for processor throughput is the network interface used to connect a machine to an external network, such as an Ethernet, therefore severely limiting a processor’s data ingest capability,” said Dr. Jonathan Smith, a program manager in DARPA’s Information Innovation Office (I2O). “Today, network throughput on state-of-the-art technology is about 1014 bits per second (bps) and data is processed in aggregate at about 1014 bps. Current stacks deliver only about 1010 to 1011 bps application throughputs.” I don't disagree with this, but see the challenge not so much as the transmission speed between nodes, rather I see it as the 'semantics' for expressing addressing and placement. (and yes, I am a big fan of RDMA :-)) One of the things I helped invent/design back when I was at NetApp was some network attached memory or NAM. NetApp was among a number of people who realized that as memory sizes got larger, having them associated with a single node made less and less sense from a computational perspective. One can imagine a "node" which lives in a 64 bit address space, has say 16GB of "L3" cache shared among anywhere from 12 to 12,000 instruction execution or 'compute' elements. More general purpose compute would look more like a processor of today, more specialized compute looks like a tensor element or a GPU element. "RAM" or generally accessible memory would reside in its own unit and could be 'volatile' or 'involatile' (backed by some form of storage). With attributes like access time, volatility, redundancy, etc. That sort of architecture starts to look more like a super computer (albeit a non-uniform one) than your typical computer. With nodes "booting" by going through a self-test and then mapping their memory from the fabric and restarting from where they left off. I thought I might get a chance to build that system at Google but realized that for the first time in my career that I was aware of I was prevented by my lack of a PhD. And what was worse, given the politics at the time, no one with a PhD would sign on as a 'figurehead' (I reached out to a couple and we had some really great conversations around this) because of the way in which Google was evaluating performance at the time. (think grad school ++ but where only one author seems to get all the credit). Now that I've had some free time I've built some of it in simulation to explore the trade-offs and the ability to predict performance based on the various channels. Slow going though.
- SisypheanLife 7y agoThese types of multi-tier compute systems/MITM processors are now being implemented. Microsoft publicly discusses their efforts here: https://www.microsoft.com/en-us/research/project/project-catapult/ https://www.microsoft.com/en-us/research/project/project-cat...
- berbec 7y ago400Gbps switches exist today [1]. They cost the annual budget of the NFL, but they do exist. 10Tbps "just" requires an increase of 25. Now getting this in consumer-level hardware... 1: https://www.router-switch.com/n9k-c9316d-gx.html https://www.router-switch.com/n9k-c9316d-gx.html
- zamadatix 7y agoIn switching, not NICs (yet).
- berbec 7y agoThey are only available switch<->switch? There are no nics? What's infiband up to now? Looks like it's up to 1.2Tbps https://en.m.wikipedia.org/wiki/InfiniBand https://en.m.wikipedia.org/wiki/InfiniBand
- GhettoMaestro 7y ago400Gbps interface, or total capacity? I assume you mean per-interface. Yeah they are crazy expensive currently.
- berbec 7y agoYeah I think it has 6.4Tb switching capacity.
- 7y ago
- virtuallynathan 7y agoThat’s well and good, but the limitation here is PCIe or other Bus bandwidth, and memory bandwidth. PCIe 5.0 gets us to 1Tbps per x16 slot. The top-of-the-line CPUs Max our at 6.4Tbps of memory bandwidth (Power10 when it is released). This would require a large ASIC or FPGA to be useful, loaded with high-flocked HBM2.
- AdrianB1 7y agoThey are looking at the entire system and they are willing to challenge (improve or replace) everything, including buses like PCIe. This is also an effort not meant for regular servers, but very high end. We already have 128 PCIe lanes or more in a single server, that means 8 Tbps; not in a single connection or slot, but aggregated. Also dual socket CPU's are very popular and more than that is still accessible; multiply the memory bandwidth by the number of sockets. Think about total throughput, not per bus, per socket, per NIC, etc.
- wmf 7y agoThey are looking at the entire system and they are willing to challenge (improve or replace) everything That's also what I thought; just put the NIC inside the processor and connect it to the internal fabric. (This still leaves plenty of software challenges.) But then DARPA says "The hardware solutions must attach to servers via one or more industry-standard interface points, such as I/O buses, multiprocessor interconnection networks, and memory slots, to support the rapid transition of FastNICs technology." Even if your "NIC" uses all 128 lanes of PCIe 5 that's only 4+4 Tbps. If you get rid of serdes and use something like IFOP that's ~600 Gbps per port then you'd still need something like 16 of those links.
- chongli 7y agoYou would also need to replace the operating system and all the software. The traditional operating system driver stack does way too much copying for any of this to work. Perhaps a better model would involve some way of time-sharing or multiplexing direct access to the hardware by user applications. I'm not an operating system expert so take my comments with a grain of salt.
- rsmets 7y agoNapatech also has some state of the art programable NIC product offerings as well. https://www.napatech.com/products/napatech-smartnics/ https://www.napatech.com/products/napatech-smartnics/ I know a number of cloud providers are customers of theirs. Nice to see practical use cases for FPGAs.
- shaklee3 7y agoPlease don't post advertisements disguised as a real post. If you work there, it should be disclosed.
- peter_d_sherman 7y agoExcerpt: "Enabling this significant performance gain will require a rework of the entire network stack – from the application layer through the system software layer, down to the hardware." Let's start with TCP. TCP is an abstraction layer over IP, creating the abstraction of reliable, ordered, guaranteed data delivery -- over an unreliable network, which does not guarantee that any given packet will arrive, much less which order it will arrive in, if it even does arrive. That abstraction is called a "Connection". But connections come with a high maintenance overhead. The network stack must check periodically if a connection is still active, must periodically send out keep-alive packets on connections that are, must allocate and deallocate memory for each connection's buffer, must order packets when they come in in each connection's buffer, and must do I/O to whatever subsystem(s) communicate with the network stack, etc. Speeding up that infrastructure would mean rethinking all of that... Here are some of the most fundamental questions to that thought process: 1) What new set of criteria will constitute a Connection -- on IP's packet-based, connectionless nature? 2) Who will be permitted to connect? 3) How will you authenticate #2? 4) Where (outside NIC, inside NIC, computer network stack, ?) will you perform the algorithmic tasks necessary for #2, #3? 5) What are you willing to compromise for faster speed? E.g, you could use raw datagrams, but not only are they not guaranteed to arrive, but their source can be spoofed... how do you know that a datagram is from the IP address it claims to be without further verification, without the Connection (and further verification of the Connection level, like SSL/Certificates,etc.)? In other words, rethinking TCP/IP brings with it no shortage of potential problems and security concerns... It might be faster to simply make the NIC cards faster, as the article talks about... Or have the Clients or Server software be more selective about what data they send or receive... or what they accept as Connections, from whom, and why... That is, maybe it's not a speed problem... maybe it's a selectivity problem... Still, I'm all for faster hardware if DARPA can realize it. :-)
- blibble 7y agoas a note TCP doesn't periodically check if a connection is still alive, or send out keep-alives (unless you explictly turn it on)
- chongli 7y agoIt sounds like they don't go far enough. Perhaps the way to go is to redesign the CPU as well. CPUs could consume raw datagrams directly off the wire. To authenticate them we could use HMAC [1], which would presumably be built right into the CPU cache. [1] https://en.wikipedia.org/wiki/HMAC https://en.wikipedia.org/wiki/HMAC
- gumby 7y agoI’ve long had the fantasy that machines that communicate a lot amongst themselves could develop an optimized “argot” just as humans do. For example we’re on the local lan and could dispense with the fragmentation infrastructure. Or there are only six of us so could we just use three-bit “nicknames” instead of 48 bits of MAC. Etc. Perhaps some nice adaptive machine learning could drive this.
- chrisweekly 7y agoSlight tangent, but FYI for an accessible, authoritative, useful "book" on networking from the perspective of a modern web/app developer, I highly recommend "High-Performance Browser Networking". https://hpbn.co/ https://hpbn.co/
- chrisweekly 7y agoOP is about reinventing the network stack. My comment was a pointer to a great reference about the current network stack. Someone decided that was downvote-worthy (why?)... so its visibility is reduced to 0. I don't care about my karma points per se (may well lose more for this one), but I feel compelled to complain about having been muted/censored after posting a concise, relevant comment. :/
- diffserv 7y agoI am genuinely curious to see what types of "compute-intensive" applications fit the bill here. Outside of storage workloads (syncing data, etc.), why would you need a 100x improvement in data transfer rates between the machines? (We have TPUs and their specialized network architecture for ML-like workloads ...) Physical distance between the machines in a DC prevents RAM style "shared-memory" architectures, at least ones that aim to have 30~60ns access times (10-20 meters). Unless there are new paradigms for computation in a distributed setting, I don't see the benefit for this ... Also, what are the fundamental limitations/research problems of todays hardware that prevent us from building a 400G NIC? I cannot think of anything outside of PCI-e bus getting saturated. We already have 400G ports on switches ...
- scottlocklin 7y ago>I am genuinely curious to see what types of "compute-intensive" applications fit the bill here. Well, for example, physical simulations using finite element or boundary value approaches. Pretty much anything you'd use with MPI or do on a supercomputer is going to run better on a machine with a nice network stack like this. Even large scale storage (think backtesting on petabytes of options data) that uses a map-reduce paradigm and is properly sharded for the data access paths and aggregates would benefit from something like this.
- diffserv 7y agoDo you have numbers or papers that support your argument? That these applications are bottlenecked by the network? There is a 2015 paper [1] that argues that improving network performance isn't gonna help MapReduce/data analytics type of jobs much: " .. none of the workloads we studied could improve by a median of more than 2% as a result of optimizing network performance. We did not use especially high bandwidth machines in getting this result: the m2.4xlarge instances we used have a 1Gbps network link." Granted things might have changed by now, but I am curious to see how and by how much? [1]: https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-ousterhout.pdf https://www.usenix.org/system/files/conference/nsdi15/nsdi15...
- nwmcsween 7y agoBefore 100gbe or anything remotely close on consumer systems most (all?) in kernel networking stacks are going to need to be scrapped. By then you might as well redesign the OS to something less painful to use.