8 ms·
Tesla’s TTPoE at Hot Chips 2024: Replacing TCP for Low Latency Applications
- fragmede 2y agoInteresting! No mention of UDP, or the application being run, or the GPU/TPUs on the nodes, so it'll have to be a mystery as to how much bang for their buck they're getting with this particular bit of work. What's disappointing is that it's impossible to do a new protocol on the Internet because of all the middleware boxes that drop packets that aren't IMCP or TCP or UDP.
- KaiserPro 2y agoUDP means that upper layers have to figure out if the data is shit or not. to do that in hardware limits your data types, flows and developments. Its far more simple to have a "reliable" data transport system, than it is to deal with a lossy protocol in hardware.
- 7e 2y ago100Gbps Ethernet cards? The world has moved way past that for training. Their accelerator stack must be really slow if this is good enough for them.
- teruakohatu 2y agoIt doesn't say they can't aggregate links. I wouldn't say the world has moved way past that yet, but Tesla probably doesn't want to be dependant, like everyone else but Apple, on Nvidia (InfiniBand).
- krasin 2y ago> like everyone else but Apple, on Nvidia (InfiniBand) Google also does not depend on NVIDIA, thx to TPUs. Rents NVIDIA GPUs to external customers - sure, it's a nice side business, but internally TPUs are king and there's no dependency on NVIDIA for that.
- morepork 2y agoLikewise Microsoft and Meta have developed in house AI chips, but I don't know what fraction of their AI workloads run on them.
- theincredulousk 2y agoNot a meaningful amount :). Their “AI chips” are, for now, marketing.
- SuchAnonMuchWow 2y agoFrom what I know, Meta AI chips are used in production today, but are made for their recommendations tasks which is a very different IA than GPTs and LLMs for which they still rely on GPUs.
- theincredulousk 2y agoTPUs are essentially garbage compared to NVIDIA hardware. TPUs are king of nothing, but a primary ingredient in Kool-Aid
- foobiekr 2y agoThis is not true at all.
- krasin 2y agoIt's very hard to respond to a comment like that, since there's no specifics, just a plain disagreement. On my side, I would like to point that the today HN thread ([1]) that discusses a paper GameNGen ([2]) that runs Doom with diffusion models was trained on TPUs. I don't see a dependency on NVIDIA there. If there's a more specific rebuttal to my original statement, please, don't hesitate to state it. 1. https://news.ycombinator.com/item?id=41375548 https://news.ycombinator.com/item?id=41375548 2. https://gamengen.github.io/ https://gamengen.github.io/ 3. https://arxiv.org/abs/2408.14837 https://arxiv.org/abs/2408.14837
- 2y ago
- stonogo 2y agoYou can get 400gbit ethernet from half a dozen vendors.
- cout 2y agoThis kind of computing must be a different kind of world than the one I work in. 80 microseconds of latency seems high to me when infiniband can do single digit latency with unreliable datagrams, which turn out to be mostly reliable due to the credit system.
- exabrial 2y agowhats the credit system?
- xtacy 2y agoOP is referring to "Credit based flow control", which is a way to ensure a sender does not overwhelm a receiver with more data than it can handle. Usually, this is line-rate, but if the other side is slow for whatever reason (say the consumer is not draining data), you wouldn't want the sender to continue sending data. If you also have N hosts sending data to 1 host, you would need some way of distributing the bandwidth among the N hosts. That's another scenario where the credit system comes. Think of it as an admission control for packets so as to guarantee that no packets are lost. Congestion control is a looser form of admission control that tolerates lossy networks, by retransmitting packets should they be lost.
- yumraj 2y ago> infiniband Or I guess even RoCE
- 2y ago
- xtacy 2y agoIt's also a bit odd that they do not implement congestion control. Congestion control is fundamental unless you only have point-to-point data transfers, which is rarely the case. All-reduce operation during training requires N to 1 data transfer. In these scenarios the sender needs to control its data transfer rates so as to not overwhelm not just the receiver, but also the network... if this is not done, it will cause congestion collapse (https://en.wikipedia.org/wiki/Network_congestion#:~:text=service%20attack.-,Congestive%20collapse,-%5Bedit%5D https://en.wikipedia.org/wiki/Network_congestion#:~:text=ser...).
- jcims 2y agoI probably shouldn't be commenting because I don't have any experience at this level, but given it's a closed system where they control supply and demand it seems they could manage away most congestion issues with scheduling/orchestration. They still have a primitive flow control in the protocol and it seems like you could create something akin to a virtual sliding window just by instrumenting the retransmits. But now I am curious with the distribution of observed window sizes is in the wild. Edit: I'd bet the simpler protocol is more vulnerable to various spoofing attacks though. Edit2: Lol I hope the frame IDs are for illustrative purposes only - https://chipsandcheese.com/2024/08/27/teslas-ttpoe-at-hot-chips-2024-replacing-tcp-for-low-latency-applications/hc2024_tesla_ttp_handshakes/ https://chipsandcheese.com/2024/08/27/teslas-ttpoe-at-hot-ch...
- xtacy 2y agoIn principle, with perfect knowledge of flows at any given instant, you can assign credits/rate-of-transmission for each flow to prevent congestion. But, in practice this is somewhat nuanced to build, and there are various tradeoffs to consider: what happens if the flows are so short that coordinating with a centralised scheduler incurs a latency overhead that is comparable to the flow duration? There's been research to demonstrate that one can strike a sweet spot, but I don't think it's practical nor has it been really deployed in the wild. And of course, this scheduler has to be made reliable as it's a single point of failure. Such ideas are, however, worth revisiting when the workload is unique enough (in this case, it is), and the performance gains are so big enough...
- choilive 2y agoThis is all technically impressive but was it all technically necessary? Was infiniband really just not good enough? All this R&D for a custom protocol and custom NICs seems to just be a massive flex of Tesla's engineering muscle.
- karlgkk 2y agoIt wasn't technically necessary, but neither was RISC-V. It's a matter of licensing independence.
- marcinzm 2y ago> It's a matter of licensing independence. And supply chain independence. I've heard that some GPU clouds are delayed because their Infiniband hardware was delayed due to the Israel–Hamas war. Optimally you probably want to avoid critical hardware that's being manufactured in a high risk of disruption zone.
- 2rsf 2y agoIt's not only about being "good enough" but also about reliability and maintenance, new protocols and hardware may take time to mature while other solutions are already there. Ah, wait, Tesla doesn't care about those kind of things too much...
- logicchains 2y ago> Was infiniband really just not good enough Infiniband suppliers charge crazy prices due to having little competition. It might actually be cheaper for them to design their own than to pay the Infiniband tax.
- speransky 2y agoTuned RoCE with udp is really low latency and no need to implement extra layer of silicon. May be there are more motivation then described in article
- efitz 2y agoYeah I never FIN my connections eithRST
- jeroenvlek 2y agoSeems like Tesla could really benefit from this about to be released optimizer that reduces intra-GPU communication [0]. [0] https://github.com/NousResearch/DisTrO/blob/main/A_Preliminary_Report_on_DisTrO.pdf https://github.com/NousResearch/DisTrO/blob/main/A_Prelimina...
- sgt 2y agoIsn't this what Dolphin Interconnect have been doing for a couple of decades? https://www.dolphinics.com/ https://www.dolphinics.com/
- SilverBirch 2y agoThis screams "Not Invented Here" syndrome. Massive yikes at the digram showing TCP in software in the OSI model. There have been hardware accelerated TCP stacks for decades. They called TCP Offload Engines, they work great, have done for ages. Why are you building one and giving it a new name? Seems like a pretty enormous amount of work and you would've gotten 90+% of the gains by just implementing a standard TOE. I guess the only good reason I can think to do this yourself is that they'd left it so late to get to this that all the companies that were good at this got bought (Solareflare, Mellanox etc).
- MisterTea 2y agoI assume they ignore other technologies and research because new shiny things gives them visibility and therefor promotions.
- lnsru 2y agoI have built dozen of different FPGA based cameras in the past. There is GigE vision protocol: https://en.m.wikipedia.org/wiki/GigE_Vision https://en.m.wikipedia.org/wiki/GigE_Vision TCP is used for “normal” connection and UDP for low latency video data. Such system could be used for other low latency applications as well.
- MisterTea 2y agoI do industrial controls so very familiar. IP is a lot of overhead that doesn't really do anything for the user in a tightly defined automation network local to a machine. EtherCAT goes a step lower and drops IP in favor of just sending Ethernet frames of type 0x88A4. It uses a unique ring topology. It does not use traditional switching or repeaters with the IO devices containing a special controller called the ESC, the EtherCAT Subordinate Controller. The master only needs a standard Ethernet controller. You can get cycle times in the 10's of microseconds allowing for up to 50khz update rates on IO devices. This allows you to do do high performance servo motor control where you close the current loop in the master CPU over 100mbit Ethernet and easily reach 10+ kHz update rates. With FPGAs using commodity SFP or Ethernet PHY's you can certainly build stuff that runs circles around traditional Ethernet and associated overhead from protocols like IP.
- throwaway984393 2y ago[dead]
- darby_nine 2y agoThe "Tesla" in the article appears to refer to the car manufacturer, for those as confused as I.
- joezydeco 2y agoAll this to...train a model? Having worked with automakers that agonize over pennies on a component how does Tesla amortize this much expense, if they even achieve full self-driving at all?
- itishappy 2y agoThe $1 billion spent on Dojo amortized over 2 million cars sold per year is $500 per car. I'm betting it's viable for more than a year.
- bee_rider 2y agoMaybe the point is to just continue to be able to point to FSD as some figure thing that you are working on. That should keep investors happy! Once they’ve built this cluster, maybe they can make money renting out compute or something.
- jcims 2y agoIt's a plant not a product.
- throw0101d 2y ago> Having worked with automakers that agonize over pennies on a component how does Tesla amortize this much expense, if they even achieve full self-driving at all? Or Elon is using the resources of one company to do work for another company (?): * https://en.wikipedia.org/wiki/XAI_(company) https://en.wikipedia.org/wiki/XAI_(company) * https://electrek.co/2024/04/03/elon-musk-xai-poaches-engineer-tesla-clear-conflict-of-interest/ https://electrek.co/2024/04/03/elon-musk-xai-poaches-enginee...
- MisterTea 2y agoCongrats. You just reinvented the wheel: http://doc.cat-v.org/plan_9/4th_edition/papers/il/ http://doc.cat-v.org/plan_9/4th_edition/papers/il/
- chronicileiee 2y agoMeanwhile high frequency traders work 1-2 orders of manicure faster in the tens to hundreds of nanoseconds.
- gruturo 2y agoIf they are so concerned with low latency, how come they are wasting an entire roundtrip (the "OPEN + OPEN/ACK") before sending any data? I mean in TCP it's not allowed (Even though, super-theoretically, it's not completely forbidden) to carry a payload in the initial TCP SYN. If you're so latency-obsessed to create your own protocol, that's the first thing I'd address.
- newsclues 2y agoDo you need low latency right away, or just after the car has started or entered a sport/race mode? If you don’t need it all the time, why bother?
- iamleppert 2y agoThere are already high speed, low latency video interfaces that have been around for ages. MIPI and HDMI. There are ICs you can buy off the shelf for electronic routing and switching of these interfaces.