21 ms·
Google opens Falcon, a reliable low-latency hardware transport, to the ecosystem
- up2isomorphism 3y agoWhen you have enough scale you can claim a certain particular way of doing things are better than the others, which in most cases is just one way of doing things. This is what we see here.
- DoingIsLearning 3y agoSame thing goes for their single monolith repository philosophy.
- deely3 3y agoCould you elaborate on this please?
- DoingIsLearning 3y agoThere is a fairly amount of technical writing describing Google's use of a single repository across a lot of enginering teams. This works for them at their scale with quite a lot of internal tooling specifically to make it work. However many people defend it as _the_ solution whereas it is a way with trade offs like most solutions.
- IshKebab 3y agoMicrosoft, Facebook and Twitter all use a monorepo too. It's not just Google. Granted, Facebook have written their own VCS and Microsoft heavily modified Git to make it usable with monorepos (but only on Windows). Unfortunately stock Git is bad at multirepos and monorepos. When you have "hundreds of people working full time on a project" scale, stock Git doesn't have a good answer.
- rstuart4133 3y agoI normally like Google blog announcements, as they are usually heavy on technical details. But not this one. Quoting, the meat of it is: > Fine-grained hardware-assisted round-trip time (RTT) measurements with flexible, per-flow hardware-enforced traffic shaping, and fast and accurate packet retransmissions, are combined with multipath-capable and PSP-encrypted Falcon connections ... flexible ordering semantics and graceful error handling ... hardware and software are co-designed to work together to help achieve the desired attributes of high message rate, low latency, and high bandwidth So like QUIC, but designed for low latency. Maybe. There is no indication of how they achieve it if it is, nor is there a link to further details. The bulk of the article is literally name dropping. Protocol names, FAANG company names, standards organisation names. It reads like C-suite bait. "Come join us boys - all the big guys already have. So it's a sure winner."
- KingLancelot 3y agoReplace “The Ecosystem” with “The Jungle”. Edit: It’s yet another meta protocol built on top of TCP/UDP.
- junon 3y agoI want to see more layer 4 protocols, not stuff built on top of them, personally. That's where things get interesting.
- gymbeaux 3y agoHonestly I think there’s a lack of talent for that, not only to develop the protocol, but to support it. How many of us really understand OSI?
- luma 3y agoOSI is just a model, there’s not all that much to understand there. It’s simply a way to categorize network technologies using a common language.
- cjensen 3y agoOSI is an entire networking stack designed by committee that died from disuse [1]. The only thing we now remember of it are the functional layers within the protocol. For example, people sometimes refer to TCP as a "Layer 4" protocol even though (a) TCP predates the invention of Layer 4 and (b) TCP is a square peg that does not exactly fit into the round hole that is Layer 4. [1] https://en.wikipedia.org/wiki/OSI_model https://en.wikipedia.org/wiki/OSI_model
- AlotOfReading 3y agoI wish we could forget the other remnants of OSI like x.509, ASN.1, and LDAP. Those were the things good enough to be used for real systems, and I'd still rather crawl over broken glass than implement any of them. Imagine how bad the rest of it was.
- KRAKRISMOTT 3y agoInfinilink?
- jbotdev 3y agoIt sounds like this builds on top of Ethernet to provide a higher performance alternative to UDP/TCP, with some sort of hardware acceleration. I may be in over my head since I’m not an HPC/datacenter expert, but not sure I understand how you’d use this on the software side. Maybe someone is aware of specific examples? (beyond the vague “HPC/AI”) edit: as another comment mentioned, the diagram shows it’s on top of UDP/IP, so it’s mostly an alternative to TCP/IP
- wmf 3y agoFrom a software perspective you probably wouldn't see Falcon at all. You'd use, say, the RDMA verbs API and under the hood it's accelerated by Falcon.
- jiggawatts 3y agoI guarantee that there will eventually be a vaguely similar (but different!) stack published by each of: NetFlix, Microsoft, Amazon, and Apple. Just kidding, Apple won't publish anything. The IT ecosystem has fragmented into mutually incompatible cliques. You are either in the Google ecosystem, the Amazon ecosystem, or some other one, but there are no more truly open and industry-wide standards. Look at WebAuthN: it enables a mobile device from "any" vendor to sign on to web pages without a password. Great! Can I transfer secrets from an Apple iPhone to a Google Android phone? Yes? No? Hello? Anyone there? I just got a new camera. It can take HDR still images, which look astonishingly good. Can I send that to an Apple device? Sure! Can I send it to a Google device? Err... not without transcoding it first... on a Microsoft Windows box. Can I send it to a mailing list of people with mixed-vendor devices? Ha-ha... no. This is the best argument I've seen for splitting up the FAANGs + Microsoft + NVIDIA. Once they get to this behemoth trillion-dollar scale, they become nations onto themselves and no longer need to cooperate, no longer need to use any open standards at all, and can start dictating and pushing third parties around. Another random example is HTTP/3, which is basically the "What's best for Google" protocol. Or gRPC, which is "What Google needs in their data centre". And now Falcon, which is "The transport Google needs for their workloads". Does it work for anyone else? I don't know, but it's a certainty that Google doesn't care and never will, because they don't need to.
- Accujack 3y ago>The IT ecosystem has fragmented into mutually incompatible cliques. You are either in the Google ecosystem, the Amazon ecosystem, or some other one, but there are no more truly open and industry-wide standards. This is one excellent example of the reason that increased/renewed anti-trust actions by the FTC are necessary.
- tgma 3y agoThis is exaggerated to the point that I consider it fiction. BTW Google doesn't substantially use gRPC within their datacenters. The industry has always been this way. Back in the day there were many many processor ISAs that are now consolidated. There was been many networking standards that consolidated (IPX/SPX anyone?) New things often diverge because of new requirements not out of spite. There is a push and pull between standardization and innovation. Doesn't make it particularly unhealthy unless you can point to specific metrics and compare trends throughout the long arc of time.
- docandrew 3y agoI was confused by the reference to “lossy” networks in this page. Does this have a different meaning in this context than something like lossy compression where data is actually discarded?
- wmf 3y agoIt means packets may be dropped, usually due to congestion.
- drewtato 3y agoI think it means error correction would be handled by a higher level.
- rincebrain 3y agoEthernet, unlike, say, Infiniband, doesn't promise things will get where you sent them just because it didn't error initially, so other protocols handle this at higher levels to notice the failure cases.* For an example of what this means, try setting your MTU above the limit, and watch the raw traffic. * - it's been years since I cared about the formal definition, my apologies if I got it wrong.
- networkchad 3y agoYou are correct in general. Lossy = no guarantee mechanism / state-based acceptance; it’s handled by a higher level proto (tcp).
- spoonjim 3y agoAll networks are “lossy” because any cable can be cut, etc. A “lossy” protocol is one that doesn’t attempt to compensate for that. In most cases, but not all, that means a higher level protocol will need to ensure that every bit of data has made it through. (An example protocol that might not care is one for watching broadcast TV on the Internet… if you miss a few seconds it’s not a big deal).
- mintplant 3y agoOh, I was hoping this would be something built more directly over Ethernet, rather than on top of UDP/IP (if I'm understanding the layer diagram correctly). I've been working with Ethernet devices a lot lately, using the network as a communication bus, essentially. I find that there's a lot of complexity that we simply don't need: ARP, DHCP, DNS... So many points of failure. We know all the devices on our LAN and their unique MAC addresses, and could do everything we need to addressing-wise at Layer 2. But everything's built on Layer 3 and up, so we're effectively working backward to map devices to IP addresses and vice versa. It's unsatisfying.
- wmf 3y ago1. Your forwarding table would have to be larger because Ethernet uses exact match instead of longest prefix match. For example, you might be limited to 128K servers total while Google has millions. 2. The Ethernet header has less entropy for ECMP than a UDP/IP header. Maybe you could add entropy somewhere but ASICs may not support it. 3. You're breaking compatibility with... everything. Maybe Google could afford this but no one else could.
- Dylan16807 3y ago1. In a single data center scale computing system, though? 2. Use some bytes outside of the header? 3. I get the impression this needs hardware to really use well anyway.
- smokefoot 3y agoI don’t understand networking all that well. Is it interesting that the telcos and non-tech companies are moving away from specialized hardware toward software defined networks while the hyperscalers are using hardware acceleration?
- wmf 3y ago"Software defined networks" doesn't mean anything. If you want to understand networking you have to strike it from your vocabulary. It's possible for different vendors to be on different points on the wheel of reincarnation at the same time. https://www.computerhope.com/jargon/w/wor.htm https://www.computerhope.com/jargon/w/wor.htm
- smokefoot 3y agoHa, I get it. That is a helpful model.
- eightysixfour 3y agoThese things are partially related - telcos running things in the cloud, in software, is actually running on top of these hardware innovations, it is just abstracted from them.
- dilyevsky 3y agoNobody wants to depend on hardware development cycle hence “fuck it, ill do it in software”
- duped 3y agoI don't think it's that hard to rely on hardware development, it's more of a problem of rolling out a fleet of new hardware. It's just not realistic to take all the switches, routers, and other garbage you've got in between points in the network off the rack/ceiling/wall/pole because the hardware can't support some protocol. Good evidence for this is the rollout of fiber, which has been happening neighborhood by neighborhood and house by house for a decade.
- DrReachAround 3y agoI'm confused by this because we've been using Falcon at work for over a year now, perhaps longer, as I just started a year ago. What are they making available that wasn't already?
- sargun 3y agoWhat were you using Falcon in, and for what?
- adsharma 3y agoCan anyone explain the difference between Falcon and RoCE v2?
- wmf 3y agoIt sounds like Falcon adds encryption, packet spraying, and better congestion control.
- packetslave 3y agofewer Pause Frames of Doom?
- deleted 3y ago[deleted]
- StillBored 3y agoHmmm, so much of this looks like an attempt to solve the problems that were solved with fibre channel a couple decades back. Which I guess is standard NIH, with the advantage of not having to pay the FC consortium 95% HW margins. But still, you would think that some of those lessons could be learned before replacing it. AKA FC routes IP as one of its many protocols on top of the lower levels providing far more service guarantees than one normally gets with ethernet. Much of the QoS/latency/etc metrics were designed into FC from the beginning as a use on storage area networks (SANs). It just never took off as a IP transport because it cost 10x as much as ehernet, including a decade ago when these same groups tried to dump it on an ethernet MAC only to discover that it requires special switches which were $$$$ because "enterprise markup" defeating the whole point of cheap ethernet phy's. See FCoE.. And yet today, there is NVMEoF on FC, which is what one runs when its important that someone scp'ing a file on your network doesn't cause your database queries to slow down. What I don't get is why OCP doesn't just actually build some of these adapters/etc with a "we won't be greedy" take and sell them not only to the hyperscalers but on the open market. That way someone could actually build say, a FC adapter that has a price similar to an ethernet adapter.
- wmf 3y agoMaybe a better example would be Infiniband which is a simple and efficient protocol... but it's basically owned by Nvidia. For whatever reason Broadcom won't make Infiniband ASICs and Google doesn't want to be locked in to Nvidia so they have to use Ethernet.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- Double_a_92 3y agoIs it just me getting older / less smart, or did articles about products really start to sound like a jumbled mess or buzzwords lately? What is "Hardware transport", what is "the ecosystem"? And then there is dozens of random products and technologies that I've never heard of... This sounds more like a humble brag, than an article trying to inform people about technologies that might actually be useful to them.
- seanp2k2 3y agoThis is how you get promoted at Google.
- nottorp 3y agoBtw, now that someone got promoted, how dependent is this on Google's continued support? As we all know, interest at Google will now wane since you can't get promotions out of this any more.
- stonogo 3y ago"Hardware transport" is kind of a misnomer, because this is a networking protocol. It just happens to be a networking protocol that requires hardware acceleration on the NIC. We already had that in things like Infiniband and Omnipath, but Nvidia bought one and Intel rugpulled the other. Meanwhile ethernet has been approaching parity in terms of throughput, but TCP introduces unpleasant latency, so this is Google's NIH-flavored DIY on the topic. It's something of a rite of passage for a company to convince itself that building this sort of thing in-house is necessary, and that it will revolutionize high-performance computing in all the ways that previous, nearly-identical projects have not. "The ecosystem" is The Open Compute Project [1], a trade association which mostly puts together quasi-standards that provide targets so computer manufacturers can produce bleeding-edge gear with some hope that it will be interoperable. An example OCP production is the newer 21" racks that are starting to appear in datacenters. 1 - https://www.opencompute.org/ https://www.opencompute.org/
- jacquesm 3y ago
- axegon_ 3y agoSigh another project called "falcon".
- mgaunard 3y agoTo this day I still haven't seen a more sensible API for low-latency Ethernet than Exablaze (was the market leader in low-latency trading, then got bought by Cisco). The only thing blocking these from becoming standard is that it means userland has direct control of hardware.
- dur-randir 3y agoAnd closes in ..?
- user3939382 3y ago[flagged]
- bluGill 3y agoGoogle has lost trust with so many of the things they have released in the past becoming unsupported and obsolete.
- Grazester 3y agoIf this is an open standard then anyone who uses it and wishes to support it can do so no, or am I mistaken?
- bluGill 3y agoMaybe, but often in practice if the amount of supported needed gets high. It needs some very involved people to keep support going, otherwise it will bit rot and fail to function after a time.