11 ms·

It's TCP vs. RPC All over Again

154 points by zdw 4y ago

ktpsns 4y ago
What are the best recommendations for modern day low latency RPC? Google protocol buffers? Or some message queue such as ZeroMQ?
- tonymillion 4y ago
  You’re thinking too high level, this is talking about replacing TCP or UDP with a new transport protocol designed specifically with RPC in mind. Instead of opening a TCP (or UDP) socket and writing data (or data grams) to it you’d open a (for want of a better term) RPCP socket and write data that represents the procedure call and parameters to it. I assume the protocol would have assurances of some kind of reliability (unlike UDP) and procedure call data boundaries (unlike TCP) built in at the lowest levels along with request/response sequences to allow out of order completions.
- dist1ll 4y ago
  Check out eRPC.io It leverages DPDK and is perfect for low-latency RPC in lossless DCs
  - jiggawatts 4y ago
    It would be nice to have something like that, but have libraries for most common languages.
  - jeffbee 4y ago
    Where would I find one of these lossless fabrics?
- drbruced 4y ago
  Part of what we're arguing in this article is that there is no good answer today because of the assumption that TCP was good enough. Homa is challenging that assumption (as many others have tried before). My view is that if you want something better than TCP for RPC today, QUIC is the best thing available, as discussed here: https://systemsapproach.substack.com/p/quic-is-not-a-tcp-replacement https://systemsapproach.substack.com/p/quic-is-not-a-tcp-rep... What you run over that for your RPC is a matter of taste - gRPC is certainly popular, and can be run over QUIC.
- omginternets 4y ago
  I'm a huge fan of capnproto.org
- en4bz 4y ago
  Flatbuffers, Cap'n Proto, or SBE over RDMA.
iambvk 4y ago
Good to learn a bit of history about TCP. It always felt weird developing RPC layers on top of TCP cause requests and responses end up tied to the underlying socket -- which doesn't need to be the case ever.
- AceJohnny2 4y ago
  How else would the kernel know to which application it should route a response?
  - cjsplat 4y ago
    Destinations can be described in the packet headers. That is how the current systems work.
  - iambvk 4y ago
    Destination port number should be sufficient. For example, imagine if there are two sockets established between server and client processes (due to multiple ips or roaming, time lags, etc.), then in theory, request received from one socket could be responded through another socket. From application pov it doesn't need to care about the socket. We can do this over TCP, but connection-oriented nature of TCP makes this weird.
deleted 4y ago
[deleted]
irq-1 4y ago
Really interesting article. Wish there was a discussion of performance for various RPCs from the past.
- cryptonector 4y ago
  There's so many... Apollo Domain's. ONC RPC. DCE RPC (and MSFT RPC, which derives from DCE RPC). SOAP. LustreRPC. And who knows how many others. The biggest problem with RPC in general is that historically it was synchronous because that's what was easy to implement in the 80s. Fix that and an RPC sucks only as much as the encoding system it uses. But RPC == remote procedure call, and that causes people to instantly think "synchronous", and that is a kiss of death.
  - cjsplat 4y ago
    Well of course once you say "procedure call", people think synchronous. For the sender, asynchronous RPC is just a convenient marshalling and return interface for the sender. It is pretty clunky for the A-RPC caller to create an illusion in the code of the RPC being "just like" a local procedure call. For the called procedure an Async RPC can look just like a local sync procedure call, except for that whole address space thing. I don't think that sync RPC was popular because it was easy to implement. I think it was popular because it was easy to code to. Multi-threaded coding is easy to get wrong, and was poorly understood at the time.
    - cryptonector 4y ago
      > I think it was popular because it was easy to code to. That's not that unlike it being easy to implement. > Multi-threaded coding is easy to get wrong, [...] Yeah, but the better design is async/await and so on, not threading, but that stuff was not very popular in the 80s. The approach to concurrency in the 80s was heavy on context switching because that's what was inherited from the 70s. It feels like the thinking was along the lines of "during the time you're waiting for a response some other process will run, and since we're used to heavy latency, who cares!", but that became less and less tenable. There's a reason all the distributed operating system research of the 80s died except for filesystems -- with filesystems you can consciously avoid hot ping pong by just not accessing shared resources concurrently, while the filesystem itself remains super useful.
cryptonector 4y ago
> the Internet seems to be missing a standard protocol for the request/response paradigm, with repeated attempts to force-fit TCP leading to inevitable mismatches HTTP is that. QUIC is that over UDP. We've discussed Ousterhout's paper about replacing TCP in the data center. Last I looked the answer was "not with what is described in the paper". > I have never understood why the Internet has worked so persistently to adapt TCP in support of request/reply workloads instead of standardizing an RPC transport protocol to complement TCP. > ... > But my original question was to ask why that hasn’t happened. The only answer I can come up with is that judgment often reflects biases. I can explain this. Some of it is bias, as TFA surmises, but there's a lot more to it. First off, there is an Internet RPC protocol, and it's the ONC RPC protocol, the one that NFS uses. ONC RPC does run over TCP and UDP, which, yes, TFA doesn't seem to want. Second point: adding new upper layer protocols (ULPs) like TCP and UDP and SCTP is fraught with middlebox and over-design problems, and you can look at SCTP for details. The third point is that the ability to implement and iterate quickly means that having to wait for kernel support for the new thing is a non-starter. This is the strongest reason why you won't see a new RPC ULP happen quickly and why there has been little interest in one. So between the middlebox issues and iteration latency issues the only way to do a new ULP is as a new binding of the RPC which must also have TCP and UDP bindings. I.e., all request-response protocols we have pretty much have to run on top of TCP or UDP. This fact of Internet life may suck, but that's just how it is. Making it go away will require lots of time. Developers don't have lots of time. Fourth point: running a request response protocol over UDP is quite reasonable, especially in the datacenter. Fifth point: building a new ULP will risk yielding the same over-design issues as SCTP all over again unless it's a new binding of another protocol. I'm beating a dead horse here; let's switch to the "bias" issue. RPC got a really bad rap in the 90s and 00s. The reason is that the RPC frameworks that came out of the 80s, and the more modern versions that followed them in the 90s and 00s, were all [mostly] synchronous. There's no reason for that: it was the easiest way to implement back then. Another aspect of synchrony is that RPC == remote procedure call and comes from an era when it was fashionable to build things like distributed remote shared memory (which is a performance disaster). The idea that RPC is like a function call instantly evokes synchrony even though RPCs are not condemned to be synchronous. That suggestion leads to aversion. Another reason that RPC frameworks got a back rap back then was the encoding systems used. It doesn't matter if the encoding system was trivial like XDR or complex like NDR or DER or PER or whatever -- the tooling for encoding rules from the 80s and 90s was just not remotely universal. As well we had the XML era (e.g., SOAP), which was very buzzword-rich and efficiency-poor. That bad reputation is undeserved, really, but there it is. Nowadays RPC is making a comeback via gRPC and friends, but it shouldn't surprise that they typically run over HTTP. Meanwhile the web folks iterated quickly. HTTP user-agent stacks quickly became async. HTTP APIs iterated from XML to JSON, and either way, with JavaScript and HTML, web apps got a great deal of flexibility and performed great by comparison to RPC apps. Plus the web security model is atrocious mainly because of the most useful thing about HTML: pages can have cross-site links. Cross-site linking is so incredibly useful that it can't be done without. And now you need a fancy user-agent (browsers), so the web folks built one (browsers). HTTP won out over RPC. Why not use HTTP for everything, then? Well, HTTP/1.1 is... not efficient. H2 is much better, but still over TCP. H3/QUIC should be perfectly fine in the datacenter, and for some things on the web (and eventually maybe all things on the web?), and runs over UDP. But HTTP is a request/response protocol. Just like any RPC! Plus ça change, plus ça devient la même chose. My argument: HTTP is the protocol you're looking for. And if ever we need a new ULP and are confident that we can deploy it, then we should design one for HTTP because a) that will satisfy the RPC need, b) it will tend to quash over-design instincts, c) it will have the same APIs/semantics as a protocol we're all already very familiar with, d) having one protocol with bindings to all applicable ULPs will help us get over the kernel support issues. (d) is especially appealing: you get to target one API and you get the best ULP locally supported. We do have a widely-deployed Internet RPC protocol called HTTP, and you get to run it over TCP or UDP, and its semantics are stable when we add new ULP support. That's a pretty good outcome. Though maybe it requires getting over one's own biases :) > That we have since turned HTTP into the Internet’s de facto RPC protocol (and then now realizing that it is suboptimal, are trying to optimize it by collapsing all the layers into the new QUIC protocol), is only a testament to how small a role technical rationale plays in what happens in industry. I object to the characterization of HTTP as suboptimal for RPC. TFA is clearly referring to TCP as being suboptimal for RPC, but not really covering how UDP is suboptimal for RPC. And TFA does not discuss why NFS switched from using UDP to TCP, say. It feels like TFA is just emoting that we ought to have a ULP specifically for pure RPC. That seems like a very biased position. Suppose we built a ULP for RPC though. We could easily expect HTTP to get adapted to run over it. What does HTTP add that an RPC doesn't need? Mostly headers, which the RPC can just not use, and URI components, which the RPC can also set to the smallest possible values, say. There's not that much light between RPC and HTTP. Both are request-response protocols. Accept HTTP as an RPC and you might be happy.
SillyUsername 4y ago
If you change the stack at TCP level... - How many hardware network components would need replacing/upgrading to support a layer not expected to change? - How many network + application software libraries would need to support parsing at packet level instead of byte level? - How many OSI diagrams showing that high level byte and string based protocols run above the transport layer need to be re-written to say that's no longer the case. - How many versions of RPC would need to be supported... just look at the numerous revisions of CORBA/RMI/SOAP(Document or RPC Style!)/REST/GRPC to see how many times you can re-invent (and misinterpret, break standards of) a wheel. - Look how long IPv6 needed buy in and refinement, and then adoption. That was a necessary change with no alternative too! There are many applications that shouldn't need TCP, such as those only doing short network hops or needing real time traffic with expected packet loss. They should use UDP but often don't. TCP is ubiquitous for the very reason that it is the chameleon mentioned in the article - TCP is not ideal, but it is convenient and well understood.
bob1029 4y ago
There are some things that current RPC frameworks simply cannot touch. I've been up to my elbows in TCP lately. Working on a low-latency, select-based socket server for streaming gaming applications. Each instance runs on 1 thread so there is absolutely no context switching delay when servicing player requests. The TCP streams are responsible for moving player inputs and game state as quickly as possible between system elements. There's also an externally facing websocket being serviced inside of these threads. The whole networking chunk of the loop usually completes within few tens of uS. Socket selects hang out for up to 10uS. I considered something like AspNetCore/kestrel, but that's a lot of machinery, GC liability, thread contention, et.al. gRPC also crossed my mind but I found some headache with the tooling.
- Karrot_Kream 4y ago
  Do you mean the `select` call? libuv and friends can offer higher throughput than `select`.
  - User23 4y ago
    It’s been many many years since I’ve done low level multiplexing but I think epoll() is the no longer quite so new hotness? Ah I see libuv is a cross platform lib. I’d probably use that for production code, but it’s still really valuable to work with the low level API enough to understand it.
    - Karrot_Kream 4y ago
      Yeah epoll is what libuv calls on Linux. And agreed I think it's very valuable to try and use epoll or equivalent yourself at least once to understand what's happening.
  - bob1029 4y ago
    Whatever OS primitive this API effectively calls is what I use: https://learn.microsoft.com/en-us/dotnet/api/system.net.sockets.socket.select?view=net-7.0#system-net-sockets-socket-select(system-collections-ilist-system-collections-ilist-system-collections-ilist-system-int32) https://learn.microsoft.com/en-us/dotnet/api/system.net.sock... Edit: This is the source: https://source.dot.net/#System.Net.Sockets/System/Net/Sockets/SocketPal.Unix.cs,a83ac8f2e1e985a6 https://source.dot.net/#System.Net.Sockets/System/Net/Socket...
manv1 4y ago
This paragraph is just wrong: "Another explanation is that the Internet has unnecessarily coupled the transport protocol with the rest of the RPC framework. Conflating the two naturally follows from the purpose-built examples I just gave: SMTP is bundled with MIME; SNMP is bundled with MIB; and HTTP is bundled with HTML. " I think at this point he's suffering from what old-timers used to call recto-cranial inversion.
- codeflo 4y ago
  I think I feel dumber for reading this. (The referenced paper by Ousterhout is quite good though, and doesn’t confuse the terminology. Maybe that one’s what HN should link and discuss.)
  - matjet 4y ago
    To someone only weakly aware in this area, can you explain what is wrong with the statement? From a cursory view, these protocols indeed seem to normally carry the mentioned payloads.
    - lozenge 4y ago
      It's not like HTTP stops working if you choose to send something besides HTML. Most of its features still work and make sense for JSON payloads and the usual building blocks still work (eg HTTPS, load balancers, proxies, status codes, redirects...) There are a few bits that don't make sense eg cookies where the server sends something and the client is expected to remember it but they can just be ignored.
      - codeflo 4y ago
        Exactly. The HTTP standard doesn't really mention HTML, except tangentially in examples. And regarding cookies, there are even people who use cookies in RPC contexts, but this is rather rare, and as you said, entirely optional. Also, as a minor point, even when browsing the web, a minority of the actual requests deliver HTML -- most of them are for various kinds of media referenced by the HTML document.
codeflo 4y ago
I for one think we should finally replace DNS with JPEG. That makes exactly as much sense as this article.
- jbverschoor 4y ago
  jxl so the requests can be lossless, transparent, and multi-channeled
hardwaresofton 4y ago
If you'd like to watch Ousterhoust explain Homa and his vision of replacing TCP: https://www.youtube.com/watch?v=o2HBHckrdQc https://www.youtube.com/watch?v=o2HBHckrdQc Spoiler: Ousterhout is right, and the data shows he is right. Maybe not everyone knows it yet, or it's the kind of "right" that not everyone will ever get to (and we'll settle for "good enough" for much longer), but he is almost certainly right. Not to disparage the poster of this article of course -- I always love a good hot take or contrarian piece, but if you can't come with the same amount of data to support your thesis... Homa is probably dismissable on UX grounds (are people really going to switch to it?), but it's clear that long hard thinking has gone into the case for Homa (and it's implementation).
- deathanatos 4y ago
  I think you're right that he's "right", but the writing is shooting itself in the foot here. I agree that there's a mostly-unexplored middle between TCP & UDP, but you need to not utter things like "HTTP is bundled with HTML". I also think RPC is a poor label for "request/response" protocol (HTTP is request/response, and many of the problems here apply to HTTP too … but it isn't RPC). I'd like more discussion of the contenders in the space: why do they not work? QUIC is barely mentioned beyond "we tried it and nope". SCTP isn't mentioned on the OP at all, but if you click through the Homa stuff a bit, it's written off as "a WAN protocol" — whatever that is? I get that his focus is datacenters, but "WAN protocol" is what I feel most of us are interested in, and I don't immediately see why WAN vs. DC matters¹. What I'm also not getting from the Homa page is … what is the protocol, what are it's packet structures? Is there no documentation? (I've not hit all the third-degree links because I'm not a spider, but nothing seemed relevant or it seemed a little relevant but didn't hold the answers.) The old adage about data structures over algorithms seems to apply: > "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975) ¹I'd also want to know why a good "WAN" protocol that solves the stated problems wouldn't equally apply to the datacenter, modulo needing some adjustments around assumptions made about the network's latency and maybe reliability: a DC network, I'd hope, is going to have much lower latency. But like Raft timers, it seems like a by-use-case adjustable constant.
Animats 4y ago
There have been lots of RPC protocols. Here are some still in use. Transport level: * Sun RPC [1]. QNX still uses this. It can run over UDP or over raw Ethernet. It just transfers an array of bytes and gets an array of bytes back - marshalling is a higher level problem. It handles messages bigger than one packet, and retransmission. It's simple and performance is good, but there is no security. * Stream Control Transmission Protocol. Telcos still use this. It's how Signalling System 7 is sent over IP. Although many implementations exist for both, they've never been popular outside their niches. Marshalling level: * CORBA - you define interfaces in a special language and compile them. Data is not self-describing. * SOAP, from the XML era. * Google Protocol Buffers - another system where you compile definitions. * HTTP/JSON - where we are now. Plus a lot of Microsoft-specific stuff. Either you have a system where both ends have to exactly agree on format, you have a verbose format with unneeded description data in every message, or you have a very complicated negotiation at connection time. This leads to much disagreement. [1] https://en.wikipedia.org/wiki/Sun_RPC https://en.wikipedia.org/wiki/Sun_RPC
- cjsplat 4y ago
  Yeah, but the question at hand is why layer it over TCP. SunRPC over UDP is close, but then you push reliability and congestion management into every application.
  - deathanatos 4y ago
    SCTP, from the parent's list, is a "over IP" protocol, not over TCP. (Though it can be tunneled through UDP, to get around Internet ossification.)
- danans 4y ago
  > CORBA - you define interfaces in a special language and compile them. Data is not self-describing. OMG the Common Object Request Broker Architecture (which coincidentally was created by OMG). I never thought I'd see that acronym again, and all these years later, I still don't understand what it is.
  - 4y ago
cjsplat 4y ago
So the question is why hasn't an optimized RPC implementation emerged for the data center that avoids the headaches that come with layering over TCP. Turns out integrating with the control plane is hard at data center scale. When everything is working, an optimized protocol stack is great and everyone is happy. When congestion happens (fabric, kernel, memory bus, cache footprint, etc), and your optimized app causes a legacy production app to behave strangely, all hell breaks loose. For example, a distributed file system SRE sees some strange tail latency effects and doesn't know that a cluster is shared with a non-TCP app and of course the optimized stack doesn't respond to congestion the way TCP would. Worst case, the SREs notice this just after an update is pushed in their app, and they spend a random amount of time trying to figure out if the update is what changed the behavior. Best case, they know there are non-TCP apps in the area, and they ping the SRE for that stack and say "my app is behaving strange, please hit your big red off button so I can see if that fixes the problem." Someplace in between those options, the SRE is using TCP aware network debugging tools to try to figure out why this day is different from yesterday, or this cluster is different from the one where everything is working fine. Regardless, you get an unhappy SRE. But of course you never get just one unhappy SRE. So you need to generate a lot of value got justify their pain.
nubinetwork 4y ago
I'd be happy if DNS and NTP would just get off their butts and switch to TCP. No, DoH doesn't count.
- cjsplat 4y ago
  Why would NTP be better with retransmisson of stale packets and go-back-N semantics on drops?
  - nubinetwork 4y ago
    UDP can be spoofed for amplification attacks, and since ISPs won't implement reverse path filtering, the only other option is to get rid of UDP.
- seanhunter 4y ago
  Normal DNS will switch to TCP for any large request and has done for quite some time.
  - noselasd 4y ago
    Unless you're using the musl libc - which is in heavy use in conteainerized applications
    - seanhunter 4y ago
      OK well if it does zone transfers those are over TCP only and always have been.
      - deleted 4y ago
        [deleted]
      - nubinetwork 4y ago
        There is more to DNS than just zone transfers...
roncesvalles 4y ago
You could just use UDP instead of TCP and implement some subset of TCP features that you need in-band, like sequence numbers. This is not a new concept. The Facebook Memcache paper [1] in 2013 described using UDP for intra-datacenter requests. [1] https://research.facebook.com/publications/scaling-memcache-at-facebook/ https://research.facebook.com/publications/scaling-memcache-...
- cjsplat 4y ago
  Sure, but what is UDP bringing to the party then? The pseudoheader? A reasonable RPC layer could just layer on IP.
  - wmf 4y ago
    The UDP header will give you proper ECMP hashing.
    - cjsplat 4y ago
      Yup - this is the correct answer
  - roncesvalles 4y ago
    IIRC, UDP just adds port numbers and a checksum to IP. It's about as bare bones as it can get, unless you want to get rid of the checksum too.
    - rezonant 4y ago
      Some implementations do skip checksum (leaving it as zero), for instance you can configure a Linux UDP socket to skip it if you can't afford the time it would take to calculate them.
  - cryptonector 4y ago
    > Sure, but what is UDP bringing to the party then? Interoperability. It's been how many years since SCTP was published? How's that going?
kwantam 4y ago
Is this a good time to bring up T/TCP? https://www.rfc-editor.org/rfc/rfc1644 https://www.rfc-editor.org/rfc/rfc1644 This memo specifies T/TCP, an experimental TCP extension for efficient transaction-oriented (request/response) service. This backwards-compatible extension could fill the gap between the current connection-oriented TCP and the datagram-based UDP. I have no illusions about the likelihood it'll ever see use, but I've always thought T/TCP was cool and fun.
aidenn0 4y ago
Osterhout's response to Pepelnjak's criticism is solid, but he mentions not being familiar with SCTP. Homa is much more fairly compared to SCTP than TCP, since there is a large intersection with the problems they solve. Osterhout seems to think that ACK-based sender congestion avoidance is a significant problem in the datacenter. SCTP uses ACK-based sender congestion avoidance, but lacks the head-of-line blocking that TCP has[1]. Comparing Homa to SCTP would be a good way to test this thesis. 1: Or rather it's optional.
majke 4y ago
I'm a pragmatist and there are couple of points in this article that give me allergic reaction. > You want individual messages instead of a byte-stream? TCP has an option for that. That links to. https://book.systemsapproach.org/e2e/tcp.html#record-boundaries https://book.systemsapproach.org/e2e/tcp.html#record-boundar... In my opinion you can't be serious suggesting usage of URG or PUSH as message boundaries. TCP does not work with records/messages. It works with streams. There is a reason for that. The reason is that a record can be large. If it fits one packet - who cares. With small records maybe the transport protocol can concatenate/batch the records to save the network some load (Nagle's algo). With large records, you need to do retransmission if data is lost, you have head-of-line blocking, you must deal with congestion backoff, flow control (what if remote end is busy, or has scarce memory). In traditional IP the solution to RPC would be to open a new stream for each request. This is how HTTP 1.0 worked. It had certain advantages. > You want congestion control? TCP can give you one version tuned for the wide-area and another version tuned for the datacenter. You can go quite far without tuning congestion control algos inside datacenter. Cubic and BBR are good enough for almost everyone. There is cost in using DCTCP. It's in the same ballpark as ECN. Great in theory, solves real issue. But requires heavy investment to get any value off it. > You want a low-latency network stack? Well that’s a challenge TCP has a 40-year history of trying to optimize away, and when that falls short, ultimately looking to SmartNICs to solve. Okay, yes, TCP was not intended to be low latency. With traditional API's it's impossible to get zero-copy, gosh, even stuff like getting a transmission completion signal is basically impossible. Normal NIC's have offloads and they work, they save the CPU from some dumb work. If you want low latency then go for RDMA-like approach. > In contrast, RPC was designed from the start to optimize round-trip performance in low-latency networks. Err... Okay, so by this definition HTTP is not RPC. Fine. Okay, so we're talking about inter-datacenter "trusted", "fast", "low latency", with homogenous network, protocol, that is tuned to RPC-style, request-response traffic. Stuff like this comes to mind: Memcached (including binary protocol), redis, RDMA, grpc, and quic. Fine! TCP is indeed not optimal. Should you care? Nope. In real life tricks like tcp connection-pooling, raw UDP protocols (think: memcached UDP) work just fine. Furthermore, often you need encryption (even inside datacenter). The power of TCP and other generic protocols is that they work fine on the lossy, untrusted public internet. I can connect to redis over 300ms lossy stream just fine (and I often do!). Would we benefit if there is a protocol (like QUIC or SCTP) that is tuned for inter-datacenter RPC - probably yes. Should you care - nope. Look at how hard making QUIC was. I'm a pragmatist and I'm allergic to theoretical discussions. There is a reason why memcached binary protocol and memcached UDP protocols are super obscure and (almost) nobody runs them (the reason is that latency is not the most important thing. Simple code in the client is often more important). > But coming back to the specific question of RPC vs TCP in the datacenter, it still has me scratching my head about why it hasn’t happened I can give one answer: BSD sockets API. They are limiting, and think about a case of large response. Ideally the server would like to give it to the kernel and move on to next request. This is not how the API's work. I think SCTP + some kind of zercopy would give you quite a decent starting point. However, SCTP didn't catch on, and the API has serious flaws. The next big thing is QUIC. But unless someone provides a kernel API for it, it won't be "fast" or "low latency".
roeles 4y ago
The third protocol the author is talking about sounds a lot like Content Centric Networking/Named Data Networking.
yencabulator 4y ago
> SMTP was a purpose-built RPC for email SMTP is a very chatty stateful back-and-forth protocol, where what message is legal to send when depends on the state of the connection. It's roughly the opposite of a singular request-response RPC. > judgment often reflects biases As seems to be true with this article, too.
yencabulator 4y ago
> and not limited to HTTP’s five operations What is he talking about? HTTP methods are an arbitrary string. > The method token is case-sensitive because it might be used as a gateway to object-based systems with case-sensitive method names. By convention, standardized methods are defined in all-uppercase US-ASCII letters. Even if you talk about the standard methods, there's 8 listed right in the RFC: https://httpwg.org/specs/rfc9110.html#methods https://httpwg.org/specs/rfc9110.html#methods

154 points by zdw 4y ago

What are the best recommendations for modern day low latency RPC? Google protocol buffers? Or some message queue such as ZeroMQ?

tonymillion 4y ago
You’re thinking too high level, this is talking about replacing TCP or UDP with a new transport protocol designed specifically with RPC in mind. Instead of opening a TCP (or UDP) socket and writing data (or data grams) to it you’d open a (for want of a better term) RPCP socket and write data that represents the procedure call and parameters to it. I assume the protocol would have assurances of some kind of reliability (unlike UDP) and procedure call data boundaries (unlike TCP) built in at the lowest levels along with request/response sequences to allow out of order completions.
dist1ll 4y ago
Check out eRPC.io It leverages DPDK and is perfect for low-latency RPC in lossless DCs
- jiggawatts 4y ago
  It would be nice to have something like that, but have libraries for most common languages.
- jeffbee 4y ago
  Where would I find one of these lossless fabrics?
drbruced 4y ago
Part of what we're arguing in this article is that there is no good answer today because of the assumption that TCP was good enough. Homa is challenging that assumption (as many others have tried before). My view is that if you want something better than TCP for RPC today, QUIC is the best thing available, as discussed here: https://systemsapproach.substack.com/p/quic-is-not-a-tcp-replacement https://systemsapproach.substack.com/p/quic-is-not-a-tcp-rep... What you run over that for your RPC is a matter of taste - gRPC is certainly popular, and can be run over QUIC.
omginternets 4y ago
I'm a huge fan of capnproto.org
en4bz 4y ago
Flatbuffers, Cap'n Proto, or SBE over RDMA.

iambvk 4y ago

Good to learn a bit of history about TCP. It always felt weird developing RPC layers on top of TCP cause requests and responses end up tied to the underlying socket -- which doesn't need to be the case ever.

AceJohnny2 4y ago
How else would the kernel know to which application it should route a response?
- cjsplat 4y ago
  Destinations can be described in the packet headers. That is how the current systems work.
- iambvk 4y ago
  Destination port number should be sufficient. For example, imagine if there are two sockets established between server and client processes (due to multiple ips or roaming, time lags, etc.), then in theory, request received from one socket could be responded through another socket. From application pov it doesn't need to care about the socket. We can do this over TCP, but connection-oriented nature of TCP makes this weird.

deleted 4y ago

[deleted]

irq-1 4y ago

Really interesting article. Wish there was a discussion of performance for various RPCs from the past.

cryptonector 4y ago
There's so many... Apollo Domain's. ONC RPC. DCE RPC (and MSFT RPC, which derives from DCE RPC). SOAP. LustreRPC. And who knows how many others. The biggest problem with RPC in general is that historically it was synchronous because that's what was easy to implement in the 80s. Fix that and an RPC sucks only as much as the encoding system it uses. But RPC == remote procedure call, and that causes people to instantly think "synchronous", and that is a kiss of death.
- cjsplat 4y ago
  Well of course once you say "procedure call", people think synchronous. For the sender, asynchronous RPC is just a convenient marshalling and return interface for the sender. It is pretty clunky for the A-RPC caller to create an illusion in the code of the RPC being "just like" a local procedure call. For the called procedure an Async RPC can look just like a local sync procedure call, except for that whole address space thing. I don't think that sync RPC was popular because it was easy to implement. I think it was popular because it was easy to code to. Multi-threaded coding is easy to get wrong, and was poorly understood at the time.
  - cryptonector 4y ago
    > I think it was popular because it was easy to code to. That's not that unlike it being easy to implement. > Multi-threaded coding is easy to get wrong, [...] Yeah, but the better design is async/await and so on, not threading, but that stuff was not very popular in the 80s. The approach to concurrency in the 80s was heavy on context switching because that's what was inherited from the 70s. It feels like the thinking was along the lines of "during the time you're waiting for a response some other process will run, and since we're used to heavy latency, who cares!", but that became less and less tenable. There's a reason all the distributed operating system research of the 80s died except for filesystems -- with filesystems you can consciously avoid hot ping pong by just not accessing shared resources concurrently, while the filesystem itself remains super useful.

cryptonector 4y ago

> the Internet seems to be missing a standard protocol for the request/response paradigm, with repeated attempts to force-fit TCP leading to inevitable mismatches HTTP is that. QUIC is that over UDP. We've discussed Ousterhout's paper about replacing TCP in the data center. Last I looked the answer was "not with what is described in the paper". > I have never understood why the Internet has worked so persistently to adapt TCP in support of request/reply workloads instead of standardizing an RPC transport protocol to complement TCP. > ... > But my original question was to ask why that hasn’t happened. The only answer I can come up with is that judgment often reflects biases. I can explain this. Some of it is bias, as TFA surmises, but there's a lot more to it. First off, there is an Internet RPC protocol, and it's the ONC RPC protocol, the one that NFS uses. ONC RPC does run over TCP and UDP, which, yes, TFA doesn't seem to want. Second point: adding new upper layer protocols (ULPs) like TCP and UDP and SCTP is fraught with middlebox and over-design problems, and you can look at SCTP for details. The third point is that the ability to implement and iterate quickly means that having to wait for kernel support for the new thing is a non-starter. This is the strongest reason why you won't see a new RPC ULP happen quickly and why there has been little interest in one. So between the middlebox issues and iteration latency issues the only way to do a new ULP is as a new binding of the RPC which must also have TCP and UDP bindings. I.e., all request-response protocols we have pretty much have to run on top of TCP or UDP. This fact of Internet life may suck, but that's just how it is. Making it go away will require lots of time. Developers don't have lots of time. Fourth point: running a request response protocol over UDP is quite reasonable, especially in the datacenter. Fifth point: building a new ULP will risk yielding the same over-design issues as SCTP all over again unless it's a new binding of another protocol. I'm beating a dead horse here; let's switch to the "bias" issue. RPC got a really bad rap in the 90s and 00s. The reason is that the RPC frameworks that came out of the 80s, and the more modern versions that followed them in the 90s and 00s, were all [mostly] synchronous. There's no reason for that: it was the easiest way to implement back then. Another aspect of synchrony is that RPC == remote procedure call and comes from an era when it was fashionable to build things like distributed remote shared memory (which is a performance disaster). The idea that RPC is like a function call instantly evokes synchrony even though RPCs are not condemned to be synchronous. That suggestion leads to aversion. Another reason that RPC frameworks got a back rap back then was the encoding systems used. It doesn't matter if the encoding system was trivial like XDR or complex like NDR or DER or PER or whatever -- the tooling for encoding rules from the 80s and 90s was just not remotely universal. As well we had the XML era (e.g., SOAP), which was very buzzword-rich and efficiency-poor. That bad reputation is undeserved, really, but there it is. Nowadays RPC is making a comeback via gRPC and friends, but it shouldn't surprise that they typically run over HTTP. Meanwhile the web folks iterated quickly. HTTP user-agent stacks quickly became async. HTTP APIs iterated from XML to JSON, and either way, with JavaScript and HTML, web apps got a great deal of flexibility and performed great by comparison to RPC apps. Plus the web security model is atrocious mainly because of the most useful thing about HTML: pages can have cross-site links. Cross-site linking is so incredibly useful that it can't be done without. And now you need a fancy user-agent (browsers), so the web folks built one (browsers). HTTP won out over RPC. Why not use HTTP for everything, then? Well, HTTP/1.1 is... not efficient. H2 is much better, but still over TCP. H3/QUIC should be perfectly fine in the datacenter, and for some things on the web (and eventually maybe all things on the web?), and runs over UDP. But HTTP is a request/response protocol. Just like any RPC! Plus ça change, plus ça devient la même chose. My argument: HTTP is the protocol you're looking for. And if ever we need a new ULP and are confident that we can deploy it, then we should design one for HTTP because a) that will satisfy the RPC need, b) it will tend to quash over-design instincts, c) it will have the same APIs/semantics as a protocol we're all already very familiar with, d) having one protocol with bindings to all applicable ULPs will help us get over the kernel support issues. (d) is especially appealing: you get to target one API and you get the best ULP locally supported. We do have a widely-deployed Internet RPC protocol called HTTP, and you get to run it over TCP or UDP, and its semantics are stable when we add new ULP support. That's a pretty good outcome. Though maybe it requires getting over one's own biases :) > That we have since turned HTTP into the Internet’s de facto RPC protocol (and then now realizing that it is suboptimal, are trying to optimize it by collapsing all the layers into the new QUIC protocol), is only a testament to how small a role technical rationale plays in what happens in industry. I object to the characterization of HTTP as suboptimal for RPC. TFA is clearly referring to TCP as being suboptimal for RPC, but not really covering how UDP is suboptimal for RPC. And TFA does not discuss why NFS switched from using UDP to TCP, say. It feels like TFA is just emoting that we ought to have a ULP specifically for pure RPC. That seems like a very biased position. Suppose we built a ULP for RPC though. We could easily expect HTTP to get adapted to run over it. What does HTTP add that an RPC doesn't need? Mostly headers, which the RPC can just not use, and URI components, which the RPC can also set to the smallest possible values, say. There's not that much light between RPC and HTTP. Both are request-response protocols. Accept HTTP as an RPC and you might be happy.

SillyUsername 4y ago

If you change the stack at TCP level... - How many hardware network components would need replacing/upgrading to support a layer not expected to change? - How many network + application software libraries would need to support parsing at packet level instead of byte level? - How many OSI diagrams showing that high level byte and string based protocols run above the transport layer need to be re-written to say that's no longer the case. - How many versions of RPC would need to be supported... just look at the numerous revisions of CORBA/RMI/SOAP(Document or RPC Style!)/REST/GRPC to see how many times you can re-invent (and misinterpret, break standards of) a wheel. - Look how long IPv6 needed buy in and refinement, and then adoption. That was a necessary change with no alternative too! There are many applications that shouldn't need TCP, such as those only doing short network hops or needing real time traffic with expected packet loss. They should use UDP but often don't. TCP is ubiquitous for the very reason that it is the chameleon mentioned in the article - TCP is not ideal, but it is convenient and well understood.

bob1029 4y ago

There are some things that current RPC frameworks simply cannot touch. I've been up to my elbows in TCP lately. Working on a low-latency, select-based socket server for streaming gaming applications. Each instance runs on 1 thread so there is absolutely no context switching delay when servicing player requests. The TCP streams are responsible for moving player inputs and game state as quickly as possible between system elements. There's also an externally facing websocket being serviced inside of these threads. The whole networking chunk of the loop usually completes within few tens of uS. Socket selects hang out for up to 10uS. I considered something like AspNetCore/kestrel, but that's a lot of machinery, GC liability, thread contention, et.al. gRPC also crossed my mind but I found some headache with the tooling.

Karrot_Kream 4y ago
Do you mean the `select` call? libuv and friends can offer higher throughput than `select`.
- User23 4y ago
  It’s been many many years since I’ve done low level multiplexing but I think epoll() is the no longer quite so new hotness? Ah I see libuv is a cross platform lib. I’d probably use that for production code, but it’s still really valuable to work with the low level API enough to understand it.
  - Karrot_Kream 4y ago
    Yeah epoll is what libuv calls on Linux. And agreed I think it's very valuable to try and use epoll or equivalent yourself at least once to understand what's happening.
- bob1029 4y ago
  Whatever OS primitive this API effectively calls is what I use: https://learn.microsoft.com/en-us/dotnet/api/system.net.sockets.socket.select?view=net-7.0#system-net-sockets-socket-select(system-collections-ilist-system-collections-ilist-system-collections-ilist-system-int32) https://learn.microsoft.com/en-us/dotnet/api/system.net.sock... Edit: This is the source: https://source.dot.net/#System.Net.Sockets/System/Net/Sockets/SocketPal.Unix.cs,a83ac8f2e1e985a6 https://source.dot.net/#System.Net.Sockets/System/Net/Socket...

manv1 4y ago

This paragraph is just wrong: "Another explanation is that the Internet has unnecessarily coupled the transport protocol with the rest of the RPC framework. Conflating the two naturally follows from the purpose-built examples I just gave: SMTP is bundled with MIME; SNMP is bundled with MIB; and HTTP is bundled with HTML. " I think at this point he's suffering from what old-timers used to call recto-cranial inversion.

codeflo 4y ago
I think I feel dumber for reading this. (The referenced paper by Ousterhout is quite good though, and doesn’t confuse the terminology. Maybe that one’s what HN should link and discuss.)
- matjet 4y ago
  To someone only weakly aware in this area, can you explain what is wrong with the statement? From a cursory view, these protocols indeed seem to normally carry the mentioned payloads.
  - lozenge 4y ago
    It's not like HTTP stops working if you choose to send something besides HTML. Most of its features still work and make sense for JSON payloads and the usual building blocks still work (eg HTTPS, load balancers, proxies, status codes, redirects...) There are a few bits that don't make sense eg cookies where the server sends something and the client is expected to remember it but they can just be ignored.
    - codeflo 4y ago
      Exactly. The HTTP standard doesn't really mention HTML, except tangentially in examples. And regarding cookies, there are even people who use cookies in RPC contexts, but this is rather rare, and as you said, entirely optional. Also, as a minor point, even when browsing the web, a minority of the actual requests deliver HTML -- most of them are for various kinds of media referenced by the HTML document.

codeflo 4y ago

I for one think we should finally replace DNS with JPEG. That makes exactly as much sense as this article.

jbverschoor 4y ago
jxl so the requests can be lossless, transparent, and multi-channeled

hardwaresofton 4y ago

If you'd like to watch Ousterhoust explain Homa and his vision of replacing TCP: https://www.youtube.com/watch?v=o2HBHckrdQc https://www.youtube.com/watch?v=o2HBHckrdQc Spoiler: Ousterhout is right, and the data shows he is right. Maybe not everyone knows it yet, or it's the kind of "right" that not everyone will ever get to (and we'll settle for "good enough" for much longer), but he is almost certainly right. Not to disparage the poster of this article of course -- I always love a good hot take or contrarian piece, but if you can't come with the same amount of data to support your thesis... Homa is probably dismissable on UX grounds (are people really going to switch to it?), but it's clear that long hard thinking has gone into the case for Homa (and it's implementation).

deathanatos 4y ago
I think you're right that he's "right", but the writing is shooting itself in the foot here. I agree that there's a mostly-unexplored middle between TCP & UDP, but you need to not utter things like "HTTP is bundled with HTML". I also think RPC is a poor label for "request/response" protocol (HTTP is request/response, and many of the problems here apply to HTTP too … but it isn't RPC). I'd like more discussion of the contenders in the space: why do they not work? QUIC is barely mentioned beyond "we tried it and nope". SCTP isn't mentioned on the OP at all, but if you click through the Homa stuff a bit, it's written off as "a WAN protocol" — whatever that is? I get that his focus is datacenters, but "WAN protocol" is what I feel most of us are interested in, and I don't immediately see why WAN vs. DC matters¹. What I'm also not getting from the Homa page is … what is the protocol, what are it's packet structures? Is there no documentation? (I've not hit all the third-degree links because I'm not a spider, but nothing seemed relevant or it seemed a little relevant but didn't hold the answers.) The old adage about data structures over algorithms seems to apply: > "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975) ¹I'd also want to know why a good "WAN" protocol that solves the stated problems wouldn't equally apply to the datacenter, modulo needing some adjustments around assumptions made about the network's latency and maybe reliability: a DC network, I'd hope, is going to have much lower latency. But like Raft timers, it seems like a by-use-case adjustable constant.

Animats 4y ago

There have been lots of RPC protocols. Here are some still in use. Transport level: * Sun RPC [1]. QNX still uses this. It can run over UDP or over raw Ethernet. It just transfers an array of bytes and gets an array of bytes back - marshalling is a higher level problem. It handles messages bigger than one packet, and retransmission. It's simple and performance is good, but there is no security. * Stream Control Transmission Protocol. Telcos still use this. It's how Signalling System 7 is sent over IP. Although many implementations exist for both, they've never been popular outside their niches. Marshalling level: * CORBA - you define interfaces in a special language and compile them. Data is not self-describing. * SOAP, from the XML era. * Google Protocol Buffers - another system where you compile definitions. * HTTP/JSON - where we are now. Plus a lot of Microsoft-specific stuff. Either you have a system where both ends have to exactly agree on format, you have a verbose format with unneeded description data in every message, or you have a very complicated negotiation at connection time. This leads to much disagreement. [1] https://en.wikipedia.org/wiki/Sun_RPC https://en.wikipedia.org/wiki/Sun_RPC

cjsplat 4y ago
Yeah, but the question at hand is why layer it over TCP. SunRPC over UDP is close, but then you push reliability and congestion management into every application.
- deathanatos 4y ago
  SCTP, from the parent's list, is a "over IP" protocol, not over TCP. (Though it can be tunneled through UDP, to get around Internet ossification.)
danans 4y ago
> CORBA - you define interfaces in a special language and compile them. Data is not self-describing. OMG the Common Object Request Broker Architecture (which coincidentally was created by OMG). I never thought I'd see that acronym again, and all these years later, I still don't understand what it is.
- 4y ago

cjsplat 4y ago

So the question is why hasn't an optimized RPC implementation emerged for the data center that avoids the headaches that come with layering over TCP. Turns out integrating with the control plane is hard at data center scale. When everything is working, an optimized protocol stack is great and everyone is happy. When congestion happens (fabric, kernel, memory bus, cache footprint, etc), and your optimized app causes a legacy production app to behave strangely, all hell breaks loose. For example, a distributed file system SRE sees some strange tail latency effects and doesn't know that a cluster is shared with a non-TCP app and of course the optimized stack doesn't respond to congestion the way TCP would. Worst case, the SREs notice this just after an update is pushed in their app, and they spend a random amount of time trying to figure out if the update is what changed the behavior. Best case, they know there are non-TCP apps in the area, and they ping the SRE for that stack and say "my app is behaving strange, please hit your big red off button so I can see if that fixes the problem." Someplace in between those options, the SRE is using TCP aware network debugging tools to try to figure out why this day is different from yesterday, or this cluster is different from the one where everything is working fine. Regardless, you get an unhappy SRE. But of course you never get just one unhappy SRE. So you need to generate a lot of value got justify their pain.

nubinetwork 4y ago

I'd be happy if DNS and NTP would just get off their butts and switch to TCP. No, DoH doesn't count.

cjsplat 4y ago
Why would NTP be better with retransmisson of stale packets and go-back-N semantics on drops?
- nubinetwork 4y ago
  UDP can be spoofed for amplification attacks, and since ISPs won't implement reverse path filtering, the only other option is to get rid of UDP.
seanhunter 4y ago
Normal DNS will switch to TCP for any large request and has done for quite some time.
- noselasd 4y ago
  Unless you're using the musl libc - which is in heavy use in conteainerized applications
  - seanhunter 4y ago
    OK well if it does zone transfers those are over TCP only and always have been.
    - deleted 4y ago
      [deleted]
    - nubinetwork 4y ago
      There is more to DNS than just zone transfers...

roncesvalles 4y ago

You could just use UDP instead of TCP and implement some subset of TCP features that you need in-band, like sequence numbers. This is not a new concept. The Facebook Memcache paper [1] in 2013 described using UDP for intra-datacenter requests. [1] https://research.facebook.com/publications/scaling-memcache-at-facebook/ https://research.facebook.com/publications/scaling-memcache-...

cjsplat 4y ago
Sure, but what is UDP bringing to the party then? The pseudoheader? A reasonable RPC layer could just layer on IP.
- wmf 4y ago
  The UDP header will give you proper ECMP hashing.
  - cjsplat 4y ago
    Yup - this is the correct answer
- roncesvalles 4y ago
  IIRC, UDP just adds port numbers and a checksum to IP. It's about as bare bones as it can get, unless you want to get rid of the checksum too.
  - rezonant 4y ago
    Some implementations do skip checksum (leaving it as zero), for instance you can configure a Linux UDP socket to skip it if you can't afford the time it would take to calculate them.
- cryptonector 4y ago
  > Sure, but what is UDP bringing to the party then? Interoperability. It's been how many years since SCTP was published? How's that going?

kwantam 4y ago

Is this a good time to bring up T/TCP? https://www.rfc-editor.org/rfc/rfc1644 https://www.rfc-editor.org/rfc/rfc1644 This memo specifies T/TCP, an experimental TCP extension for efficient transaction-oriented (request/response) service. This backwards-compatible extension could fill the gap between the current connection-oriented TCP and the datagram-based UDP. I have no illusions about the likelihood it'll ever see use, but I've always thought T/TCP was cool and fun.

aidenn0 4y ago

Osterhout's response to Pepelnjak's criticism is solid, but he mentions not being familiar with SCTP. Homa is much more fairly compared to SCTP than TCP, since there is a large intersection with the problems they solve. Osterhout seems to think that ACK-based sender congestion avoidance is a significant problem in the datacenter. SCTP uses ACK-based sender congestion avoidance, but lacks the head-of-line blocking that TCP has[1]. Comparing Homa to SCTP would be a good way to test this thesis. 1: Or rather it's optional.

majke 4y ago

I'm a pragmatist and there are couple of points in this article that give me allergic reaction. > You want individual messages instead of a byte-stream? TCP has an option for that. That links to. https://book.systemsapproach.org/e2e/tcp.html#record-boundaries https://book.systemsapproach.org/e2e/tcp.html#record-boundar... In my opinion you can't be serious suggesting usage of URG or PUSH as message boundaries. TCP does not work with records/messages. It works with streams. There is a reason for that. The reason is that a record can be large. If it fits one packet - who cares. With small records maybe the transport protocol can concatenate/batch the records to save the network some load (Nagle's algo). With large records, you need to do retransmission if data is lost, you have head-of-line blocking, you must deal with congestion backoff, flow control (what if remote end is busy, or has scarce memory). In traditional IP the solution to RPC would be to open a new stream for each request. This is how HTTP 1.0 worked. It had certain advantages. > You want congestion control? TCP can give you one version tuned for the wide-area and another version tuned for the datacenter. You can go quite far without tuning congestion control algos inside datacenter. Cubic and BBR are good enough for almost everyone. There is cost in using DCTCP. It's in the same ballpark as ECN. Great in theory, solves real issue. But requires heavy investment to get any value off it. > You want a low-latency network stack? Well that’s a challenge TCP has a 40-year history of trying to optimize away, and when that falls short, ultimately looking to SmartNICs to solve. Okay, yes, TCP was not intended to be low latency. With traditional API's it's impossible to get zero-copy, gosh, even stuff like getting a transmission completion signal is basically impossible. Normal NIC's have offloads and they work, they save the CPU from some dumb work. If you want low latency then go for RDMA-like approach. > In contrast, RPC was designed from the start to optimize round-trip performance in low-latency networks. Err... Okay, so by this definition HTTP is not RPC. Fine. Okay, so we're talking about inter-datacenter "trusted", "fast", "low latency", with homogenous network, protocol, that is tuned to RPC-style, request-response traffic. Stuff like this comes to mind: Memcached (including binary protocol), redis, RDMA, grpc, and quic. Fine! TCP is indeed not optimal. Should you care? Nope. In real life tricks like tcp connection-pooling, raw UDP protocols (think: memcached UDP) work just fine. Furthermore, often you need encryption (even inside datacenter). The power of TCP and other generic protocols is that they work fine on the lossy, untrusted public internet. I can connect to redis over 300ms lossy stream just fine (and I often do!). Would we benefit if there is a protocol (like QUIC or SCTP) that is tuned for inter-datacenter RPC - probably yes. Should you care - nope. Look at how hard making QUIC was. I'm a pragmatist and I'm allergic to theoretical discussions. There is a reason why memcached binary protocol and memcached UDP protocols are super obscure and (almost) nobody runs them (the reason is that latency is not the most important thing. Simple code in the client is often more important). > But coming back to the specific question of RPC vs TCP in the datacenter, it still has me scratching my head about why it hasn’t happened I can give one answer: BSD sockets API. They are limiting, and think about a case of large response. Ideally the server would like to give it to the kernel and move on to next request. This is not how the API's work. I think SCTP + some kind of zercopy would give you quite a decent starting point. However, SCTP didn't catch on, and the API has serious flaws. The next big thing is QUIC. But unless someone provides a kernel API for it, it won't be "fast" or "low latency".

roeles 4y ago

The third protocol the author is talking about sounds a lot like Content Centric Networking/Named Data Networking.

yencabulator 4y ago

> SMTP was a purpose-built RPC for email SMTP is a very chatty stateful back-and-forth protocol, where what message is legal to send when depends on the state of the connection. It's roughly the opposite of a singular request-response RPC. > judgment often reflects biases As seems to be true with this article, too.

yencabulator 4y ago

> and not limited to HTTP’s five operations What is he talking about? HTTP methods are an arbitrary string. > The method token is case-sensitive because it might be used as a gateway to object-based systems with case-sensitive method names. By convention, standardized methods are defined in all-uppercase US-ASCII letters. Even if you talk about the standard methods, there's 8 listed right in the RFC: https://httpwg.org/specs/rfc9110.html#methods https://httpwg.org/specs/rfc9110.html#methods