11 ms·
Post-REST
- anentropic 8y agoI stumbled on this the other day http://rsocket.io/ http://rsocket.io/ Does anyone have an opinion if it's a good idea and likely to gain adoption? It sounds to me like it implements the sort of thing being discussed in this post
- woody83 8y agoRSocket is gaining use at Facebook and is beginning to get traction at Alibaba. Spring 5.2 appears that it will also have first-class support for RSocket. There is also a company, founded by some of the creators of RSocket, that are building an open-source application development platform for it called Proteus.
- cryptonector 8y agoREST for the control plane works because there are few enough "resources" that you can have a URI for each and CRUD works. Much more importantly, the pressure to have complex multi-resource transactions is low because the control plane tends to have simple schemas and there is no need to do complex multi-resource transactions with any kind of atomicity. The data plane is not at all like that. There can be many many resources, and very complex transactions. PostgREST, a REST interface to PostgreSQL, shows you can have REST for the data plane, but it achieves that by forcing one to have simple transactions with any and all complexity wrapped into PG functions so that the REST interface can remain simple. The place where REST falls down, really, is transactions. For simple enough schemas and transactions involving only one relation, it's easy enough to have URIs for resources and collections. But for anything much more complex than that REST simply fails. Instead one ends up writing an end-point that accepts POSTs whose request bodies encode complex transactions, and this ends up resembling SOAP (basically RPC over HTTP) rather than REST. Imagine an HTTP/1.1 verb named TRANSACT whose body consists of a concatenation of the requests that you really want to do, and with a header controlling / reflecting transaction control options (e.g., atomicity). If HTTP had such a thing, then it would be a lot easier to build a proper RESTful interface to the data plane. But HTTP has no such thing, and probably never will. Incidentally, it's useful to consider how NFSv3 had RPCs for every little thing, while NFSv4 has only one RPC (COMPOUND) that is... a lot like the TRANSACT I imagine above -- it's just not that crazy a thing to imagine. The transaction complexity problem in HTTP is not unique to HTTP!
- dragonwriter 8y ago> The place where REST falls down, really, is transactions. For simple enough schemas and transactions involving only one relation, it's easy enough to have URIs for resources and collections. But for anything much more complex than that REST simply fails. Instead one ends up writing an end-point that accepts POSTs whose request bodies encode complex transactions, and this ends up resembling SOAP (basically RPC over HTTP) rather than REST. It's perfectly RESTful; resources that represents actions are just as valid as any other resources. I feel like the mistake here extends a common mistake of object-oriented analysis where the valid idea that objects should correspond to nouns are somehow combined with forgetting that actions are nouns. If you have an API with complex transactions (especially if you need to expose state of those transactions), why wouldn't they be resources? Why wouldn't you post a representation to a collection endpoint? How is that not exactly REST?
- cryptonector 8y ago> It's perfectly RESTful; ... Yeah, so this is where just what the meaning of REST is comes in. There's no representation of the "transaction" resource that you can GET, therefore they are not like the other resources (the ones you can HEAD/GET). You can only POST representations of transactions that internally contain ad-hoc encodings of CRUD operations on multiple other resources. This really feels like it should have been a first-class operation in HTTP rather than constructed in an ad-hoc way as needed.
- niftich 8y agoIf you design in this model, 'GET transaction-uri' should get, y'know, the payload of the transaction, or a status resource, or whichever you asked for based on proactive or reactive content negotiation, or whichever the server wishes to make available. It's just such a mental departure from the last 20 years of 'do-the-verb-to-the-noun' object oriented design that it's not often done.
- cryptonector 8y ago
- burtonator 8y agoI think one big win could be to just hide the underlying APIs with better tooling. For example, with GRPC your client bindings are automatically compiled and you just use them. Is it using protocol buffers? Do you care? To a certain extent we do care. A lot of these things "just work" out of the box but there are some practical real world implications here. At Datastreamer we push about about 100GB of JSON to our customers per day. This is all news and blog content. Sometimes our customers complain that using JSON is a waste. Realistically though it costs them about $150 extra per month to parse the JSON vs using protocol buffers. We made this decision because implementing JSON is a LOT easier than implementing protocol buffers. Most of our customers end up putting a junior or intermediate engineer on the implementation and they're already familiar with JSON. However, if the tooling hides the encoding and even the wire protocol this means we can upgrade faster. We're still not at a point yet where the tooling is free. You tend to get stuck around something like GRPC or GraphQL anyway. Hopefully this is resolve in the future and we can get all this infrastructure optimization for free.
- denormalfloat 8y ago> Is it using protocol buffers? Do you care? I don't, but other people do. For anything low latency, the cost of encoding/decoding starts to become extremely difficult to avoid. But that said, I'm pretty sure GRPC can use JSON internally, so it's up to you which encoding to use.
- philwelch 8y ago> For anything low latency, the cost of encoding/decoding starts to become extremely difficult to avoid. Whether this is an argument against JSON and in favor of Protobuf or vice-versa depends on which language you're operating in. Most benchmarks I've seen show that Protobuf is faster to encode and decode for Java and Golang while JSON is faster for Python and Javascript. IMO, if you're worried about low latency, you're also likely using a compiled language where Protobuf will be faster anyway.
- AceJohnny2 8y ago> For anything low latency, the cost of encoding/decoding starts to become extremely difficult to avoid. Consider this: At the scale Google operates, fractions of a % of improvement are worth doing for saving the company (tens/hundreds/more) thousands of dollars in CPU/energy/anything, and they have the people to throw at the problem. Does an alternative solution really improve any/all of throughput/latency/reliability?
- turblety 8y agoI get the points of why people don't like REST, and especially the latency argument is quite true, however somewhat mitigated by http2. But I like that REST is quite easy for developers to reason about. I actually built a websockets library (more proof of concept) [1] that uses a rest like syntax, implementing something similar to an express API. It exposes both a websockets server and http. It's FAR from anywhere near complete and just a concept. I have no doubt there are better solutions out there, but I just don't think REST is all that bad. But I do also love @burtonator's idea of maybe not using JSON, but hiding the complexity of something like RPC. [1] https://github.com/markwylde/restocket https://github.com/markwylde/restocket
- denormalfloat 8y agoREST is a good fit if the operations you want to do are CRUD, or if the data model you have is hierarchical. This is for most people, but it breaks down when there involve multiple steps. A bank transaction is a good example, because it involves multiple updates that need to be done atomically. Mapping this to REST is complex. It's possible of course, but more complicated than the equivalent RPC counterpart. Hence, REST is not better in some common cases.
- anentropic 8y agoREST resources don't have to map 1-to-1 with your db tables The transaction itself can be a resource This seems almost maybe an argument against taking microservices too far until a single service can't own the whole transaction
- vidarh 8y agoThis is a great point, and I think this is a major point of confusion with REST. People think of resources as a table or row with CRUD operations rather than in the wider sense as an object with a generic set of operations to trigger transitions to a new state. It's worth reiterating that REST operations are conceptually much more generic than CRUD. Nothing says that what you GET or PUT/POST need correspond directly to attributes of an object or columns in a database - in the widest sense they simply represent the current state of a resource and a state transition you wish to invoke respectively. The point is not to (only) encode persistent state, but any state. In a sense representing state that is not persisted on the server is more essential - resources that are persistent on the server can be operated on without always sending the full state. But information that is not persisted on the server needs to be encoded and transmitted with the request/response in some way.
- rb808 8y agoThe article is named Post-REST, but the summary says: "...REST still provides a good clean way to decompose complicated problems, and its extreme simplicity and resilience..." Is there a reason not to use REST then? I didn't quite understand.
- deleted 8y ago[deleted]
- umvi 8y ago> Is there a reason not to use REST then? I didn't quite understand There can be. For example in the world of satellite internet where every byte is precious and every tcp connection is a massive performance penalty (3 way handshake * distance to satellite at speed of light = huge latencies), a REST API can be very wasteful. Something like GraphQL is much more suited to that case where you can get only exactly what you need, nothing more, and on a single connection.
- icebraining 8y agoNothing in REST prevents you from asking and getting only what you need. Assuming you're using HTTP, that's a common use of URL parameters; alternatively, you can pass a more detailed request in the body (yes, even in GET).
- deleted 8y ago[deleted]
- dragonwriter 8y ago> Nothing in REST prevents you from asking and getting only what you need. Assuming you're using HTTP, that's a common use of URL parameters; alternatively, you can pass a more detailed request in the body (yes, even in GET). GET is specified as not having a body and conformant HTTP implementations ignore the body of a GET, so in addition to violating RESTs general proscription on violating the underlying protocol (HTTP, in this example), you break compatibility with standard toolchain components if you do this. What you seem to want is SEARCH pulled out from under the weight of WebDAV, i.e.: https://tools.ietf.org/html/draft-snell-search-method-01 https://tools.ietf.org/html/draft-snell-search-method-01
- adamcharnock 8y agoPlugging a personal project: I’ve been working on a project called Lightbus [1] which has been very useful for me in creating evented systems. It is still early days, but so far myself and a couple of others have found it useful. [1]: https://lightbus.org https://lightbus.org
- VBprogrammer 8y agoI was recently working on an API where we decided to use protobuf. I remember the first time I saw the message as encoded by protobuf; it really made me aware of how wasteful JSON can be. I know that usually that all comes out in the gzipping but there is nothing quite like seeing half a page of JSON become 1 line of gibberish to demonstrate the point.
- XCSme 8y agoProtobuf is really slow with encoding/decoding and also does unnecessary validation of the data. If you are just starting out with binary data serialization I recommend FlatBuffers instead of protobuf.
- Matthias247 8y agoReally slow compared to what? Json? A handwritten binary encoding? For most use cases protobuf should be fast enough. I also don’t understand the unnecessary validation part. If you receive data over the network you should always validate it. Even if it’s from within the same company network
- moogly 8y agoMaybe things have improved since, but the last time I tried FlatBuffers (about 2 years ago), I found the API (generated C#) to create the FlatBuffers cumbersome, inelegant and quite error prone.
- tannhaeuser 8y agoIf REST ultimately results in heavy use of message queues and async URL responses, the question is why not use eg. AMQP or other messaging protocol straight away rather than bother using HTTP facades/services as unecessary SPoFs at least between backends.
- jimbokun 8y agoDepends on how tightly coupled you want to be. Directly exposing your queue to external consumers seems very dangerous. Having a well defined and well documented REST API for external consumers, and using queues for communicating between tightly coupled services seems like a much better solution.
- tannhaeuser 8y agoWell-defined and well-documented isn't exactly what I'd use to describe "REST" APIs.
- Dowwie 8y agoWhy not?
- vemv 8y agoLooks like event-driven systems are all the rage now! I feel there's still too much competition/innovation in the message queue space though. Too many choices, with maddeningly different characteristics. And seemingly you can tweak most of those to behave somewhat similarly between them, which makes judgement even harder.
- denormalfloat 8y agoIsn't competition good for consumers though? Eventually, one of these network architectures will become more popular and we don't have to try them all out personally ourselves. I benefit greatly from people in the 90s testing out HTTP over other competing protocols, which, at the time may not have been obviously better.
- jimbokun 8y ago"I feel there's still too much competition/innovation in the message queue space though." Interesting, to me Kafka seems like the clear winner and the obvious choice for all of my queuing needs. What am I missing out on from other queuing systems?
- jimsmart 8y agoI recently evaluated a bunch of different messaging systems, and found NATS to have the widest support for different client languages, plus those clients are also maintained by the same parent company. With other MQs we have used, we have found much variety in the quality of the client libs. YYMV.
- cryptonector 8y agoAs it should be. The question is: can you use HTTP to transport event notifications? Actually, can you use TCP to transport event notifications? Yes, you can, but it's a bit unwieldy because you need some sort of keepalive / heartbeat with which to distinguish inactivity from transient network problems (and then attempt to reconnect). Then again, even if you were using UDP you'd have to do some sort of keepalive / heartbeat. I use an HTTP server that supports "tail -f" via chunked transfer-encoding responses to indeterminate byte range requests (Range: bytes=${offset}-) that don't end until the file is removed, renamed away, or replaced (renamed over). This server also supports weak ETags constructed from (device ID, inode number, generation number), If-Match, and If-None-Match. This allows clients to GET a resource whose body is the event stream and hang in there waiting for events. Again, you need to add a heartbeat in the event stream as HTTP chunked transfer-encoding does not have this, and you need to detect network problems. But otherwise this works fantastically well and it is HTTP and RESTful.
- crb002 8y agoReposting here. At Global Day of Code Retreat this past weekend I was surprised that our Game of Life implementation in SQL was the easiest. Why are we doing data schema type erasure at the Application layer? The "flexibility" and "encapsulation" aren't worth the manual hell of hand coding it back in all the way to the front end. Why not expose SQL schemas all the way to the front end? Even if the Application layer makes a view that the data store doesn't know about, everything downstream could benefit through automation. From there you get into issues mostly having to do with APIs that can be paginated, and how to optimize those connections to balance caching with network latency.
- jmull 8y ago> Why not expose SQL schemas all the way to the front end? Your API is a promise and if you break your promise bad things happen. If your API is SQL against some tables, you're promising you won't make any breaking changes to that -- or else you're promising you can roll out an update to all the servers and clients at the same time. SQL against tables has a very large surface area, and there are many details that are different depending on the version of your database software. So, you'll find you can't update your database software without also updating (and testing) all of your clients. The more complex and more widely distributed your software is, the more time-consuming and expensive it will be to update anything. If you have clients outside your direct control and running on the same release cycle, you'll find you can't update your database software at all without breaking your clients.
- TheDong 8y ago> SQL against tables has a very large surface area, and there are many details that are different depending on the version of your database software So you only offer an ANSI SQL compatible parser, not specific to any database (and it would then be compatible / compile-able to postgres, sqlite, whatever), and you only expose specific views into the database. Once you create such a public view, you treat it as immutable, and all API changes require new views. Each old view must function forever. This is not actually all that different from REST apis today; we version them, old versions must exist forever (or all clients must be updated), you can't update your server software that serves the rest API without being darn sure it's backwards compatible... The only real difference is we encode our CRUD operations in a non-sql language right now (ad-hoc json schemas usually) vs encoding them as the ANSI subset of SQL against some fictional database views. All of your complaints apply equally well to REST apis I think.
- peterwwillis 8y agoProblems with these assumptions: - QUIC. Yes, it's UDP, so its connections don't behave the way TCP ones do, but it's still based on IP and UDP connections, so it still breaks down under a number of circumstances (rotating backends, mobile clients changing connection types, firewalls/proxies, etc) - HTTP-based. If you use HTTP, you must accept its warts, like being stateless. Even considering a transport protocol that can multiplex connections, you still need to track all these individual stateless calls. Perhaps a protocol that took care of state might give you some benefits (such as workflowing?) - Workflow orchestration is just distributed centralized state machines. It shouldn't have much to do with a replacement for REST, because the API isn't necessarily controlling a state machine, and the API (in this case HTTP) isn't stateful anyway... Why don't we evolve APIs into a sort of language? Why make 4 separate requests (PUT this, GET that, munge it, POST a result, GET to confirm) when you know exactly what you want to do? We can instead build a language or spec where we can shove a variety of instructions (even a "workflow") into a request, let the backend handle the method of accomplishing it for us, and retrieve the result. This can be extended throughout microservice architectures. The transport protocol is a separate issue. So you want lower latency, and you want less round trips, and you want to survive network interrupt. Design a transport protocol for that (QUIC is a good base). If you want it to survive even more interruption and allow clients to split networks, start looking at intelligent tunnels with advanced routing mechanisms and spread them along the path. But that's a lot more funky than simply replacing TCP.
- acdha 8y ago> - QUIC. Yes, it's UDP, so its connections don't behave the way TCP ones do, but it's still based on IP and UDP connections, so it still breaks down under a number of circumstances (rotating backends, mobile clients changing connection types, firewalls/proxies, etc) Firewall support is a big concern but the other points you mentioned are design goals for QUIC — see section 9 of https://datatracker.ietf.org/doc/draft-ietf-quic-transport/?include_text=1 https://datatracker.ietf.org/doc/draft-ietf-quic-transport/?... (“Connection Migration”): > The use of a connection ID allows connections to survive changes to endpoint addresses (that is, IP address and/or port), such as those caused by an endpoint migrating to a new network. … Not all changes of peer address are intentional migrations. The peer could experience NAT rebinding: a change of address due to a middlebox, usually a NAT, allocating a new outgoing port or even a new outgoing IP address for a flow.
- tuukkah 8y ago> To start with, it’s not as though client developers can compose arbitrary queries, limited only by the semantics of GraphQL, and expect to get uniformly decent performance. (To be fair, the same is true of SQL.) Maybe I'm missing something, but isn't that exactly what you can expect from a GraphQL API as a sweet spot between SQL and REST? Semantics of GraphQL enable a schema-dependent set of queries, not arbitrary queries. > Anyhow, I see GraphQL as a convenience feature designed to make synchronous APIs run more efficiently. For asynchronicity, GraphQL has subscriptions. That being said, I agree GraphQL (of today at least) won't be a solution to every problem and I bet transports such as HTTP/2 and HTTP/3 will cause new things to be built which don't look like just a faster REST, GraphQL or MQTT. I would mention event sourcing and persistent logs in addition to just stream/queue processing. Tooling (like API management gateways and API consoles) is not still all there for GraphQL and MQTT either.
- tootie 8y agoI'm surprised we haven't seen more development already with HTTP/2 REST APIs. It let's you do your API at whatever logical level makes sense and then do your orchestration independently.
- cryptonector 8y ago> Semantics of GraphQL enable a schema-dependent set of queries, not arbitrary queries. I mean, in SQL useful queries are generally schema-dependent. Sure, you could have XPath/JSONPath/whatever extensions that allow you to query something closer to free-form documents, but your queries will still be schema-dependent in some way.
- tuukkah 8y agoSQL schema defines data and you cannot limit which queries are available on that data, whereas GraphQL schema defines both data and which queries are available.
- deleted 8y ago[deleted]
- deathanatos 8y ago> Setting up and tearing down an HTTP connection for every little operation you want to do is not free. "Doctor, it hurts when I do this." So don't do that; many HTTP libraries come with connection pooling / persistence built in, so it's quite common to not even need to handle this yourself. Should we also stop using relational databases because setting up a connection takes time? As for the rest of it, sure, you might have a queue, or some orchestration layer. Is that your consumer-facing API? Just b/c I push a message into some queue implementation does not mean I want to expose that implementation to my caller.
- jimbokun 8y agoHe has an entire section on "Post-REST: Persistent connections" which you seem to have missed. "As for the rest of it, sure, you might have a queue, or some orchestration layer. Is that your consumer-facing API? Just b/c I push a message into some queue implementation does not mean I want to expose that implementation to my caller." REST for external APIs, and queues for internal, more tightly coupled micro services seems like a good way to architect things.
- otabdeveloper1 8y ago> He has an entire section on "Post-REST: Persistent connections" which you seem to have missed. That section is really stupid. He goes on about HTTP/2 and QUIC while seemingly being ignorant of HTTP/1.1's existence. Keep-alive has existed for decades. Use it, it solves the problem definitively without adding any new ones. (People have yet to figure out what the point of HTTP/2 and QUIC was. Except for Google's not-invented-here syndrome, of course.)
- temporallobe 8y agoFor web applications that use "traditional" RESTful APIs with AJAX, most of the overhead is from setting up an AJAX call for every single call instead of using an "always-on" listener like websockets.
- dogpuncher 8y ago> The idea is you get a request, you validate it, maybe you do some computation on it, then you drop it on a queue and forget about it. > The next stage of request handling is implemented by services that read the queue and either route an answer back to the original requester or passes it on to another service stage. In this scenario, what are the mechanisms by which a service could route an answer back to the original requester?
- Aeolun 8y agoMy question exactly. Can’t exactly use long polling for this.
- pier25 8y agoTurns out most people are perfectly happy with their little SUV even if you can't transport a couple of tons of steel or win a Formula 1 race. It's ridiculous to think that anything can cover all use cases.
- Dowwie 8y agoWhere are the tech manager blog posts that discuss time, money, and effort in addition to tech pros and cons? GraphQL and SPAs are unconstrained tech solutions with questionable upside. Only well funded organizations can afford to throw money at projects that are more about attracting engineers than improving the bottom line. You don't need a huge marketing campaign and a non-profit institution to drive adoption of technology that has a compelling value proposition.