7 ms·

Read about the important of idempotence in distributed systems HTTP GET is idempotent, which seems trivial, but it's worth more than people think. With JSON-R

by chubot 4y ago

Read about the important of idempotence in distributed systems HTTP GET is idempotent, which seems trivial, but it's worth more than people think. With JSON-RPC you can't safely retry any RPC, and that matters at scale! (If the transport is HTTP, I'm pretty sure all JSON-RPC are POSTs. JSON-RPC can be good for local communication like language servers, where you don't really care about retrying or caching) FWIW Google had/has a very RPC-like model, and when I left many years ago there, I remember they came around to giving explicit guidance to use a REST-like model (which you can do in an RPC system). That is, where you have more nouns than verbs, and you GET nouns.

dastbe 4y ago
i think the benefit of client expectations around verbs isn’t the be-all-end-all. i can still write a non-idempotent fairly easily. writing resource oriented apis does benefit quite a bit, as it often forces you into good api patterns around idempotency. many aws apis and almost all new ones follow this pattern and they are much betterto integrate with because of it.
pshirshov 4y ago
> HTTP GET is idempotent Yeah, def getHandler() = { val a = randomString() + readDatabase(); writeDatabase(a); return a;}
- yandie 4y ago
  Technically you can implement whatever you want under a `GET` - you can even stuff a body to a `GET` even though the RFC doesn't allow it (see ElasticSearch API). However that doesn't mean it's a good implementation. What you're doing there is gonna break caching for example, unless that's intended. If I'm calling a `GET` and your service is behaving like a `POST`, I'm not gonna be able to rely on standard semantics for a lot of things!!!
  - pshirshov 4y ago
    If you can't enforce a contract it's not a contract.
    - BoorishBears 4y ago
      Maybe should learn what an RFC is, and check out this one before you keep trying to counter the document that formally minted the verbs in the first place: https://www.rfc-editor.org/rfc/rfc2616#section-9.1.2 https://www.rfc-editor.org/rfc/rfc2616#section-9.1.2 See section 9.1.2: Idempotent Methods, where they succinctly address the point: > Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them. - It's funny, I wouldn't have thought the RFC needed to say this. I'd have thought it was a waste of words since it'd be obvious to absolutely anyone that you can't... what? magically force developers to not write a bit of code in a system you have literally no control over? Yet here we are, proving the authors well-prepared all these years later.
      - pshirshov 4y ago
        > Naturally, it is not possible to ensure that the server does not generate side-effects Of course it's possible, there are ways to guarantee that and to prove that. That's an area of ongoing research. E.g. https://link.springer.com/chapter/10.1007/978-3-642-36594-2_13 https://link.springer.com/chapter/10.1007/978-3-642-36594-2_... Though I've said just one thing - you can't expect that a GET query would be idempotent. You may only hope.
        ninthcat 4y ago
        It is not possible to prove that calling the server will not produce side effects. The link you provided does not address side effects, but correctness, which is much weaker. A trivial example is that calling the server will consume electricity.
  - BoorishBears 4y ago
    I saw a truly truly awful codebase get around the caching bit by appending a random string of bytes to the query string. If you're willing to ignore the rules you can get away with pretty awful stuff that no one should really need to account for in a general conversation.
    - imiric 4y ago
      > I saw a truly truly awful codebase get around the caching bit by appending a random string of bytes to the query string. This is a common cache busting technique. Why is it a bad thing?
      - BoorishBears 4y ago
        Why do I get the impression this is a leading question... If you have the option between cache busting and just using the correct HTTP verb, please choose the latter. As to whether there was an option in this particular case, I think the fact I referred to it as a "truly truly awful codebase" should provide a hint.
      - talideon 4y ago
        The correct solution for cache busting is generally to put a low TTL on the resource in question. Adding random crap in the end is only good if you're dealing with utterly broken caches or your webserver does some tremendously stupid header rewriting. In my experience, when people resort to that, it's down to a lack of understanding of HTTP itself. YMMV, however.
- kelnos 4y ago
  HTTP GET being idempotent is an agreed-upon contract. If you violate it, that's up to you, but that would be a red flag to me as to the quality of your APIs.
  - pshirshov 4y ago
    These guys may somehow disagree with you: Etag, Last Modified, the whole If family.
daxfohl 4y ago
Pretty much this. I was very anti-REST back circa 2015-2017 when I started doing web dev. I was doing solo work and all my services were JSON-RPC (I can't remember whether that was an actual standard then or if I just rigged it). I was (and still am!) into functional programming, and RPC seemed to fit better: it's just a remote function call. Nakedly PUTting a resource just seemed to violate my idea of good architecture. In 2017 I left to join Azure, which is REST-heavy (in all this, I mean "modern" REST, not "Fielding" REST). I hated it at first, but eventually saw how nice it is to have the conventions. Less to explain in public API docs, easier to generate scaffolding, common themes of how to destructure incoming requests in middleware handlers and middle-tier services, a standard way to serialize references to resources across the platform. I was a convert. Moving to google in 2021, they're still very RPC, and it feels so cluttered and arbitrary. I'm not there anymore. Yes, REST is roughly just some conventions around RPC at the core (though you get things like caching for free), but ultimately it's a really nice set of conventions that are easy to follow and limit surprise, and are standard across the industry. As one user commented, you do sometimes have to work around REST to get it to fit, typically futzing in headers or request metadata, but to me the pros outweigh the cons.
- gspetr 4y ago
  Can you point to some sources of information about the modern REST you are talking about?
  - talideon 4y ago
    "Modern" REST generally disposes of things like hypertext and representations as being too complex, instead leaning on things Swagger/OpenAPI to declare an IDL equivalent, so you're getting effectively an RPC system with some of the guarantees you'd get from REST. Azure's APIs are more RESTful than most: they actually bother using resource URLs as identifiers. I've never understood what people find so hard about HATEOAS. It's like people have never interacted with a HTML form or followed links to stuff, or have difficulty figuring out how webpages can consist of multiple linked resources.
    - 4y ago
wruza 4y ago
What prevents you from making specific calls idempotent? You can even specify it via a POST/PUT method in any sort of RPC call to make HTTP proxies (or automatic resubmitters) aware of this detail. Idempotency is not exclusive to REST. When you invoke File-Save twice in Notepad it doesn’t create two different files either.
nullandvoid 4y ago
Isn't the cache situation somewhat solved via defining your RPC as either a "query" or "mutation" (for example in systems such as https://trpc.io/ https://trpc.io/ > https://trpc.io/docs/caching https://trpc.io/docs/caching)?
- chubot 4y ago
  How would proxies and load balancers know about this? You would have to encode your app-specific logic in every proxy or load balancer. In HTTP, they already exist. The top two comments in this thread explain this more: https://news.ycombinator.com/item?id=34228122 https://news.ycombinator.com/item?id=34228122
  - still_grokking 4y ago
    > How would proxies and load balancers know about this? By using a proper protocol. Not everything is nail just because the modern thin-client is only able to wield the HTTP hammer.

by chubot 4y ago

dastbe 4y ago

i think the benefit of client expectations around verbs isn’t the be-all-end-all. i can still write a non-idempotent fairly easily. writing resource oriented apis does benefit quite a bit, as it often forces you into good api patterns around idempotency. many aws apis and almost all new ones follow this pattern and they are much betterto integrate with because of it.

pshirshov 4y ago

> HTTP GET is idempotent Yeah, def getHandler() = { val a = randomString() + readDatabase(); writeDatabase(a); return a;}

yandie 4y ago
Technically you can implement whatever you want under a `GET` - you can even stuff a body to a `GET` even though the RFC doesn't allow it (see ElasticSearch API). However that doesn't mean it's a good implementation. What you're doing there is gonna break caching for example, unless that's intended. If I'm calling a `GET` and your service is behaving like a `POST`, I'm not gonna be able to rely on standard semantics for a lot of things!!!
- pshirshov 4y ago
  If you can't enforce a contract it's not a contract.
  - BoorishBears 4y ago
    Maybe should learn what an RFC is, and check out this one before you keep trying to counter the document that formally minted the verbs in the first place: https://www.rfc-editor.org/rfc/rfc2616#section-9.1.2 https://www.rfc-editor.org/rfc/rfc2616#section-9.1.2 See section 9.1.2: Idempotent Methods, where they succinctly address the point: > Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them. - It's funny, I wouldn't have thought the RFC needed to say this. I'd have thought it was a waste of words since it'd be obvious to absolutely anyone that you can't... what? magically force developers to not write a bit of code in a system you have literally no control over? Yet here we are, proving the authors well-prepared all these years later.
    - pshirshov 4y ago
      > Naturally, it is not possible to ensure that the server does not generate side-effects Of course it's possible, there are ways to guarantee that and to prove that. That's an area of ongoing research. E.g. https://link.springer.com/chapter/10.1007/978-3-642-36594-2_13 https://link.springer.com/chapter/10.1007/978-3-642-36594-2_... Though I've said just one thing - you can't expect that a GET query would be idempotent. You may only hope.
      - ninthcat 4y ago
        It is not possible to prove that calling the server will not produce side effects. The link you provided does not address side effects, but correctness, which is much weaker. A trivial example is that calling the server will consume electricity.
- BoorishBears 4y ago
  I saw a truly truly awful codebase get around the caching bit by appending a random string of bytes to the query string. If you're willing to ignore the rules you can get away with pretty awful stuff that no one should really need to account for in a general conversation.
  - imiric 4y ago
    > I saw a truly truly awful codebase get around the caching bit by appending a random string of bytes to the query string. This is a common cache busting technique. Why is it a bad thing?
    - BoorishBears 4y ago
      Why do I get the impression this is a leading question... If you have the option between cache busting and just using the correct HTTP verb, please choose the latter. As to whether there was an option in this particular case, I think the fact I referred to it as a "truly truly awful codebase" should provide a hint.
    - talideon 4y ago
      The correct solution for cache busting is generally to put a low TTL on the resource in question. Adding random crap in the end is only good if you're dealing with utterly broken caches or your webserver does some tremendously stupid header rewriting. In my experience, when people resort to that, it's down to a lack of understanding of HTTP itself. YMMV, however.
kelnos 4y ago
HTTP GET being idempotent is an agreed-upon contract. If you violate it, that's up to you, but that would be a red flag to me as to the quality of your APIs.
- pshirshov 4y ago
  These guys may somehow disagree with you: Etag, Last Modified, the whole If family.

daxfohl 4y ago

Pretty much this. I was very anti-REST back circa 2015-2017 when I started doing web dev. I was doing solo work and all my services were JSON-RPC (I can't remember whether that was an actual standard then or if I just rigged it). I was (and still am!) into functional programming, and RPC seemed to fit better: it's just a remote function call. Nakedly PUTting a resource just seemed to violate my idea of good architecture. In 2017 I left to join Azure, which is REST-heavy (in all this, I mean "modern" REST, not "Fielding" REST). I hated it at first, but eventually saw how nice it is to have the conventions. Less to explain in public API docs, easier to generate scaffolding, common themes of how to destructure incoming requests in middleware handlers and middle-tier services, a standard way to serialize references to resources across the platform. I was a convert. Moving to google in 2021, they're still very RPC, and it feels so cluttered and arbitrary. I'm not there anymore. Yes, REST is roughly just some conventions around RPC at the core (though you get things like caching for free), but ultimately it's a really nice set of conventions that are easy to follow and limit surprise, and are standard across the industry. As one user commented, you do sometimes have to work around REST to get it to fit, typically futzing in headers or request metadata, but to me the pros outweigh the cons.

gspetr 4y ago
Can you point to some sources of information about the modern REST you are talking about?
- talideon 4y ago
  "Modern" REST generally disposes of things like hypertext and representations as being too complex, instead leaning on things Swagger/OpenAPI to declare an IDL equivalent, so you're getting effectively an RPC system with some of the guarantees you'd get from REST. Azure's APIs are more RESTful than most: they actually bother using resource URLs as identifiers. I've never understood what people find so hard about HATEOAS. It's like people have never interacted with a HTML form or followed links to stuff, or have difficulty figuring out how webpages can consist of multiple linked resources.
  - 4y ago

wruza 4y ago

What prevents you from making specific calls idempotent? You can even specify it via a POST/PUT method in any sort of RPC call to make HTTP proxies (or automatic resubmitters) aware of this detail. Idempotency is not exclusive to REST. When you invoke File-Save twice in Notepad it doesn’t create two different files either.

nullandvoid 4y ago

Isn't the cache situation somewhat solved via defining your RPC as either a "query" or "mutation" (for example in systems such as https://trpc.io/ https://trpc.io/ > https://trpc.io/docs/caching https://trpc.io/docs/caching)?

chubot 4y ago
How would proxies and load balancers know about this? You would have to encode your app-specific logic in every proxy or load balancer. In HTTP, they already exist. The top two comments in this thread explain this more: https://news.ycombinator.com/item?id=34228122 https://news.ycombinator.com/item?id=34228122
- still_grokking 4y ago
  > How would proxies and load balancers know about this? By using a proper protocol. Not everything is nail just because the modern thin-client is only able to wield the HTTP hammer.