4 ms·

Optimising REST APIs

52 points by 6ren 15y ago

motter 15y ago
The standard approach to expressing errors in a REST API is to use HTTP status codes. As is often the case with REST, this works fine for simple systems, but is simply too limiting for more sophisticated systems, particularly those which might submit a JSON or XML document to describe a POST request. That's what the response body is for. If a client PUTs (or POSTs) a malformed representation, it's reasonable for the response to be 400 with error detail in the body. The "depth" parameter is kind of interesting, since it gives the client a choice as to whether multiple resources are included in the response. I'd be interested to see how others implement this (or if the client is given no choice at all). But essentially this is a straightforward choice between latency and response size.
- azylman 15y ago
  The author doesn't seem to realize that response bodies can exist for anything other than a 200 response. He says he uses Django almost exclusively. I've never used Django - is this perhaps a weakness of Django? As far as passing options in the header through Accept, I would think it would make more sense to pass it as GET parameters with the request, e.g.: GET /orders/432544?depth=1 instead of GET /orders/432544 Accept: application/json,application/json;q=1;depth=1
  - motter 15y ago
    It's not a limitation of django - subclasses of HttpResponse exist for certain error codes, and their content can be set in the normal manner. https://docs.djangoproject.com/en/dev/ref/request-response/#httpresponse-subclasses https://docs.djangoproject.com/en/dev/ref/request-response/#... Agreed - 'depth' makes more sense as a GET parameter, since it's a resource in its own right.
    - sanderjd 15y ago
      It definitely makes sense to have it in the query string, and I think people are already doing it that way, but the author makes a good point - it isn't a resource in its own right, it's merely a different representation. But in my experience, requiring Accept manipulation outside of .json versus .xml makes your service less usable by clients, which often don't expose anything besides the url.
      - masklinn 15y ago
        I think both interpretations can be defended. Though if you're playing with the Accept: header it's important to correctly set Vary: as well. > But in my experience, requiring Accept manipulation outside of .json versus .xml makes your service less usable by clients, which often don't expose anything besides the url. Browsers have that limitation (and numerous others), but if a programmatic HTTP library does not expose this, you need to throw it out. The Accept header is not part of the "immutable" headers of XHR either.
      - justincormack 15y ago
        it is not a different representation, it is a different resource. Imagune if you PUT it, it has a different meaning with different depth.
bsaunder 15y ago
While I like the standardization afforded by REST, I frequently run into some artificial limitations that are imposed on the different request methods. For example, I find that I would frequently like to pass structure information in a GET that would be more useful to send in a body (as afforded by a POST). One prime example relevant to this article would be a GET that was effectively a query by example. So a client could basically submit a skeleton of the data that it would like. This would let a client "go deep" on some portions of the structure and shallow on other aspects. Imagine the following GET request for that calorie counting Android App in the post: GET /orders/432544 { "toppings": [{ "calories": ""}] } With a response that would be: 200 OK { "toppings": [ { "calories": 100 } , { "calories": 25 } ] } This would give any client the ability to ask for exactly what it wanted with different depths to the structure if necessary. While you could serialize that into a URL, that seems like a kludge. I think in general, the HTTP methods made sense for their original design, but as we move on to building more flexible, integrated data system, a more powerful flexible API mechanism may be warranted.
- motter 15y ago
  Have you looked at MQL? - http://wiki.freebase.com/wiki/MQL http://wiki.freebase.com/wiki/MQL There are some good examples here: http://wiki.freebase.com/wiki/MQL_Cookbook http://wiki.freebase.com/wiki/MQL_Cookbook
  - bsaunder 15y ago
    I think I read about MQL a couple of years ago. I like it a lot. Not sure how/if it fits into the REST paradigm.
- masklinn 15y ago
  > This would give any client the ability to ask for exactly what it wanted with different depths to the structure if necessary. While you could serialize that into a URL, that seems like a kludge. Not sure why: your structure looks like an ad-hoc (and hard to read) selection query language (e.g. XPath, CSS selectors, ...). I'm sure it sounds good when throwing the idea in the air "oh the response has the exact same shape as the body", but I'd expect it to not be very good in practice (much like Go's weird-ass date formats)
6ren 15y ago
Most REST APIs are simple, returning flattish records, shallowly nested, with a maximum depth of one list, and some choices within. Simple is attractive, but this leads to the REST problem of "chatty" APIs, going back-and-forth to the server for each lookup/join, to locate the resource. As soon as you start "denormalizing", it makes sense to consider RDB/SQL solutions: there, you avoid joins/lookups between "normalized" flat records by having the data already pre-assembled in the form needed by frequent requests. Here, the problem isn't performance of database joins, but the latency of the whole network stack. Still, the same solution works. My playful RESTful proposal to achieve this is to consider an SQL query ("SELECT... WHERE..." etc) as the name of the resource returned. Through caching these names for these resources, it is scalable in the REST way. This is odd, because the SQL grammar allows all sorts of "names" to identify a "resource", so they don't look like names. But if different queries return the same table - these are just different names for the same resource, which REST permits. (It's more scalable to use the same name for the same resource, but that tends to happen anyway through convention.) SELECT/*/WHERE/a/AND/b # is the same as: SELECT/*/WHERE/b/AND/a --- Is this a real problem that businesses need to solve (will pay to fix), beyond annoying developers? Many website have become slow, as they go back-and-forth aggregating content. Bandwidth is fine; latency is the problem. It's one motivation for Amazon's Silk browser, which assembles content in the cloud, which is local to the services hosted there. EDIT "naming resources" is a database; a hierarchical database, the sort that relational databases (largely) replaced. It's not surprising that relational algebra is applicable to REST problems.

52 points by 6ren 15y ago

The standard approach to expressing errors in a REST API is to use HTTP status codes. As is often the case with REST, this works fine for simple systems, but is simply too limiting for more sophisticated systems, particularly those which might submit a JSON or XML document to describe a POST request. That's what the response body is for. If a client PUTs (or POSTs) a malformed representation, it's reasonable for the response to be 400 with error detail in the body. The "depth" parameter is kind of interesting, since it gives the client a choice as to whether multiple resources are included in the response. I'd be interested to see how others implement this (or if the client is given no choice at all). But essentially this is a straightforward choice between latency and response size.

azylman 15y ago
The author doesn't seem to realize that response bodies can exist for anything other than a 200 response. He says he uses Django almost exclusively. I've never used Django - is this perhaps a weakness of Django? As far as passing options in the header through Accept, I would think it would make more sense to pass it as GET parameters with the request, e.g.: GET /orders/432544?depth=1 instead of GET /orders/432544 Accept: application/json,application/json;q=1;depth=1
- motter 15y ago
  It's not a limitation of django - subclasses of HttpResponse exist for certain error codes, and their content can be set in the normal manner. https://docs.djangoproject.com/en/dev/ref/request-response/#httpresponse-subclasses https://docs.djangoproject.com/en/dev/ref/request-response/#... Agreed - 'depth' makes more sense as a GET parameter, since it's a resource in its own right.
  - sanderjd 15y ago
    It definitely makes sense to have it in the query string, and I think people are already doing it that way, but the author makes a good point - it isn't a resource in its own right, it's merely a different representation. But in my experience, requiring Accept manipulation outside of .json versus .xml makes your service less usable by clients, which often don't expose anything besides the url.
    - masklinn 15y ago
      I think both interpretations can be defended. Though if you're playing with the Accept: header it's important to correctly set Vary: as well. > But in my experience, requiring Accept manipulation outside of .json versus .xml makes your service less usable by clients, which often don't expose anything besides the url. Browsers have that limitation (and numerous others), but if a programmatic HTTP library does not expose this, you need to throw it out. The Accept header is not part of the "immutable" headers of XHR either.
    - justincormack 15y ago
      it is not a different representation, it is a different resource. Imagune if you PUT it, it has a different meaning with different depth.

bsaunder 15y ago

While I like the standardization afforded by REST, I frequently run into some artificial limitations that are imposed on the different request methods. For example, I find that I would frequently like to pass structure information in a GET that would be more useful to send in a body (as afforded by a POST). One prime example relevant to this article would be a GET that was effectively a query by example. So a client could basically submit a skeleton of the data that it would like. This would let a client "go deep" on some portions of the structure and shallow on other aspects. Imagine the following GET request for that calorie counting Android App in the post: GET /orders/432544 { "toppings": [{ "calories": ""}] } With a response that would be: 200 OK { "toppings": [ { "calories": 100 } , { "calories": 25 } ] } This would give any client the ability to ask for exactly what it wanted with different depths to the structure if necessary. While you could serialize that into a URL, that seems like a kludge. I think in general, the HTTP methods made sense for their original design, but as we move on to building more flexible, integrated data system, a more powerful flexible API mechanism may be warranted.

motter 15y ago
Have you looked at MQL? - http://wiki.freebase.com/wiki/MQL http://wiki.freebase.com/wiki/MQL There are some good examples here: http://wiki.freebase.com/wiki/MQL_Cookbook http://wiki.freebase.com/wiki/MQL_Cookbook
- bsaunder 15y ago
  I think I read about MQL a couple of years ago. I like it a lot. Not sure how/if it fits into the REST paradigm.
masklinn 15y ago
> This would give any client the ability to ask for exactly what it wanted with different depths to the structure if necessary. While you could serialize that into a URL, that seems like a kludge. Not sure why: your structure looks like an ad-hoc (and hard to read) selection query language (e.g. XPath, CSS selectors, ...). I'm sure it sounds good when throwing the idea in the air "oh the response has the exact same shape as the body", but I'd expect it to not be very good in practice (much like Go's weird-ass date formats)

6ren 15y ago

Most REST APIs are simple, returning flattish records, shallowly nested, with a maximum depth of one list, and some choices within. Simple is attractive, but this leads to the REST problem of "chatty" APIs, going back-and-forth to the server for each lookup/join, to locate the resource. As soon as you start "denormalizing", it makes sense to consider RDB/SQL solutions: there, you avoid joins/lookups between "normalized" flat records by having the data already pre-assembled in the form needed by frequent requests. Here, the problem isn't performance of database joins, but the latency of the whole network stack. Still, the same solution works. My playful RESTful proposal to achieve this is to consider an SQL query ("SELECT... WHERE..." etc) as the name of the resource returned. Through caching these names for these resources, it is scalable in the REST way. This is odd, because the SQL grammar allows all sorts of "names" to identify a "resource", so they don't look like names. But if different queries return the same table - these are just different names for the same resource, which REST permits. (It's more scalable to use the same name for the same resource, but that tends to happen anyway through convention.) SELECT/*/WHERE/a/AND/b # is the same as: SELECT/*/WHERE/b/AND/a --- Is this a real problem that businesses need to solve (will pay to fix), beyond annoying developers? Many website have become slow, as they go back-and-forth aggregating content. Bandwidth is fine; latency is the problem. It's one motivation for Amazon's Silk browser, which assembles content in the cloud, which is local to the services hosted there. EDIT "naming resources" is a database; a hierarchical database, the sort that relational databases (largely) replaced. It's not surprising that relational algebra is applicable to REST problems.