5 ms·

Pagination is one of those things I feel like so many of these things get wrong. LIMIT/OFFSET (or as MS likes to call it, TOP/SEEK) style results in O(n²) opera

Pagination is one of those things I feel like so many of these things get wrong. LIMIT/OFFSET (or as MS likes to call it, TOP/SEEK) style results in O(n²) operations; an automated tool trying to pull the entirety of a large collection in such a scenario is not good. I have to again recommend the excellent "Pagination done the Right Way" presentation[1] from Use the Index, Luke (an equally excellent site). Just return a link to the "next page" (make the next page opaque); while this removes the ability to of the client to go to an arbitrary page on its own, in practice, I've never seen that matter, and the generalized URL format allows you to seek instead of by LIMIT/OFFSET by something indexable by the DB; HTTP even includes a standardized header for just this purpose[2]. I also think the generalized URL is more in line with Fielding's definition of REST, in that the response links to another resource. (I don't know if being in the header invalidates this; to me, it does not.) If you get the metadata out of the data section of your response, and move it to the headers, where it belongs, this usually then lets you keep the "collection" of items you are return as an array (because if you need to put metadata along side it, you need: { "the_data_you_really_want": [1, 2, 3], "metadata, e.g., pagination": {} } vs. ["the", "data", "you", "wanted", "in a sensible format"] ) (I've seen metadata-pushed-into-the-body turn API endpoints that literally need to return no more than "true" or "false" into object that then force a client to know and look up the correct key in an object… sigh.) [1]: http://use-the-index-luke.com/no-offset http://use-the-index-luke.com/no-offset [2]: https://tools.ietf.org/html/rfc5988 https://tools.ietf.org/html/rfc5988

brandonbloom 10y ago
Big +1 to killing offset in favor of order-by and an "after" or "before" criteria. However, -1 to the idea of sticking any important "metadata" in headers. For one, it's dramatically easier to interact with response bodies than headers at the terminal or with simple programming libraries and tools. Having only one way of representing data requires fewer special cases. JSON is far superior to key/value pairs in terms of ability to represent structure. Lastly, if you ever decide to use a different "transport" protocol other than HTTP, you'll be glad you are already using a common representation without a side channel.
- ferentchak 10y ago
  How do you feel about using StatusCodes to return api information. /tacos/8 ... not found 404 vs { "meta":{ "errors":["not found"] } } or something like that without getting into the form of the JSON returned.
  - seanp2k2 10y ago
    The actual HTTP status should be relevant. I don't want to get HTTP/200 with {error: not found}. Instead, useful errors should be like: HTTP/400 {error: invalid foobar in request. Foobar should be a string, but instead got '123'} Or something like that. Tell developers what the problem actually was, and potentially how to fix it. http://jsonapi.org/format/#error-objects http://jsonapi.org/format/#error-objects has some good ideas and suggests a standard. That said, even discarding the body, you should be able to determine if a request was successful by the HTTP status code alone, IMO. If you made a new thing, 201, thing ain't there, 404, your request sucks, 400, etc. Don't make people using your API parse your [especially true if you've invented your own format] JSON errors if they don't want to.
    - ChrisAtWork 10y ago
      "Success With Info" is something that just causes problems. The number of examples we found of "Success with Info" going horribly wrong was abundant. Turns out checking return codes is something people have failed to do pretty much since the invention of the return code...
      - dwightgunning 10y ago
        Is that a failure of the people or failure of the spec?
        ferentchak 10y ago
        I think in many ways it's just tradition. Old browsers did wonky stuff with non 200 codes and that just created inertia that made people never quite take the effort to follow the spec.
vmarsy 10y ago
From the parent Microsoft guideline [1], they make the explicit differentiation between Server-driven and Client-driven paging, the Server-driven seems to be exactly what you describe in your first paragraph: > Paginated responses MUST indicate a partial result by including a continuation token in the response. The absence of a continuation token means that no additional pages are available. > Clients MUST treat the continuation URL as opaque, which means that query options may not be changed while iterating over a set of partial results. > Example: { ..., "value": [...], "@nextLink": "{opaqueUrl}" } They then explain Client-driven paging, if the client really wants to do it: > Clients MAY use $top and $skip query parameters to specify a number of results to return and an offset. The server SHOULD honor the values specified by the client; however, clients MUST be prepared to handle responses that contain a different page size or contain a continuation token. But if you want to obtain the entire collection, you would not use Client-driven paging, but only rely on Server-driven paging [1] https://github.com/Microsoft/api-guidelines/blob/master/Guidelines.md#98-pagination https://github.com/Microsoft/api-guidelines/blob/master/Guid...
agentgt 10y ago
Making the next page opaque as you said is a good thing also because limit and offset will often become unstable or incorrect if the dataset is changing without a proper timestamp. That is you may get more or less records then you are expecting which maybe important if you need to allocate size. Almost all of my rest services I require sending a timestamp for batch processing (that is only records created/updated/removed before given timestamp are shown).
foota 10y ago
Returning just an array from a json-endpoint that may contain sensitive data is a vulnerability: http://haacked.com/archive/2008/11/20/anatomy-of-a-subtle-json-vulnerability.aspx/ http://haacked.com/archive/2008/11/20/anatomy-of-a-subtle-js... (edit: may be mitigated in newer browsers)
- niftich 10y ago
  The obvious mitigation is don't visit untrusted endpoints from your API client, and/or don't run a script interpreter inside your API client.
  - foota 10y ago
    That's not what the vulnerability is here. If you have a server endpoint that returns an array without wrapping it as a property on an object then an attacker can write a webpage that overrides the Array prototype and requests the json from your page as a script, thereby bypassing the cross origin check it would be subject to as an xhr request. So if you have a page which returns sensitive information as an array you (were, it seems browsers probably don't allow you to override the array prototype anymore) are vulnerable to the attacker stealing a signed in users secrets.
    - niftich 10y ago
      On the contrary, that's precisely the vulnerability: 1. The API server needs authentication, which gets cached in the user-agent. 2. The same user-agent, with the same authentication context, is then directed to a malicious site. 3. The user-agent contains a javascript interpreter, and the malicious website serves a script to override the Array prototype. 4. The same script then executes a CSRF request to the API server, reusing the cached authentication context, and therefore stealing the response. A browser fits all the necessary conditions that must exist for this vulnerability to be exploited, but if you're the consumer of the API, you don't have to use a browser as your user-agent. You can write your own, which you'd do if you're making these requests programmatically in a non-web application. Regardless as to which user-agent you use, a viable mitigation is to not visit untrusted endpoints with the same cookie jar or authentication context. Another viable mitigation is to omit including a script interpreter in your custom API client. This is in fact usually the case.
draw_down 10y ago
> while this removes the ability to of the client to go to an arbitrary page on its own Yeah, that's the whole problem with this approach.
- deathanatos 10y ago
  Sometimes, you can look at what the index is and find a better thing than page numbers to use; for example, if your collection is ordered/indexed on a timestamp, you can navigate to dates&times instead of page numbers. (e.g., in a UI presentation you might expose the ability to choose a date to go to, instead of a page)

brandonbloom 10y ago

Big +1 to killing offset in favor of order-by and an "after" or "before" criteria. However, -1 to the idea of sticking any important "metadata" in headers. For one, it's dramatically easier to interact with response bodies than headers at the terminal or with simple programming libraries and tools. Having only one way of representing data requires fewer special cases. JSON is far superior to key/value pairs in terms of ability to represent structure. Lastly, if you ever decide to use a different "transport" protocol other than HTTP, you'll be glad you are already using a common representation without a side channel.

ferentchak 10y ago
How do you feel about using StatusCodes to return api information. /tacos/8 ... not found 404 vs { "meta":{ "errors":["not found"] } } or something like that without getting into the form of the JSON returned.
- seanp2k2 10y ago
  The actual HTTP status should be relevant. I don't want to get HTTP/200 with {error: not found}. Instead, useful errors should be like: HTTP/400 {error: invalid foobar in request. Foobar should be a string, but instead got '123'} Or something like that. Tell developers what the problem actually was, and potentially how to fix it. http://jsonapi.org/format/#error-objects http://jsonapi.org/format/#error-objects has some good ideas and suggests a standard. That said, even discarding the body, you should be able to determine if a request was successful by the HTTP status code alone, IMO. If you made a new thing, 201, thing ain't there, 404, your request sucks, 400, etc. Don't make people using your API parse your [especially true if you've invented your own format] JSON errors if they don't want to.
  - ChrisAtWork 10y ago
    "Success With Info" is something that just causes problems. The number of examples we found of "Success with Info" going horribly wrong was abundant. Turns out checking return codes is something people have failed to do pretty much since the invention of the return code...
    - dwightgunning 10y ago
      Is that a failure of the people or failure of the spec?
      - ferentchak 10y ago
        I think in many ways it's just tradition. Old browsers did wonky stuff with non 200 codes and that just created inertia that made people never quite take the effort to follow the spec.

vmarsy 10y ago

From the parent Microsoft guideline [1], they make the explicit differentiation between Server-driven and Client-driven paging, the Server-driven seems to be exactly what you describe in your first paragraph: > Paginated responses MUST indicate a partial result by including a continuation token in the response. The absence of a continuation token means that no additional pages are available. > Clients MUST treat the continuation URL as opaque, which means that query options may not be changed while iterating over a set of partial results. > Example: { ..., "value": [...], "@nextLink": "{opaqueUrl}" } They then explain Client-driven paging, if the client really wants to do it: > Clients MAY use $top and $skip query parameters to specify a number of results to return and an offset. The server SHOULD honor the values specified by the client; however, clients MUST be prepared to handle responses that contain a different page size or contain a continuation token. But if you want to obtain the entire collection, you would not use Client-driven paging, but only rely on Server-driven paging [1] https://github.com/Microsoft/api-guidelines/blob/master/Guidelines.md#98-pagination https://github.com/Microsoft/api-guidelines/blob/master/Guid...

agentgt 10y ago

Making the next page opaque as you said is a good thing also because limit and offset will often become unstable or incorrect if the dataset is changing without a proper timestamp. That is you may get more or less records then you are expecting which maybe important if you need to allocate size. Almost all of my rest services I require sending a timestamp for batch processing (that is only records created/updated/removed before given timestamp are shown).

foota 10y ago

Returning just an array from a json-endpoint that may contain sensitive data is a vulnerability: http://haacked.com/archive/2008/11/20/anatomy-of-a-subtle-json-vulnerability.aspx/ http://haacked.com/archive/2008/11/20/anatomy-of-a-subtle-js... (edit: may be mitigated in newer browsers)

niftich 10y ago
The obvious mitigation is don't visit untrusted endpoints from your API client, and/or don't run a script interpreter inside your API client.
- foota 10y ago
  That's not what the vulnerability is here. If you have a server endpoint that returns an array without wrapping it as a property on an object then an attacker can write a webpage that overrides the Array prototype and requests the json from your page as a script, thereby bypassing the cross origin check it would be subject to as an xhr request. So if you have a page which returns sensitive information as an array you (were, it seems browsers probably don't allow you to override the array prototype anymore) are vulnerable to the attacker stealing a signed in users secrets.
  - niftich 10y ago
    On the contrary, that's precisely the vulnerability: 1. The API server needs authentication, which gets cached in the user-agent. 2. The same user-agent, with the same authentication context, is then directed to a malicious site. 3. The user-agent contains a javascript interpreter, and the malicious website serves a script to override the Array prototype. 4. The same script then executes a CSRF request to the API server, reusing the cached authentication context, and therefore stealing the response. A browser fits all the necessary conditions that must exist for this vulnerability to be exploited, but if you're the consumer of the API, you don't have to use a browser as your user-agent. You can write your own, which you'd do if you're making these requests programmatically in a non-web application. Regardless as to which user-agent you use, a viable mitigation is to not visit untrusted endpoints with the same cookie jar or authentication context. Another viable mitigation is to omit including a script interpreter in your custom API client. This is in fact usually the case.

draw_down 10y ago

> while this removes the ability to of the client to go to an arbitrary page on its own Yeah, that's the whole problem with this approach.

deathanatos 10y ago
Sometimes, you can look at what the index is and find a better thing than page numbers to use; for example, if your collection is ordered/indexed on a timestamp, you can navigate to dates&times instead of page numbers. (e.g., in a UI presentation you might expose the ability to choose a date to go to, instead of a page)