11 ms·

a) Use standardized error codes, not standardized error messages. Clients are responsible for internationalization, which includes presenting error messages in

a) Use standardized error codes, not standardized error messages. Clients are responsible for internationalization, which includes presenting error messages in the user's language. If you document a set of error codes as an enum, the client can present a user-friendly error message in the user's language based on the error code. If there are dynamic parts of the error message, i.e. "404: There is no user with ID 123456 in the system", then the user ID should be extracted into the error response body, so that it can be provided correctly to the user in the user's language. b) Pagination is the devil. The state of the server can change while the user is paginating, leading to fragile clients. Don't paginate your API. If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter. This ensures consistency from one API call to the next.

CrimsonRain 5y ago
b -> What happens if the "list of IDs" is 10k or 100k or 1M+?
- solatic 5y ago
  There's typically business-logic ways to require a range to be specified, and then you specify a maximum range. For example, if the resource in question is audit events, you require the user to specify a time range, up to a maximum of a day, or seven days, or whatever performance / budget constraints allow for.
  - akvadrako 5y ago
    That's just like paging by day or whatever.
  - CrimsonRain 5y ago
    So if I specify a range and something changes between in that range while I'm interacting with those, sounds like same problem is back? Just same issue; different package :) If I do a search, a paginated search result can have an ID so I can paginate between the data without a new data messing up my pagination. But for normal entities, simple pagination is (mostly) more than enough. The solution you are describing is overkill and almost no benefit at all.
JimDabell 5y ago
> Pagination is the devil. The state of the server can change while the user is paginating, leading to fragile clients. Don't paginate your API. The problem is not pagination, and the solution is not to avoid pagination. The problem is offset-based pagination, and the solution is to use cursor-based pagination. > If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter. This has the same problem as you described above. The state of the server can change between fetching the list of IDs and operating on them.
- solatic 5y ago
  Cursor-based pagination doesn't solve your state issue, it forces the server to create a copy of state for the cursor request. This is complex to implement correctly - for example, if the user does not actually paginate through all the entries, when do you dump the unused cursor? If the user issues the same request over and over, do you return a cached cursor or re-copy the state into a new cursor? If you re-copy the state, are you defended from a malicious actor who sends 1,000 new requests? Of course, all these concerns can be mitigated, but it's easier to just design without pagination in the first place if you can. > The state of the server can change between fetching the list of IDs and operating on them. Right, for example, a returned ID may have been deleted before the user can issue a query for that resource. But this is usually far more comprehensible to clients, particularly if the resource requires authorization such that only the client is permitted to delete that resource.
  - JimDabell 5y ago
    Cursor-based pagination doesn’t require anything at all like you describe. Are you sure you aren’t mistaking it for something else?
  - sigmaml 5y ago
    You appear to be referring to a database cursor. It is quite simple to implement pagination in the application layer using a unique record identifier (primary key or ULID or ...) as an anchor for the navigation. From that unique ID, we can then fetch the previous `n` or the next `n` records, depending on the direction of the navigation. This way, the server remains stateless, since the anchor (possibly sent as an encoded / obfuscated token, which can include some other parameters such as page size) is supplied by the client with each pagination request. Unless I am missing something in your argument.
eyelidlessness 5y ago
Pagination by offset can be bad. Pagination by cursor which is specifically designed for your list of resources is perfectly reasonable. Use stable pagination queries, not ?*
nuttingd 5y ago
Pagination is harder than it seems to get right. I think pagination is only predictable under these conditions: 1) The offset used for the next fetch must be based on a pointer to a unique key. We can't rely on the number of rows previously seen. With this rule, deletes which occur to rows in previous pages will not cause unpredictable contractions. 2) The paged result set must be sorted based on a monotonically increasing field, like created_at, plus enough other fields to create a unique key. You could lean on the PK for this, i.e.: ORDER BY (created_at, id) ASC. With this rule, new inserts which occur during page enumeration will only affect unseen pages (and we'll see them eventually) The API call looks roughly like this: /orders/?region=US&offset=(2022-03-12T07:05:58Z&ord_1234)&limit=100 The DB query looks roughly like this: SELECT * FROM orders WHERE (created_at, id) > (:offset_created_at, :offset_id) OR ( :offset_created_at IS NULL AND :offset_id IS NULL ) ORDER BY (created_at, id) ASC LIMIT :page_size EDIT: formatting
alkonaut 5y ago
Any pagination is brittle. Regardless of whether its by page, cursor or something else. You can't have equal sized pages if there is a risk that there are deletions on a previous page or at the exact index you use as a cursor etc. The solution is usually simple: assume it doesn't matter. Write a spec for your feature and explicitly state that "In the solution we assume it doesn't matter if the same record is ocassionally reported twice or a record is missing from the pagination in some cases". Done.

CrimsonRain 5y ago

b -> What happens if the "list of IDs" is 10k or 100k or 1M+?

solatic 5y ago
There's typically business-logic ways to require a range to be specified, and then you specify a maximum range. For example, if the resource in question is audit events, you require the user to specify a time range, up to a maximum of a day, or seven days, or whatever performance / budget constraints allow for.
- akvadrako 5y ago
  That's just like paging by day or whatever.
- CrimsonRain 5y ago
  So if I specify a range and something changes between in that range while I'm interacting with those, sounds like same problem is back? Just same issue; different package :) If I do a search, a paginated search result can have an ID so I can paginate between the data without a new data messing up my pagination. But for normal entities, simple pagination is (mostly) more than enough. The solution you are describing is overkill and almost no benefit at all.

JimDabell 5y ago

> Pagination is the devil. The state of the server can change while the user is paginating, leading to fragile clients. Don't paginate your API. The problem is not pagination, and the solution is not to avoid pagination. The problem is offset-based pagination, and the solution is to use cursor-based pagination. > If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter. This has the same problem as you described above. The state of the server can change between fetching the list of IDs and operating on them.

solatic 5y ago
Cursor-based pagination doesn't solve your state issue, it forces the server to create a copy of state for the cursor request. This is complex to implement correctly - for example, if the user does not actually paginate through all the entries, when do you dump the unused cursor? If the user issues the same request over and over, do you return a cached cursor or re-copy the state into a new cursor? If you re-copy the state, are you defended from a malicious actor who sends 1,000 new requests? Of course, all these concerns can be mitigated, but it's easier to just design without pagination in the first place if you can. > The state of the server can change between fetching the list of IDs and operating on them. Right, for example, a returned ID may have been deleted before the user can issue a query for that resource. But this is usually far more comprehensible to clients, particularly if the resource requires authorization such that only the client is permitted to delete that resource.
- JimDabell 5y ago
  Cursor-based pagination doesn’t require anything at all like you describe. Are you sure you aren’t mistaking it for something else?
- sigmaml 5y ago
  You appear to be referring to a database cursor. It is quite simple to implement pagination in the application layer using a unique record identifier (primary key or ULID or ...) as an anchor for the navigation. From that unique ID, we can then fetch the previous `n` or the next `n` records, depending on the direction of the navigation. This way, the server remains stateless, since the anchor (possibly sent as an encoded / obfuscated token, which can include some other parameters such as page size) is supplied by the client with each pagination request. Unless I am missing something in your argument.

eyelidlessness 5y ago

Pagination by offset can be bad. Pagination by cursor which is specifically designed for your list of resources is perfectly reasonable. Use stable pagination queries, not ?*

nuttingd 5y ago

Pagination is harder than it seems to get right. I think pagination is only predictable under these conditions: 1) The offset used for the next fetch must be based on a pointer to a unique key. We can't rely on the number of rows previously seen. With this rule, deletes which occur to rows in previous pages will not cause unpredictable contractions. 2) The paged result set must be sorted based on a monotonically increasing field, like created_at, plus enough other fields to create a unique key. You could lean on the PK for this, i.e.: ORDER BY (created_at, id) ASC. With this rule, new inserts which occur during page enumeration will only affect unseen pages (and we'll see them eventually) The API call looks roughly like this: /orders/?region=US&offset=(2022-03-12T07:05:58Z&ord_1234)&limit=100 The DB query looks roughly like this: SELECT * FROM orders WHERE (created_at, id) > (:offset_created_at, :offset_id) OR ( :offset_created_at IS NULL AND :offset_id IS NULL ) ORDER BY (created_at, id) ASC LIMIT :page_size EDIT: formatting

alkonaut 5y ago

Any pagination is brittle. Regardless of whether its by page, cursor or something else. You can't have equal sized pages if there is a risk that there are deletions on a previous page or at the exact index you use as a cursor etc. The solution is usually simple: assume it doesn't matter. Write a spec for your feature and explicitly state that "In the solution we assume it doesn't matter if the same record is ocassionally reported twice or a record is missing from the pagination in some cases". Done.