5 ms·

“Once the originally desired resource is created, there are two alternative ways to deal with the temporary task resource: API client must issue DELETE request

“Once the originally desired resource is created, there are two alternative ways to deal with the temporary task resource: API client must issue DELETE request, so that server purges it. Until then, server responds with 303 See Other status. Once deleted, 404 Not Found will be returned for subsequent GET /queue/12345 requests. or, garbage collection can be a server’s job to do: once task is complete server can safely remove it and respond with the 410 Gone on subsequent GET /queue/12345 requests.” How do you distinguish between an id referring to a finished task that no longer exists (410 Gone), and an id that never referred to any task (404 Not Found)? Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale...

delinka 8y ago
I'd say "gone" is a temporary state that exists shortly after task completion. A task shows as "gone" for, say, 15min, and then is deleted, producing a 404 in subsequent requests. On further thought, it seems to me that we necessarily need to keep completed tasks around until asked by a client to delete them. If a task produces an exception, we need to notate that for when the client next asks about the task's status. If the task ran successfully to completion, we need to notate that for when the client next asks. If the client needs to know all this, then it's the client who needs to ask for deletions as well - we shouldn't assume it's OK to remove records about tasks (unless there's a published policy that completed/error data will be purged after a certain period, e.g. 30, 60, 90 days...)
vfaronov 8y ago
https://tools.ietf.org/html/rfc7231#section-6.5.9 https://tools.ietf.org/html/rfc7231#section-6.5.9 > It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.
naasking 8y ago
> Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale... Why doesn't that scale?
- aaaaaaaaaab 8y ago
  Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed.
  - DougWebb 8y ago
    If your IDs are completely unique, you could add them to a Bloom filter [0] when cleaning up the resource. Then you could use that to determine if an incoming ID has possibly been used before or definitely not. [0] https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter
    - jaggederest 8y ago
      Or you could just have your IDs be monotonically increasing and look at the minimum. Any task prior to that number is "410 Gone" by definition.
    - striking 8y ago
      Bloom filters are probabilistic, meaning they don't always produce the right result. From the article you posted: > False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed (though this can be addressed with a "counting" filter); the more elements that are added to the set, the larger the probability of false positives. So I don't think they're the right choice for this problem, if you actually depend on the result "not here" vs "never was here" being right.
  - hinkley 8y ago
    You could keep it very tight. IME it’s the index overhead that kills (1000 times as much data makes inserts 10x slower), and ever since partial indexes became a thing that’s a bit less of a problem. Also deleting is far from free of side effects.
  - geezerjay 8y ago
    > Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed. The only info that needs to be stored is the ID of the finished task. If, say, the ID is expressed as a GUID then we're talking about 4 ints or 2 longs worth of data. Even so, these records are supposed to be garbage collected.
toyg 8y ago
I’d say after a period of tolerance (410) you switch to 404. If you care about historicals, move the record to a dedicated DW solution and use that to report.

delinka 8y ago

I'd say "gone" is a temporary state that exists shortly after task completion. A task shows as "gone" for, say, 15min, and then is deleted, producing a 404 in subsequent requests. On further thought, it seems to me that we necessarily need to keep completed tasks around until asked by a client to delete them. If a task produces an exception, we need to notate that for when the client next asks about the task's status. If the task ran successfully to completion, we need to notate that for when the client next asks. If the client needs to know all this, then it's the client who needs to ask for deletions as well - we shouldn't assume it's OK to remove records about tasks (unless there's a published policy that completed/error data will be purged after a certain period, e.g. 30, 60, 90 days...)

vfaronov 8y ago

https://tools.ietf.org/html/rfc7231#section-6.5.9 https://tools.ietf.org/html/rfc7231#section-6.5.9 > It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.

naasking 8y ago

> Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale... Why doesn't that scale?

aaaaaaaaaab 8y ago
Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed.
- DougWebb 8y ago
  If your IDs are completely unique, you could add them to a Bloom filter [0] when cleaning up the resource. Then you could use that to determine if an incoming ID has possibly been used before or definitely not. [0] https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter
  - jaggederest 8y ago
    Or you could just have your IDs be monotonically increasing and look at the minimum. Any task prior to that number is "410 Gone" by definition.
  - striking 8y ago
    Bloom filters are probabilistic, meaning they don't always produce the right result. From the article you posted: > False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed (though this can be addressed with a "counting" filter); the more elements that are added to the set, the larger the probability of false positives. So I don't think they're the right choice for this problem, if you actually depend on the result "not here" vs "never was here" being right.
- hinkley 8y ago
  You could keep it very tight. IME it’s the index overhead that kills (1000 times as much data makes inserts 10x slower), and ever since partial indexes became a thing that’s a bit less of a problem. Also deleting is far from free of side effects.
- geezerjay 8y ago
  > Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed. The only info that needs to be stored is the ID of the finished task. If, say, the ID is expressed as a GUID then we're talking about 4 ints or 2 longs worth of data. Even so, these records are supposed to be garbage collected.

toyg 8y ago

I’d say after a period of tolerance (410) you switch to 404. If you care about historicals, move the record to a dedicated DW solution and use that to report.