15 ms·

REST and long-running jobs (2014)

98 points by geezerjay 8y ago

aaaaaaaaaab 8y ago
“Once the originally desired resource is created, there are two alternative ways to deal with the temporary task resource: API client must issue DELETE request, so that server purges it. Until then, server responds with 303 See Other status. Once deleted, 404 Not Found will be returned for subsequent GET /queue/12345 requests. or, garbage collection can be a server’s job to do: once task is complete server can safely remove it and respond with the 410 Gone on subsequent GET /queue/12345 requests.” How do you distinguish between an id referring to a finished task that no longer exists (410 Gone), and an id that never referred to any task (404 Not Found)? Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale...
- delinka 8y ago
  I'd say "gone" is a temporary state that exists shortly after task completion. A task shows as "gone" for, say, 15min, and then is deleted, producing a 404 in subsequent requests. On further thought, it seems to me that we necessarily need to keep completed tasks around until asked by a client to delete them. If a task produces an exception, we need to notate that for when the client next asks about the task's status. If the task ran successfully to completion, we need to notate that for when the client next asks. If the client needs to know all this, then it's the client who needs to ask for deletions as well - we shouldn't assume it's OK to remove records about tasks (unless there's a published policy that completed/error data will be purged after a certain period, e.g. 30, 60, 90 days...)
- vfaronov 8y ago
  https://tools.ietf.org/html/rfc7231#section-6.5.9 https://tools.ietf.org/html/rfc7231#section-6.5.9 > It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.
- naasking 8y ago
  > Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale... Why doesn't that scale?
  - aaaaaaaaaab 8y ago
    Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed.
    - DougWebb 8y ago
      If your IDs are completely unique, you could add them to a Bloom filter [0] when cleaning up the resource. Then you could use that to determine if an incoming ID has possibly been used before or definitely not. [0] https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter
      - jaggederest 8y ago
        Or you could just have your IDs be monotonically increasing and look at the minimum. Any task prior to that number is "410 Gone" by definition.
      - striking 8y ago
        Bloom filters are probabilistic, meaning they don't always produce the right result. From the article you posted: > False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed (though this can be addressed with a "counting" filter); the more elements that are added to the set, the larger the probability of false positives. So I don't think they're the right choice for this problem, if you actually depend on the result "not here" vs "never was here" being right.
    - hinkley 8y ago
      You could keep it very tight. IME it’s the index overhead that kills (1000 times as much data makes inserts 10x slower), and ever since partial indexes became a thing that’s a bit less of a problem. Also deleting is far from free of side effects.
- toyg 8y ago
  I’d say after a period of tolerance (410) you switch to 404. If you care about historicals, move the record to a dedicated DW solution and use that to report.
wilgertvelinga 8y ago
Anybody here who used this pattern in practice and found out it is not as good as this article portrays it?
- politelemon 8y ago
  These patterns are geospecific. They are only known to work within the vicinity of the town of Theory. You may find several deviations and compromises that occur outside this region, though the intention is often to emulate a loose collection of (sometimes) consensus based best practices.
  - icebraining 8y ago
    They work just fine for millions of services; they're called "websites". When you click on a link/button that starts a long running task, you often get redirected to a status page which refreshes (polling), and which has a link to cancel.
- skinnyarms 8y ago
  It sounds pretty sensible to me, I filed it away for next time I need to do something like this.
- gravypod 8y ago
  This layout is similar to something I've done in production. In my use case we don't support stopping of jobs and I return job completion status in aggregate. I also avoid confusion in status codes and implementation by separating my resources for requests of a resource and the resource itself into different endpoints. POST /<resource>-requests { ... } GET /<resouce>-requests [ { "id": <id>, "status": "complete", ... "url": "/<resource>/<id>" } It all comes down to your queue processing system and making sure it's sensibly configured. For long running jobs (1+hr) the stack we use at work (Laravel/Lumen) come very poorly configured.
- twodave 8y ago
  Bottom line is if you’re doing this via REST and not using web sockets (which is fine, of course!) then you’re going to end up in a polling situation. That’s the important thing to think about. I had a bulk export job that ran similar to this, minus a lot of the frills. On the initial POST you get back a token that would allow you to poll the job status. Until you went to download the (often several GB) payload, all requests were extremely fast. At most we would insert a single record into a DB and queue a tiny object onto the message bus. In the status poll it was a simple key lookup. In my experience nobody seriously tries to figure out how to use your API by calling it, they’ll prefer to look at your code examples and documentation. Nobody is going to build a generic-enough REST client to make all the HATEOAS crap matter for their workflow. In my view it’s a waste of time, but maybe others have found it measurably useful?
vfaronov 8y ago
RFC 7540 also defines an optional way for the client to signal whether it wishes the request to be processed asynchronously: https://tools.ietf.org/html/rfc7240#section-4.1 https://tools.ietf.org/html/rfc7240#section-4.1
- geezerjay 8y ago
  My first impression is that RFC7540 isn't appropriate for this use case, because it is proposed as a way for the client to signal optional/prefered behavior when said behavior is already made mandatory by the server. To put it differently, why would it matter if a client POSTs with Prefer: respond-async if the server response will always be async? Whether that header is present or not, the HTTP response is already designed to always return a Location header.
sltkr 8y ago
This article is missing one detail: what if resource creation fails? How do you signal that failure when the client calls GET on the task queue resource, in a way that's distinguishable from a failure to retrieve the task queue status?
- Lazare 8y ago
  You can't map everything cleanly to HTTP status codes. When you implement the task queue resource, you'll need to come up with ways of representing what information is important to you (status, success, failure, eta, progress, etc.), and then when you implement the client you'll have to parse that.
- icebraining 8y ago
  That's like asking whether you should send a 404 when you serve an article about a city that no longer exists. HTTP status codes refer to the resource being requested (in this case, the creation task itself), not to something that resource refers.
- time0ut 8y ago
  I think it depends a lot on why the operation failed and what you need the client to do next. If it is something the client can fix, I think it is reasonable to respond with the appropriate 4xx series status and enough to detail that they can correct and proceed. Same with 5xx series and issues that are out of the client's control. There are other approaches that are more or less as valid. The important thing is that we return enough information for the client to proceed. Wether that is signalled through status codes, headers, or response body isn't as important in my opinion.
- geezerjay 8y ago
  > This article is missing one detail: what if resource creation fails? I didn't get your point. If the resource creation fails during the request then either a 4 or 5 status is returned. If the resource creation succeeds but the long-running process fails in some way, either during processing or at creation time, then the resource representing the long-running process will be updated by the server to inform the client of the process' current status.
Multicomp 8y ago
Someone once gave a talk on 'the Art of API design' that covered logical design of a RESTful API and one of their points was in close alignment with the article. Can't find and slides to share but will edit/reply to self if I can, the talk was very informative and covered not just resource creation but destruction, Discovery, and even a bit about authentication.
epaulson 8y ago
One thing this piece leaves out is that if the operation/creation is in fact expensive, you want to make sure you have some kind of two-phase option so you don't inadvertently create multiple copies. If the client crashes after the POST to create 'death star' succeeds and hence construction starts, but before it can read the response of the POST in the example and so never gets the pointer to the /queue/12345 from the response, if the client reissues the POST in the server should reject a second attempt to create a star named 'death star' just and give back the same pointer to /queue/12345. Or, if you want to be able to have two stars named 'death star', you should first POST to some endpoint to get a unique identifier from the server first, and then use that to kick off the creation of your death star so the client can reason about the current state.
- Anderkent 8y ago
  Or you could just generate an unique identifier on the client and save the roundtrip? Uuids aren't really expensive.
  - farazbabar 8y ago
    UUID is not enough to prevent duplicate resource creation. Consider the following steps: 1. Receive a request with uuid, 2. Persist the fact that system received this request, 3. who or what processes it? Even if you are working off atomic persistence, the resource creation process (the kick-off process) needs to maintain the state that it plucked one off the queue or table, atomically update the state in persistence as it creates the resource somewhere else and log the successful creation of this resource as yet another entry in the persistence store. This way, requests that exceed their SLA can be retried and the downstream systems must act with correct idempotent behavior in order for all of this to work correctly in the presence of failures. It is all very technically possible, but it goes beyond just a UUID (in other words, UUID is necessary but not sufficient).
    - Anderkent 8y ago
      The issues you list don't really seem to be relevant in regard to duplicate resource creation? They're just about handling creating the resource in the first place. As long as the uuid is persisted you can redirect all attempts to recreate that id to the right job/queue/whatever actually backs resources.
whack 8y ago
> Your first instinct might be: “What if I return HTTP 201 Created immediately, but defer the actual creation to some later point?“... Well, you can’t do that. If you do, you will be violating the HTTP/1.1: Semantics and Content protocol > We know what status code to return, what about Location header? Simple. Instead of the location to the actual resource (/stars/97865), API will return location of the queued task that got created (/queue/12345) > How to learn when resource is finally available? You need to query the queued task... Once resource is created, API should respond with 303 See Other status code on all the subsequent requests to the queued task... There are two alternative ways to deal with the temporary task resource: API client must issue DELETE request, so that server purges it. or, garbage collection can be a server’s job to do I've always wondered about how important it is to follow REST conventions, at the cost of simplicity, when you control both the server and client software. In the example given above, the recommendation to use a 202 instead of 201, sounds perfectly reasonable. However, creating and returning to the client a new queue resource, which returns a 303 when complete, and the client then has to delete... Is it really so bad to just return /stars/98765 immediately? And when queried, have the relevant GET API return a response indicating that construction of the resource is still in progress? If the client needs to wait for construction to complete before doing something else, can't it just poll /stars/98765 instead of /queue/12345? More generally speaking, if you're designing an API for use by your own client, and you're facing a tradeoff between interface/implementation simplicity vs following the official REST protocol, is it really worth choosing the latter?
- diminoten 8y ago
  You may control the client, but do you control the client library? If you want to get the most out of something like, say, Python's requests, following these conventions are your best bet.
  - vfaronov 8y ago
    Case in point: CacheControl [1]. Plug it in and you get flexible caching — interoperable with NGINX, the browser, and everybody else — for very little. But it relies on the protocol. If you respond to GET /stars/98765 with 200 (OK) and an “under construction” placeholder, it’s going to get confused. [1] https://github.com/ionrock/cachecontrol https://github.com/ionrock/cachecontrol
coredog64 8y ago
A coworker and I were discussing this very thing on Friday. He and I exhausted our Google-fu without success: Does anyone know of a framework/library that implements this? Don’t really care about the language, but per other comments we’re interested in the practical lessons learned that would show up in code.
anuraj 8y ago
Better options would be: 1. Send partial HTTP response (206) with progress details which may be handled using AJAX progress API at the client before sending final response. This is processing and network intensive but may be needed in certain scenarios like drawing graphics etc. You cannot recover from connection loss. 2. Handover the task to a job scheduler and return the jobid. Client can then poll jobid using a status API or get the status pushed using a webhook. This is the more optimal approach for majority of cases.
- wrs 8y ago
  #2 is essentially what the article describes.
- zbjornson 8y ago
  206 is only valid to respond to range requests, which can only be GET requests.
  - geezerjay 8y ago
    Additionally, the 206 suggestion fails to take into account basic use cases such as having more than one queued job.
- korijn 8y ago
  Number 2 is super easy to implement with libraries like celery.
- geezerjay 8y ago
  > 2. Handover the task to a job scheduler and return the jobid. That's the approach described in the article. Where does your suggestion differ from it?
bsenftner 8y ago
This author misses a key algorithm design option: the API does NOT offer a CREATE on such a resource, the API offers a CREATE on a JOB which generates the long-time-to-create resource. So the API client simply creates the Job, receives an ID immediately, which can be queried for status, forever. Long after the job completed it serves as a timestamped record of that specific resource's creation. It is only after the Job status reaches predefined status levels are the assets (files/data) associated with it available. I've used this pattern with success for over a decade. First with on-demand digital products, which required compiling/generating time after customer request. This pattern also supports partial generation of long-time-to-generate resources. I used it with a custom game avatar service, where various modification and customization options are visualized without creating them. The visualization of the modified product is a 'milestone' asset generated by the resource creation pipeline. If the end-user chooses not to purchase said modification, that specific asset's sub-job is killed or never started. With complex, multi-layered digital products this pattern works quite well. Even after a period of time, the end-user could return to a multi-leveled job and request an asset they previously denied. That sub-job gets timestamped just the same as if it ran originally. It just works.
- hnarayanan 8y ago
  This. I believe this is the most straightforward and robust way to handle this scenario.
- geezerjay 8y ago
  >This author misses a key algorithm design option: the API does NOT offer a CREATE on such a resource, the API offers a CREATE on a JOB which generates the long-time-to-create resource. So the API client simply creates the Job, receives an ID immediately, which can be queried for status, forever. You should read the article because that's the approach it describes.
wcoenen 8y ago
Instead you could also POST to /star-construction to get back a link to a /star-construction/foo resource, representing a long-running creation process. This resource could be polled to get progress updates, and eventually a link to the created star resource, or perhaps an error describing why it didn't work. The advantage is that you don't have to deal with two different types of resources being returned at the same URL, which complicates parsing the response. If creations are queued you also can get a view on the workload still in front of you.
shoo 8y ago
I found this discussion a bit incomplete as it didn't really discuss the consequences or costs or benefits of modelling things one way or another. Suppose we didn't use REST or HTTP for our API. What would be a good way to model this kind of thing?
- realthing 8y ago
  I suppose one way of modeling a solution to this space is to look back at goold 'ol fashioned RPC's
- jmts 8y ago
  This is essentially just an asynchronous response to some request. I expect there are probably three cases: 1. Response is unimportant: do nothing 2. Response timing is not important: client polls 3. Response timing is important: server calls back into the client somehow The article describes an implementation of point 2. Depending on the technologies available at the time, I'd argue 3 is probably more versatile. Both cases 1 and 2, and the blocking case can be implemented with case 3, if the available technology allows it.

98 points by geezerjay 8y ago

“Once the originally desired resource is created, there are two alternative ways to deal with the temporary task resource: API client must issue DELETE request, so that server purges it. Until then, server responds with 303 See Other status. Once deleted, 404 Not Found will be returned for subsequent GET /queue/12345 requests. or, garbage collection can be a server’s job to do: once task is complete server can safely remove it and respond with the 410 Gone on subsequent GET /queue/12345 requests.” How do you distinguish between an id referring to a finished task that no longer exists (410 Gone), and an id that never referred to any task (404 Not Found)? Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale...

delinka 8y ago
I'd say "gone" is a temporary state that exists shortly after task completion. A task shows as "gone" for, say, 15min, and then is deleted, producing a 404 in subsequent requests. On further thought, it seems to me that we necessarily need to keep completed tasks around until asked by a client to delete them. If a task produces an exception, we need to notate that for when the client next asks about the task's status. If the task ran successfully to completion, we need to notate that for when the client next asks. If the client needs to know all this, then it's the client who needs to ask for deletions as well - we shouldn't assume it's OK to remove records about tasks (unless there's a published policy that completed/error data will be purged after a certain period, e.g. 30, 60, 90 days...)
vfaronov 8y ago
https://tools.ietf.org/html/rfc7231#section-6.5.9 https://tools.ietf.org/html/rfc7231#section-6.5.9 > It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.
naasking 8y ago
> Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale... Why doesn't that scale?
- aaaaaaaaaab 8y ago
  Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed.
  - DougWebb 8y ago
    If your IDs are completely unique, you could add them to a Bloom filter [0] when cleaning up the resource. Then you could use that to determine if an incoming ID has possibly been used before or definitely not. [0] https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter
    - jaggederest 8y ago
      Or you could just have your IDs be monotonically increasing and look at the minimum. Any task prior to that number is "410 Gone" by definition.
    - striking 8y ago
      Bloom filters are probabilistic, meaning they don't always produce the right result. From the article you posted: > False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed (though this can be addressed with a "counting" filter); the more elements that are added to the set, the larger the probability of false positives. So I don't think they're the right choice for this problem, if you actually depend on the result "not here" vs "never was here" being right.
  - hinkley 8y ago
    You could keep it very tight. IME it’s the index overhead that kills (1000 times as much data makes inserts 10x slower), and ever since partial indexes became a thing that’s a bit less of a problem. Also deleting is far from free of side effects.
toyg 8y ago
I’d say after a period of tolerance (410) you switch to 404. If you care about historicals, move the record to a dedicated DW solution and use that to report.

wilgertvelinga 8y ago

Anybody here who used this pattern in practice and found out it is not as good as this article portrays it?

politelemon 8y ago
These patterns are geospecific. They are only known to work within the vicinity of the town of Theory. You may find several deviations and compromises that occur outside this region, though the intention is often to emulate a loose collection of (sometimes) consensus based best practices.
- icebraining 8y ago
  They work just fine for millions of services; they're called "websites". When you click on a link/button that starts a long running task, you often get redirected to a status page which refreshes (polling), and which has a link to cancel.
skinnyarms 8y ago
It sounds pretty sensible to me, I filed it away for next time I need to do something like this.
gravypod 8y ago
This layout is similar to something I've done in production. In my use case we don't support stopping of jobs and I return job completion status in aggregate. I also avoid confusion in status codes and implementation by separating my resources for requests of a resource and the resource itself into different endpoints. POST /<resource>-requests { ... } GET /<resouce>-requests [ { "id": <id>, "status": "complete", ... "url": "/<resource>/<id>" } It all comes down to your queue processing system and making sure it's sensibly configured. For long running jobs (1+hr) the stack we use at work (Laravel/Lumen) come very poorly configured.
twodave 8y ago
Bottom line is if you’re doing this via REST and not using web sockets (which is fine, of course!) then you’re going to end up in a polling situation. That’s the important thing to think about. I had a bulk export job that ran similar to this, minus a lot of the frills. On the initial POST you get back a token that would allow you to poll the job status. Until you went to download the (often several GB) payload, all requests were extremely fast. At most we would insert a single record into a DB and queue a tiny object onto the message bus. In the status poll it was a simple key lookup. In my experience nobody seriously tries to figure out how to use your API by calling it, they’ll prefer to look at your code examples and documentation. Nobody is going to build a generic-enough REST client to make all the HATEOAS crap matter for their workflow. In my view it’s a waste of time, but maybe others have found it measurably useful?

vfaronov 8y ago

RFC 7540 also defines an optional way for the client to signal whether it wishes the request to be processed asynchronously: https://tools.ietf.org/html/rfc7240#section-4.1 https://tools.ietf.org/html/rfc7240#section-4.1

geezerjay 8y ago
My first impression is that RFC7540 isn't appropriate for this use case, because it is proposed as a way for the client to signal optional/prefered behavior when said behavior is already made mandatory by the server. To put it differently, why would it matter if a client POSTs with Prefer: respond-async if the server response will always be async? Whether that header is present or not, the HTTP response is already designed to always return a Location header.

sltkr 8y ago

This article is missing one detail: what if resource creation fails? How do you signal that failure when the client calls GET on the task queue resource, in a way that's distinguishable from a failure to retrieve the task queue status?

Lazare 8y ago
You can't map everything cleanly to HTTP status codes. When you implement the task queue resource, you'll need to come up with ways of representing what information is important to you (status, success, failure, eta, progress, etc.), and then when you implement the client you'll have to parse that.
icebraining 8y ago
That's like asking whether you should send a 404 when you serve an article about a city that no longer exists. HTTP status codes refer to the resource being requested (in this case, the creation task itself), not to something that resource refers.
time0ut 8y ago
I think it depends a lot on why the operation failed and what you need the client to do next. If it is something the client can fix, I think it is reasonable to respond with the appropriate 4xx series status and enough to detail that they can correct and proceed. Same with 5xx series and issues that are out of the client's control. There are other approaches that are more or less as valid. The important thing is that we return enough information for the client to proceed. Wether that is signalled through status codes, headers, or response body isn't as important in my opinion.
geezerjay 8y ago
> This article is missing one detail: what if resource creation fails? I didn't get your point. If the resource creation fails during the request then either a 4 or 5 status is returned. If the resource creation succeeds but the long-running process fails in some way, either during processing or at creation time, then the resource representing the long-running process will be updated by the server to inform the client of the process' current status.

Multicomp 8y ago

Someone once gave a talk on 'the Art of API design' that covered logical design of a RESTful API and one of their points was in close alignment with the article. Can't find and slides to share but will edit/reply to self if I can, the talk was very informative and covered not just resource creation but destruction, Discovery, and even a bit about authentication.

epaulson 8y ago

One thing this piece leaves out is that if the operation/creation is in fact expensive, you want to make sure you have some kind of two-phase option so you don't inadvertently create multiple copies. If the client crashes after the POST to create 'death star' succeeds and hence construction starts, but before it can read the response of the POST in the example and so never gets the pointer to the /queue/12345 from the response, if the client reissues the POST in the server should reject a second attempt to create a star named 'death star' just and give back the same pointer to /queue/12345. Or, if you want to be able to have two stars named 'death star', you should first POST to some endpoint to get a unique identifier from the server first, and then use that to kick off the creation of your death star so the client can reason about the current state.

Anderkent 8y ago
Or you could just generate an unique identifier on the client and save the roundtrip? Uuids aren't really expensive.
- farazbabar 8y ago
  UUID is not enough to prevent duplicate resource creation. Consider the following steps: 1. Receive a request with uuid, 2. Persist the fact that system received this request, 3. who or what processes it? Even if you are working off atomic persistence, the resource creation process (the kick-off process) needs to maintain the state that it plucked one off the queue or table, atomically update the state in persistence as it creates the resource somewhere else and log the successful creation of this resource as yet another entry in the persistence store. This way, requests that exceed their SLA can be retried and the downstream systems must act with correct idempotent behavior in order for all of this to work correctly in the presence of failures. It is all very technically possible, but it goes beyond just a UUID (in other words, UUID is necessary but not sufficient).
  - Anderkent 8y ago
    The issues you list don't really seem to be relevant in regard to duplicate resource creation? They're just about handling creating the resource in the first place. As long as the uuid is persisted you can redirect all attempts to recreate that id to the right job/queue/whatever actually backs resources.

whack 8y ago

> Your first instinct might be: “What if I return HTTP 201 Created immediately, but defer the actual creation to some later point?“... Well, you can’t do that. If you do, you will be violating the HTTP/1.1: Semantics and Content protocol > We know what status code to return, what about Location header? Simple. Instead of the location to the actual resource (/stars/97865), API will return location of the queued task that got created (/queue/12345) > How to learn when resource is finally available? You need to query the queued task... Once resource is created, API should respond with 303 See Other status code on all the subsequent requests to the queued task... There are two alternative ways to deal with the temporary task resource: API client must issue DELETE request, so that server purges it. or, garbage collection can be a server’s job to do I've always wondered about how important it is to follow REST conventions, at the cost of simplicity, when you control both the server and client software. In the example given above, the recommendation to use a 202 instead of 201, sounds perfectly reasonable. However, creating and returning to the client a new queue resource, which returns a 303 when complete, and the client then has to delete... Is it really so bad to just return /stars/98765 immediately? And when queried, have the relevant GET API return a response indicating that construction of the resource is still in progress? If the client needs to wait for construction to complete before doing something else, can't it just poll /stars/98765 instead of /queue/12345? More generally speaking, if you're designing an API for use by your own client, and you're facing a tradeoff between interface/implementation simplicity vs following the official REST protocol, is it really worth choosing the latter?

diminoten 8y ago
You may control the client, but do you control the client library? If you want to get the most out of something like, say, Python's requests, following these conventions are your best bet.
- vfaronov 8y ago
  Case in point: CacheControl [1]. Plug it in and you get flexible caching — interoperable with NGINX, the browser, and everybody else — for very little. But it relies on the protocol. If you respond to GET /stars/98765 with 200 (OK) and an “under construction” placeholder, it’s going to get confused. [1] https://github.com/ionrock/cachecontrol https://github.com/ionrock/cachecontrol

coredog64 8y ago

A coworker and I were discussing this very thing on Friday. He and I exhausted our Google-fu without success: Does anyone know of a framework/library that implements this? Don’t really care about the language, but per other comments we’re interested in the practical lessons learned that would show up in code.

anuraj 8y ago

Better options would be: 1. Send partial HTTP response (206) with progress details which may be handled using AJAX progress API at the client before sending final response. This is processing and network intensive but may be needed in certain scenarios like drawing graphics etc. You cannot recover from connection loss. 2. Handover the task to a job scheduler and return the jobid. Client can then poll jobid using a status API or get the status pushed using a webhook. This is the more optimal approach for majority of cases.

wrs 8y ago
#2 is essentially what the article describes.
zbjornson 8y ago
206 is only valid to respond to range requests, which can only be GET requests.
- geezerjay 8y ago
  Additionally, the 206 suggestion fails to take into account basic use cases such as having more than one queued job.
korijn 8y ago
Number 2 is super easy to implement with libraries like celery.
geezerjay 8y ago
> 2. Handover the task to a job scheduler and return the jobid. That's the approach described in the article. Where does your suggestion differ from it?

bsenftner 8y ago

This author misses a key algorithm design option: the API does NOT offer a CREATE on such a resource, the API offers a CREATE on a JOB which generates the long-time-to-create resource. So the API client simply creates the Job, receives an ID immediately, which can be queried for status, forever. Long after the job completed it serves as a timestamped record of that specific resource's creation. It is only after the Job status reaches predefined status levels are the assets (files/data) associated with it available. I've used this pattern with success for over a decade. First with on-demand digital products, which required compiling/generating time after customer request. This pattern also supports partial generation of long-time-to-generate resources. I used it with a custom game avatar service, where various modification and customization options are visualized without creating them. The visualization of the modified product is a 'milestone' asset generated by the resource creation pipeline. If the end-user chooses not to purchase said modification, that specific asset's sub-job is killed or never started. With complex, multi-layered digital products this pattern works quite well. Even after a period of time, the end-user could return to a multi-leveled job and request an asset they previously denied. That sub-job gets timestamped just the same as if it ran originally. It just works.

hnarayanan 8y ago
This. I believe this is the most straightforward and robust way to handle this scenario.
geezerjay 8y ago
>This author misses a key algorithm design option: the API does NOT offer a CREATE on such a resource, the API offers a CREATE on a JOB which generates the long-time-to-create resource. So the API client simply creates the Job, receives an ID immediately, which can be queried for status, forever. You should read the article because that's the approach it describes.

wcoenen 8y ago

Instead you could also POST to /star-construction to get back a link to a /star-construction/foo resource, representing a long-running creation process. This resource could be polled to get progress updates, and eventually a link to the created star resource, or perhaps an error describing why it didn't work. The advantage is that you don't have to deal with two different types of resources being returned at the same URL, which complicates parsing the response. If creations are queued you also can get a view on the workload still in front of you.

shoo 8y ago

I found this discussion a bit incomplete as it didn't really discuss the consequences or costs or benefits of modelling things one way or another. Suppose we didn't use REST or HTTP for our API. What would be a good way to model this kind of thing?

realthing 8y ago
I suppose one way of modeling a solution to this space is to look back at goold 'ol fashioned RPC's
jmts 8y ago
This is essentially just an asynchronous response to some request. I expect there are probably three cases: 1. Response is unimportant: do nothing 2. Response timing is not important: client polls 3. Response timing is important: server calls back into the client somehow The article describes an implementation of point 2. Depending on the technologies available at the time, I'd argue 3 is probably more versatile. Both cases 1 and 2, and the blocking case can be implemented with case 3, if the available technology allows it.