3 ms·

I've built quite a few apps that use queueing to do the heavy processing behind the scenes, and I've always used a polling system to check progress. But it alwa

by dchuk 13y ago

I've built quite a few apps that use queueing to do the heavy processing behind the scenes, and I've always used a polling system to check progress. But it always has felt like a half-baked solution. Are there any patterns or documentation projects or books or tutorials or anything else out there that cover how to have users submit something into a queue system, provide immediate feedback, then provide accurate progress updating as the job is processed, then show the results, all without having the user refresh their process or navigate away and back again? Let's say for instance, I built a service that lets a user upload 20 images at a time to be cropped and resized. Is there a commonly accepted pattern for alerting them of the resizing job progress as it is happening?

oxtopus 13y ago
Assuming your workers can periodically report on progress, Socket.IO may fit the bill, has multiple implementations and provides graceful degradation for older browsers.
habitue 13y ago
For users in a browser, use websockets for modern browsers, and long polling for older browsers (Socket.IO wraps up these solutions nicely) For sending async results to other servers, use webhooks. They give you an endpoint to call when the job is done processing and you call it when you're done.
- dchuk 13y ago
  But that still doesn't really solve the whole progress reporting part of it. As per my example, the "job" would really be 20 individual cropping/resizing jobs. In order to know that the whole set of jobs is complete, there needs to be a checking job or something similar that looks to see out of the total number of jobs started, if all are now completed, and if so, tell the user. Is there a pattern for that type of situation? Some sort of watcher pattern or master job monitoring pattern that allows accurate progress reporting?
  - habitue 13y ago
    Send progress events over your push channel. Push an event every 1% or 10% or whatever you need depending on what you need for your experience. If you're talking about generic progress reporting from a tool that doesn't know what the underlying job consists of (i.e. it only knows "started" "done" "errored"), then I'm not aware of any pattern to provide an accurate progress meter. You could perhaps keep statistics on the length of past runs (like some CI servers do for build times), but it would really depend on your application whether users are willing to accept that those progress bars are just a guess and sometimes run over.

by dchuk 13y ago

oxtopus 13y ago

Assuming your workers can periodically report on progress, Socket.IO may fit the bill, has multiple implementations and provides graceful degradation for older browsers.

habitue 13y ago

For users in a browser, use websockets for modern browsers, and long polling for older browsers (Socket.IO wraps up these solutions nicely) For sending async results to other servers, use webhooks. They give you an endpoint to call when the job is done processing and you call it when you're done.

dchuk 13y ago
But that still doesn't really solve the whole progress reporting part of it. As per my example, the "job" would really be 20 individual cropping/resizing jobs. In order to know that the whole set of jobs is complete, there needs to be a checking job or something similar that looks to see out of the total number of jobs started, if all are now completed, and if so, tell the user. Is there a pattern for that type of situation? Some sort of watcher pattern or master job monitoring pattern that allows accurate progress reporting?
- habitue 13y ago
  Send progress events over your push channel. Push an event every 1% or 10% or whatever you need depending on what you need for your experience. If you're talking about generic progress reporting from a tool that doesn't know what the underlying job consists of (i.e. it only knows "started" "done" "errored"), then I'm not aware of any pattern to provide an accurate progress meter. You could perhaps keep statistics on the length of past runs (like some CI servers do for build times), but it would really depend on your application whether users are willing to accept that those progress bars are just a guess and sometimes run over.