4 ms·

Not the GP, but the general principal is to use Redis as a task queueing system. The front-end puts a task into a redis queue. One or more C++ programs are wa

by jbester 15y ago

Not the GP, but the general principal is to use Redis as a task queueing system. The front-end puts a task into a redis queue. One or more C++ programs are waiting on the queue(s) and execute the given task (like a large DB insert). If results are needed they can be communicated back to the front-end. The front-end can poll for the result or use pub/sub messaging. This gets you a number of benefits: separation of the front-end logic and the back-end logic, better scalability - there may be a bunch of workers on distributed among different machines, and security - the C++ programs aren't as worried about unvalidated input since their input comes from the front-end.

tptacek 15y ago
The Redis interface is also so simple that it's very easy to hook up C code to it, and Redis is somewhat "typed", which reduces the amount of parsing you have to do.
mikecsh 15y ago
Can your recommend any particularly good resources/tutorials or further information on this? Thanks!
- sausagefeet 15y ago
  The web app I have been working on has the same architecture. 1. A request comes in 2. Request handler parses the request 3. Handler determines which Queue the request should go into based off the URL 4. Request handler queues the request as well as how whoever handles it can get back to them 5. Waits for response There are then multiple workers living possibly on other machines listening on the queue. They handle the request and return a response to the original request handler and pull the next bit of work off the queue. I like this because I feel like it is rather robust. I use a STOMP message queue which is very trivial to hook up other languages to. It is fast enough for my needs. It lets me do simple things like specify how many queued items a handler can handle concurrently. My web app is then broken into components that each run independently. They can run in the same process or be split into separate processes or even across computers. My web app is not particularly high demand but we run it on fairly light resources so the queuing also keeps our app from becoming overwhelmed if a lot of requests happen at once. They just get queued and a handler will get to it when it can.

by jbester 15y ago

tptacek 15y ago

The Redis interface is also so simple that it's very easy to hook up C code to it, and Redis is somewhat "typed", which reduces the amount of parsing you have to do.

mikecsh 15y ago

Can your recommend any particularly good resources/tutorials or further information on this? Thanks!

sausagefeet 15y ago
The web app I have been working on has the same architecture. 1. A request comes in 2. Request handler parses the request 3. Handler determines which Queue the request should go into based off the URL 4. Request handler queues the request as well as how whoever handles it can get back to them 5. Waits for response There are then multiple workers living possibly on other machines listening on the queue. They handle the request and return a response to the original request handler and pull the next bit of work off the queue. I like this because I feel like it is rather robust. I use a STOMP message queue which is very trivial to hook up other languages to. It is fast enough for my needs. It lets me do simple things like specify how many queued items a handler can handle concurrently. My web app is then broken into components that each run independently. They can run in the same process or be split into separate processes or even across computers. My web app is not particularly high demand but we run it on fairly light resources so the queuing also keeps our app from becoming overwhelmed if a lot of requests happen at once. They just get queued and a handler will get to it when it can.