4 ms·

How hard would it be to use a PostgreSQL backend with sneakers?

jondot 13y ago
Not hard because most things are abstracted (though - there's a trade off between too much abstraction and peformance). But I feel it might be missing the point, RabbitMQ was used to remove the typical bottleneck from the broker which exists on most Ruby background frameworks, and using Postgres would bring it back. All in all, RabbitMQ will give you a good, transparent HA story with active-active which is useful when you must not lose messages (Postgres will give you just failover). Edit: to cover @artellectual's response - I think OP meant to swap RabbitMQ with Postgres as the backend but if OP meant "is it possible to use existing Rails models / etc" - then yes, @artellectual is completely right - easy to do.
- joevandyk 13y ago
  I meant using PostgreSQL as the backend. Benefits: * when you back up your main data store, your jobs are also backed up. If you use sidekiq, are you remembering to do frequent backups of redis? * only one data store, not multiple. Simpler architecture. * jobs can be inserted/updated in the same transactions as the rest of your data * PostgreSQL supports listen/notify, don't have to poll for new jobs * can use SQL to query jobs * jobs can have foreign key constraints to the rest of your data
  - jondot 13y ago
    I understand. You're completely right about Sidekiq. For RabbitMQ - you can have jobs persisted to disk on all nodes in the cluster which is like a backup. And of course transactional messaging is a problem that needs special attention.
  - zimbatm 13y ago
    It doesn't hurt to keep a higher-level job representation in DB and sync the state at each step of the processing. I don't know about sneakers but you might want to have more states than the usual new/processing/success|fail. Obviously it really depends on what you're doing, if the jobs are re-entrant, if they're the fire-and-forget kind of types, ... Still in most of the cases I would just store the jobs as a table entry and pass the table+id to the job processor. Even if PostgreSQL was the primary data store for the job processor there would still be issues with external state that needs to be healed when restoring from backup.
artellectual 13y ago
I don't see why you can't just include ActiveRecord or datamapper and connect to the same DB as your app (assuming you want to allow your workers to have access to the application database) you might have to copy the models over though. But then if you want to keep it simple you might not need to copy your models over, since models for workers are usually much simpler than the application's model. This is of course talking about the stand-alone version. And I am talking of course about using it in tandem with the rabbitmq for job storage.

Not hard because most things are abstracted (though - there's a trade off between too much abstraction and peformance). But I feel it might be missing the point, RabbitMQ was used to remove the typical bottleneck from the broker which exists on most Ruby background frameworks, and using Postgres would bring it back. All in all, RabbitMQ will give you a good, transparent HA story with active-active which is useful when you must not lose messages (Postgres will give you just failover). Edit: to cover @artellectual's response - I think OP meant to swap RabbitMQ with Postgres as the backend but if OP meant "is it possible to use existing Rails models / etc" - then yes, @artellectual is completely right - easy to do.

joevandyk 13y ago
I meant using PostgreSQL as the backend. Benefits: * when you back up your main data store, your jobs are also backed up. If you use sidekiq, are you remembering to do frequent backups of redis? * only one data store, not multiple. Simpler architecture. * jobs can be inserted/updated in the same transactions as the rest of your data * PostgreSQL supports listen/notify, don't have to poll for new jobs * can use SQL to query jobs * jobs can have foreign key constraints to the rest of your data
- jondot 13y ago
  I understand. You're completely right about Sidekiq. For RabbitMQ - you can have jobs persisted to disk on all nodes in the cluster which is like a backup. And of course transactional messaging is a problem that needs special attention.
- zimbatm 13y ago
  It doesn't hurt to keep a higher-level job representation in DB and sync the state at each step of the processing. I don't know about sneakers but you might want to have more states than the usual new/processing/success|fail. Obviously it really depends on what you're doing, if the jobs are re-entrant, if they're the fire-and-forget kind of types, ... Still in most of the cases I would just store the jobs as a table entry and pass the table+id to the job processor. Even if PostgreSQL was the primary data store for the job processor there would still be issues with external state that needs to be healed when restoring from backup.

artellectual 13y ago

I don't see why you can't just include ActiveRecord or datamapper and connect to the same DB as your app (assuming you want to allow your workers to have access to the application database) you might have to copy the models over though. But then if you want to keep it simple you might not need to copy your models over, since models for workers are usually much simpler than the application's model. This is of course talking about the stand-alone version. And I am talking of course about using it in tandem with the rabbitmq for job storage.