3 ms·

Not criticizing, but just curious what was the reasoning behind picking Redis rather than something like sqlite, which would keep the application a little bit m

Not criticizing, but just curious what was the reasoning behind picking Redis rather than something like sqlite, which would keep the application a little bit more self-contained so to speak?

eropple 8y ago
Hey, great question! SQLite is great for locally-stored databases with relatively light relational requirements, but I chose Redis as a first go-around in part because it's kind of "the standard" for this sort of job queue (Kue, Bull, Resque, Sidekiq, etc.) and also because it was the simplest network-friendly datastore that included the primitives I wanted to use and because its simple nature--when not in clustered mode, which is currently in "you're on your own" territory for TaskBotJS, but I expect that to improve--makes reasoning through operations simpler for me. As mentioned elsewhere, I've started work on a PostgreSQL-backed backend (which is actually easier than it sounds to get right, modern Postgres includes functionality to work with advisory-locked rows) and have done some noodling on a fully AWS-backed DynamoDB/SQS backend solution, too. The FoundationDB open-source announcement has me doing some reading, too, but I don't want to say much there just because I don't know enough about it yet to know whether it makes sense.
aclave1 8y ago
I am in no way affiliated with this project, but my assumption is that since this uses workers(separate servers) to process the background jobs - they could be on completely separate machines. If you use redis for job storage, they can all communicate without having to talk back to the master server.
- askmike 8y ago
  You can also do that with sqlite: have workers query for data and lock records when they are working on it. (people have been implementing job queues in SQL based environments for decades). The point of using sqlite is that it's not something you need to install and manage separately, or something that acts as a broker. (sqlite would be very slow as the workers need to poll for jobs).
  - eropple 8y ago
    Polling is not the worst thing in the world; when using reliable queueing and BRPOPLPUSH TaskBotJS does poll, and Redis handles it okay. SQLite is more just emphatically Not A Good Network Database, and the developers themselves say so.

Not criticizing, but just curious what was the reasoning behind picking Redis rather than something like sqlite, which would keep the application a little bit more self-contained so to speak?

eropple 8y ago

Hey, great question! SQLite is great for locally-stored databases with relatively light relational requirements, but I chose Redis as a first go-around in part because it's kind of "the standard" for this sort of job queue (Kue, Bull, Resque, Sidekiq, etc.) and also because it was the simplest network-friendly datastore that included the primitives I wanted to use and because its simple nature--when not in clustered mode, which is currently in "you're on your own" territory for TaskBotJS, but I expect that to improve--makes reasoning through operations simpler for me. As mentioned elsewhere, I've started work on a PostgreSQL-backed backend (which is actually easier than it sounds to get right, modern Postgres includes functionality to work with advisory-locked rows) and have done some noodling on a fully AWS-backed DynamoDB/SQS backend solution, too. The FoundationDB open-source announcement has me doing some reading, too, but I don't want to say much there just because I don't know enough about it yet to know whether it makes sense.

aclave1 8y ago

I am in no way affiliated with this project, but my assumption is that since this uses workers(separate servers) to process the background jobs - they could be on completely separate machines. If you use redis for job storage, they can all communicate without having to talk back to the master server.

askmike 8y ago
You can also do that with sqlite: have workers query for data and lock records when they are working on it. (people have been implementing job queues in SQL based environments for decades). The point of using sqlite is that it's not something you need to install and manage separately, or something that acts as a broker. (sqlite would be very slow as the workers need to poll for jobs).
- eropple 8y ago
  Polling is not the worst thing in the world; when using reliable queueing and BRPOPLPUSH TaskBotJS does poll, and Redis handles it okay. SQLite is more just emphatically Not A Good Network Database, and the developers themselves say so.