4 ms·

Ok, I looked up your project and it seems to be a job scheduling system for Ruby. Kudos to you, but I had a look at the feature table, and the "multi-process" f

Ok, I looked up your project and it seems to be a job scheduling system for Ruby. Kudos to you, but I had a look at the feature table, and the "multi-process" feature is only available in the paid edition. IMHO this is the main problem with open-core and it is a really tough decision you have to make - is multi-process a standard feature or not? Because ultimately I would have guessed it is a fundamental feature of a job scheduling system (assuming long-running tasks) and one could say it is even at the core of a scheduling system. Don't take this the wrong way, I applaud everybody who finds a way to develop meaningful FOSS (and a reliable working background job scheduling system is not a walk in the park) and getting paid for it, so good luck with it!

mperham 10y ago
Good questions. Couple of things to note: 1. My paid features are all built on the core APIs that are open source. Almost every feature I sell has an OSS variant that someone has written. If you want to plug in a dozen different OSS libraries, integrate and test them together, you can get most of what I sell for $0. You will get varying levels of support and yearly upgrades of that stack will take days or weeks to validate. My customers recognize that building and maintaining any system like Sidekiq is non-trivial. You can spend a week or more annually maintaining your own Sidekiq "distro" or you can pay me to do it. So far, many hundreds of companies have opted for the latter. 2. Regarding multi-process specifically, each Sidekiq process is multi-threaded and will execute (by default) 25 jobs concurrently. Multi-threading is the OSS feature and delivers a huge jump in efficiently versus the old single-threaded Ruby job processors (Resque, DJ). multi-process makes it easier to spread work across many cores with a single command but nothing is stopping users from running multiple Sidekiq processes, they just won't share memory and must be managed separately.
- protomikron 10y ago
  Ok, thanks for your answers, that sounds reasonable. So if I have a multi-core machine with 8 cores I can utilize all of them if I have 8 CPU intensive jobs (considering they fit into memory)?
  - mperham 10y ago
    Not with MRI. The standard Ruby interpreter, like Python, uses a global lock to prevent threads from executing in parallel. You'd need to spin up 8 Sidekiq processes by hand or use the multi-process feature to automate that manual process. That said, your scenario is unusual. Most server-side jobs have plenty of I/O (think DB queries, memcached/redis calls, file I/O, etc), which allow the threads to context-switch; lots of I/O concurrency means lots of jobs get done quickly, even with one core.

mperham 10y ago

Good questions. Couple of things to note: 1. My paid features are all built on the core APIs that are open source. Almost every feature I sell has an OSS variant that someone has written. If you want to plug in a dozen different OSS libraries, integrate and test them together, you can get most of what I sell for $0. You will get varying levels of support and yearly upgrades of that stack will take days or weeks to validate. My customers recognize that building and maintaining any system like Sidekiq is non-trivial. You can spend a week or more annually maintaining your own Sidekiq "distro" or you can pay me to do it. So far, many hundreds of companies have opted for the latter. 2. Regarding multi-process specifically, each Sidekiq process is multi-threaded and will execute (by default) 25 jobs concurrently. Multi-threading is the OSS feature and delivers a huge jump in efficiently versus the old single-threaded Ruby job processors (Resque, DJ). multi-process makes it easier to spread work across many cores with a single command but nothing is stopping users from running multiple Sidekiq processes, they just won't share memory and must be managed separately.

protomikron 10y ago
Ok, thanks for your answers, that sounds reasonable. So if I have a multi-core machine with 8 cores I can utilize all of them if I have 8 CPU intensive jobs (considering they fit into memory)?
- mperham 10y ago
  Not with MRI. The standard Ruby interpreter, like Python, uses a global lock to prevent threads from executing in parallel. You'd need to spin up 8 Sidekiq processes by hand or use the multi-process feature to automate that manual process. That said, your scenario is unusual. Most server-side jobs have plenty of I/O (think DB queries, memcached/redis calls, file I/O, etc), which allow the threads to context-switch; lots of I/O concurrency means lots of jobs get done quickly, even with one core.