7 ms·

Happy 10th Birthday, Sidekiq

211 points by nickjj 5y ago

asim 5y ago
I remember when Mike posted about making $1m from sidekiq. It was always such an inspiration and it's amazing to see how far it's come. Kudos to Mike and I hope others find that path successful too.
dayvid 5y ago
Sidekiq's a really good library from my experience. I don't use Rails nowadays, but they've had some of the best libraries to work with. ActiveAdmin was another amazing library that saved me on a consulting project.
bpicolo 5y ago
I think Mike’s success is awesome. Sidekiq has created so much value for startups in our industry and can operate up to a scale very few companies have to actually worry about. Other folk have reproduced similar in other ecosystems to success as well (see Hangfire) and there’s another half a dozen ecosystems out there where a motivated person could capture the same value. The only downside I see is that it’s definitely led to the proliferation of commercializing OSS libs by making them slightly worse on purpose and adding a “go pro!” version. All too common these days. But again, can’t blame anybody on this path - creating good software isn’t easy and people do deserve to be paid for it. The other place ripe for opportunity is Django-admin style tools. It’s been out for a decade maintained and free and it’s just an awesome tool. There are similar frameworks in Ruby, PHP (part of laravel’s pay to play offering), and Elixir, but not too much outside that. I’m pretty darn surprised there’s no Java or JavaScript world admin framework that can be plugged into with code. (react admin and the like are too low level, and being SPA centric don’t fight the whole battle - getting 95% of the way with no code other than pointing at a model is the real value add). One could also make oodles of cash in .NET here i imagine
- jon1628492 5y ago
  I'm curious about the ones you like for Elixir! I've been looking for a bit and haven't found one that seems to be maintained and close to feature parity with Django admin
  - bpicolo 5y ago
    There definitely isn't one at feature parity, just better attempts than I've seen in other ecosystems, heh.
  - beedrillzzzzz 5y ago
    Ash is an amazing framework for Elixir that comes with functionality very equivalent to DRF/filtersets/etc, admin, as well as automatic JSON APIs, GraphQL APIs, plus works great behind LiveView. The maintainer is extremely active and is about to release a much improved website for getting started and learning. I highly suggest joining the discord if you are looking for Django for Elixir. Website: https://www.ash-elixir.org https://www.ash-elixir.org Discord: https://discord.com/invite/D7FNG2q https://discord.com/invite/D7FNG2q
  - bgentry 5y ago
    Oban is really great: https://getoban.pro https://getoban.pro Built on Postgres, it lets your background jobs achieve transactionality and consistency with your database. Its architecture is also well designed to allow it to scale up to about as much as Postgres can handle. The Pro version comes with a great web UI built with Phoenix LiveView that does just about everything one could ask for.
- lf-non 5y ago
  Regarding django-admin style tools, there are solutions like zk [1] & apache isis [2] for java which are somewhat comparable, though more enterprise oriented. [1] https://www.zkoss.org/ https://www.zkoss.org/ [2] https://isis.apache.org/ https://isis.apache.org/
- pizza234 5y ago
  > The only downside I see is that it’s definitely led to the proliferation of commercializing OSS libs by making them slightly worse on purpose and adding a “go pro!” version This is a slightly pessimistic take on open core :) I'm personally a big fan, as opposed to libraries that are not maintained and die. Sidekiq's competitor, Resque, has a somewhat uncertain situation (it's been dead for an year or so, then new maintainers took over). The advantage of open core is that nobody prevents devs to create their own plugins, especially for products that are accessible to developers without highly specific knowledge (say, a relational database); this is actually happening with Sidekiq - for example, there is a plugin for a more granular scheduler.
  - bpicolo 5y ago
    You're probably right. Just doesn't scratch quite the same egalitarian itch eh?
  - slim 5y ago
    the problem is the opensource part becomes pure marketing. for example the documentation is always about how to do things the "pro" way and becomes useless and even feels sometimes hostile to people searching for alternatives
- claytongulick 5y ago
  For JavaScript there's strapi [1] https://strapi.io/ https://strapi.io/
  - m12k 5y ago
    And Directus was recently ported from PHP to JavaScript as well: https://directus.io/ https://directus.io/
- fomojola 5y ago
  I think the lack of a dominant JavaScript domain model syntax has limited the availability of admin tools like django-admin. Maybe something driven by OpenAPI/Swagger model definitions? It is ancient and no longer upgraded (and based on angularjs), but I'd argue that ng-admin (https://github.com/marmelab/ng-admin https://github.com/marmelab/ng-admin) is the fastest SPA/JavaScript admin framework out there. It is a bit opinionated about paging and filtering (and painful to do moderately complex UI customization), but beyond that I have seen nothing that has come close to the same level of "quickly get CRUD admin UI available" in react/vue land, largely because (similar to django) it had its own domain entity model. The same team moved from there to react-admin, and looking at a couple of react-admin experiences I've been involved in I'd argue that it wasn't aiming for the same ease of use.
uptime 5y ago
This is a great summary of a positive outcome for a hybrid OSS project. I took a look at the other blog posts - has anyone seen writing from the author that covers his experience supporting the pro and enterprise versions? I imagine that would be hard to do alone!
- mperham 5y ago
  It’s worked out so far. I knew it would be tough to scale up myself as support so I set down as couple of policies: 1. Do it in public if possible via GitHub issues so others can also answer and search in the future. 2. Don’t allow more than N questions per year per customer. A few have drowned me with requests. Nope, you are buying a product, not a trainer or teacher. Read the docs, watch the videos. 3. Pour everything I know into the wiki docs. Note each wiki page will talk about semantics, edge cases, etc. That’s all there to get ahead of the support inquiries.
  - aaronmiler 5y ago
    #3 is one of my favorite things about Sidekiq. Anything I'm trying to do with Sidekiq, I check the wiki first, and it's usually there. Any time I've introduced the library to a team, the docs and wiki are always my recommendation for answering questions. Even Pro/Ent features are covered there, which is so much nicer than having the docs hidden behind a login/paywall. Thanks for the hard work you've poured into them Mike!
mos_6502 5y ago
Sidekiq is an excellent library, and a great model for open-source monetization. Avoid hostile/nonfree licensing provisions that restrict commercial use or "require corporations to pay up". Instead, create real value in diversified offerings built upon the OSS project (which is in no way artificially limited). Sidekiq is a gift, Sidekiq Pro & Enterprise are products. Offer paid features, support, SLAs, etc that really appeal to "enterprise" use-cases. Trust that users will convert as their need for such functionality is justified by their businesses' growth and scale.
JohnBooty 5y ago
That's awesome. A significant percentage of Ruby shops are running Sidekiq. I'm happy for his success. The free version is very capable too; he found a very nice balance between "free version that is still very usable" and "paid version with really useful additional features."
iamricks 5y ago
Sidekiq is the best library i have ever used, i have never ran into any issues using it and Mike has personally answered a bunch of my Stack Overflow questions that involve Sidekiq. He deserves every dollar.
ch4s3 5y ago
I'm interested to see what he decides to do approaching 2030.
hbrundage 5y ago
So glad that Mike has set up a sustainable model for building rock solid infrastructure, however: I can’t stand that you need to pay for a pro license to avoid completely avoidable data loss. Sidekiq doesn’t use RPOPLPUSH for moving jobs around until you start paying — doing the exact same thing with a different, more correct command. No new infrastructure or configuration is needed. To me this feels sketchy and like a betrayal of the pro-version business model — willfully introducing data loss in order to drive upgrades instead of just removing capabilities or features.
- grumple 5y ago
  Sidekiq is open source, so couldn't you or someone else make another gem for the change? I know there are gems already that enable sidekiq to replicate some of the paid or enterprise features without paying for those plans.
- brightball 5y ago
  This touches on one of my favorite things about Ruby as a language. The functionality you're asking for becomes pretty easy to monkey patch in yourself without breaking your upgrade path or forcing you to maintain a fork. IMO that easy of tweaking 3rd party libraries is the reason that Ruby's ecosystem is so polished.
  - nickjj 5y ago
    Yeah that is an interesting point. It's really easy to drop in an initializer, make your patch, let it ride in production for a while and then contribute your patch back to the main project than it is to set up a fork and all of the ceremony involved with keeping it up to date. It's a good example of removing barriers of entry or an idea of Ruby's "sharp knives". Great for when you need it but can also be a disaster when abused.
    - brightball 5y ago
      Yep. Definitely a great power, great responsibility tool. It's the one thing that keeps Ruby closer to true Aspect Oriented Programming than any other language (in my experience at least). It's probably the biggest thing that I miss from Ruby when working in other languages, even ones that I love like Elixir.
Aleksdev 5y ago
Nothing wrong with making a profit. Used sidekiq on all the rails projects I worked on. Great little gem :-)
bdcravens 5y ago
Been a happy Sidekiq Pro customer for years.
axelfontaine 5y ago
This is really an amazing achievement! It was Mike's IndieHackers interview (https://www.indiehackers.com/interview/how-charging-money-for-pro-features-allowed-me-quit-my-job-6e71309457 https://www.indiehackers.com/interview/how-charging-money-fo...) that really pushed me over the edge to go open-core with Flyway in 2017 and give it a sustainable future. Thanks again Mike, if you're reading this!
jakswa 5y ago
Sidekiq has stayed with me for most of my rails projects. Happy to read this. And I bet I'll continue to remember it as I run into projects that were inspired by it, like Shoryuken (which tries to be "sidekiq for AWS SNS/SQS" https://github.com/ruby-shoryuken/shoryuken https://github.com/ruby-shoryuken/shoryuken).
sickcodebruh 5y ago
Sidekiq is one of my first answers to the question, “Why would I choose Ruby in 2022?” It is a fantastic library and Mike is a remarkable developer. Congratulations on hitting 10 years and thank you for everything!
mtalantikite 5y ago
Just wanted to say the Prehistory section brought up a lot of nostalgia for me. Those early Ruby projects and articles that Ezra wrote led me to working at his company Engine Yard back in the beginning of 2008. Lots of memories of late nights trying to fix GitHub's deployment (GFS2 nightmares) as they were growing in the community. I remember when Resque came out and we started using it for so many of our customers. Tons of really smart hackers hanging out at the office/apartment on South Park. Definitely wouldn't have my career without all those early Ruby hackers working together in community. Makes me wonder where in the OSS world that community is thriving just like back then.
alberth 5y ago
Off topic: is “open-core” any different than a “Freemium commercial product you can self-host”? I don’t want this question to sidetrack into a heated debate about open source vs commercial. But I am genuinely curious if there is a difference between the two.
Murkin 5y ago
Amazing project, only sad note is the monetization side is a bit neglected. We were ready to pay for Sidekiq, but the pricing model was just off and we just used an OSS lib to patch the missing part. Mike if you are here, might be worth to invest some time in pricing research, you could probably double your revenue.
- mperham 5y ago
  I don’t really understand what you mean but I’d like to hear details.
  - Murkin 5y ago
    We needed the unique jobs feature, which only part of the enterprise plan. And since (at that stage) our budget was low, the enterprise plan was too expensive. I reached out to ask if we can do any other payment scheme, or at least get startup discount to start using, but were declined. We ended up using an OSS library for that part and even now what we grown, still don't pay for sidekiq. If it was part of Pro, or we had a 1 year discount for startups, we prob just kept on paying even now (that we can afford it).
- ed 5y ago
  > the pricing model was just off Out of curiosity (I also sell enterprise products) what would make the purchase a no-brainer? Quarterly payments? E.g. I’ve heard that ~$300 (ballpark) is the max amount an individual can spend without a review process. Is that what you mean?
  - Murkin 5y ago
    There were 3 plans offered, community, pro and enterprise. We only needed one feature that was sadly in the "enterprise" bucket. Sadly that plan was very expensive, so ended up not paying anything and just getting the feature from OSS. No the pricing, but rather the feature coupling to plans and no willingness to do any dynamic pricing.

211 points by nickjj 5y ago

asim 5y ago

I remember when Mike posted about making $1m from sidekiq. It was always such an inspiration and it's amazing to see how far it's come. Kudos to Mike and I hope others find that path successful too.

dayvid 5y ago

Sidekiq's a really good library from my experience. I don't use Rails nowadays, but they've had some of the best libraries to work with. ActiveAdmin was another amazing library that saved me on a consulting project.

bpicolo 5y ago

I think Mike’s success is awesome. Sidekiq has created so much value for startups in our industry and can operate up to a scale very few companies have to actually worry about. Other folk have reproduced similar in other ecosystems to success as well (see Hangfire) and there’s another half a dozen ecosystems out there where a motivated person could capture the same value. The only downside I see is that it’s definitely led to the proliferation of commercializing OSS libs by making them slightly worse on purpose and adding a “go pro!” version. All too common these days. But again, can’t blame anybody on this path - creating good software isn’t easy and people do deserve to be paid for it. The other place ripe for opportunity is Django-admin style tools. It’s been out for a decade maintained and free and it’s just an awesome tool. There are similar frameworks in Ruby, PHP (part of laravel’s pay to play offering), and Elixir, but not too much outside that. I’m pretty darn surprised there’s no Java or JavaScript world admin framework that can be plugged into with code. (react admin and the like are too low level, and being SPA centric don’t fight the whole battle - getting 95% of the way with no code other than pointing at a model is the real value add). One could also make oodles of cash in .NET here i imagine

jon1628492 5y ago
I'm curious about the ones you like for Elixir! I've been looking for a bit and haven't found one that seems to be maintained and close to feature parity with Django admin
- bpicolo 5y ago
  There definitely isn't one at feature parity, just better attempts than I've seen in other ecosystems, heh.
- beedrillzzzzz 5y ago
  Ash is an amazing framework for Elixir that comes with functionality very equivalent to DRF/filtersets/etc, admin, as well as automatic JSON APIs, GraphQL APIs, plus works great behind LiveView. The maintainer is extremely active and is about to release a much improved website for getting started and learning. I highly suggest joining the discord if you are looking for Django for Elixir. Website: https://www.ash-elixir.org https://www.ash-elixir.org Discord: https://discord.com/invite/D7FNG2q https://discord.com/invite/D7FNG2q
- bgentry 5y ago
  Oban is really great: https://getoban.pro https://getoban.pro Built on Postgres, it lets your background jobs achieve transactionality and consistency with your database. Its architecture is also well designed to allow it to scale up to about as much as Postgres can handle. The Pro version comes with a great web UI built with Phoenix LiveView that does just about everything one could ask for.
lf-non 5y ago
Regarding django-admin style tools, there are solutions like zk [1] & apache isis [2] for java which are somewhat comparable, though more enterprise oriented. [1] https://www.zkoss.org/ https://www.zkoss.org/ [2] https://isis.apache.org/ https://isis.apache.org/
pizza234 5y ago
> The only downside I see is that it’s definitely led to the proliferation of commercializing OSS libs by making them slightly worse on purpose and adding a “go pro!” version This is a slightly pessimistic take on open core :) I'm personally a big fan, as opposed to libraries that are not maintained and die. Sidekiq's competitor, Resque, has a somewhat uncertain situation (it's been dead for an year or so, then new maintainers took over). The advantage of open core is that nobody prevents devs to create their own plugins, especially for products that are accessible to developers without highly specific knowledge (say, a relational database); this is actually happening with Sidekiq - for example, there is a plugin for a more granular scheduler.
- bpicolo 5y ago
  You're probably right. Just doesn't scratch quite the same egalitarian itch eh?
- slim 5y ago
  the problem is the opensource part becomes pure marketing. for example the documentation is always about how to do things the "pro" way and becomes useless and even feels sometimes hostile to people searching for alternatives
claytongulick 5y ago
For JavaScript there's strapi [1] https://strapi.io/ https://strapi.io/
- m12k 5y ago
  And Directus was recently ported from PHP to JavaScript as well: https://directus.io/ https://directus.io/
fomojola 5y ago
I think the lack of a dominant JavaScript domain model syntax has limited the availability of admin tools like django-admin. Maybe something driven by OpenAPI/Swagger model definitions? It is ancient and no longer upgraded (and based on angularjs), but I'd argue that ng-admin (https://github.com/marmelab/ng-admin https://github.com/marmelab/ng-admin) is the fastest SPA/JavaScript admin framework out there. It is a bit opinionated about paging and filtering (and painful to do moderately complex UI customization), but beyond that I have seen nothing that has come close to the same level of "quickly get CRUD admin UI available" in react/vue land, largely because (similar to django) it had its own domain entity model. The same team moved from there to react-admin, and looking at a couple of react-admin experiences I've been involved in I'd argue that it wasn't aiming for the same ease of use.

uptime 5y ago

This is a great summary of a positive outcome for a hybrid OSS project. I took a look at the other blog posts - has anyone seen writing from the author that covers his experience supporting the pro and enterprise versions? I imagine that would be hard to do alone!

mperham 5y ago
It’s worked out so far. I knew it would be tough to scale up myself as support so I set down as couple of policies: 1. Do it in public if possible via GitHub issues so others can also answer and search in the future. 2. Don’t allow more than N questions per year per customer. A few have drowned me with requests. Nope, you are buying a product, not a trainer or teacher. Read the docs, watch the videos. 3. Pour everything I know into the wiki docs. Note each wiki page will talk about semantics, edge cases, etc. That’s all there to get ahead of the support inquiries.
- aaronmiler 5y ago
  #3 is one of my favorite things about Sidekiq. Anything I'm trying to do with Sidekiq, I check the wiki first, and it's usually there. Any time I've introduced the library to a team, the docs and wiki are always my recommendation for answering questions. Even Pro/Ent features are covered there, which is so much nicer than having the docs hidden behind a login/paywall. Thanks for the hard work you've poured into them Mike!

mos_6502 5y ago

Sidekiq is an excellent library, and a great model for open-source monetization. Avoid hostile/nonfree licensing provisions that restrict commercial use or "require corporations to pay up". Instead, create real value in diversified offerings built upon the OSS project (which is in no way artificially limited). Sidekiq is a gift, Sidekiq Pro & Enterprise are products. Offer paid features, support, SLAs, etc that really appeal to "enterprise" use-cases. Trust that users will convert as their need for such functionality is justified by their businesses' growth and scale.

JohnBooty 5y ago

That's awesome. A significant percentage of Ruby shops are running Sidekiq. I'm happy for his success. The free version is very capable too; he found a very nice balance between "free version that is still very usable" and "paid version with really useful additional features."

iamricks 5y ago

Sidekiq is the best library i have ever used, i have never ran into any issues using it and Mike has personally answered a bunch of my Stack Overflow questions that involve Sidekiq. He deserves every dollar.

ch4s3 5y ago

I'm interested to see what he decides to do approaching 2030.

hbrundage 5y ago

So glad that Mike has set up a sustainable model for building rock solid infrastructure, however: I can’t stand that you need to pay for a pro license to avoid completely avoidable data loss. Sidekiq doesn’t use RPOPLPUSH for moving jobs around until you start paying — doing the exact same thing with a different, more correct command. No new infrastructure or configuration is needed. To me this feels sketchy and like a betrayal of the pro-version business model — willfully introducing data loss in order to drive upgrades instead of just removing capabilities or features.

grumple 5y ago
Sidekiq is open source, so couldn't you or someone else make another gem for the change? I know there are gems already that enable sidekiq to replicate some of the paid or enterprise features without paying for those plans.
brightball 5y ago
This touches on one of my favorite things about Ruby as a language. The functionality you're asking for becomes pretty easy to monkey patch in yourself without breaking your upgrade path or forcing you to maintain a fork. IMO that easy of tweaking 3rd party libraries is the reason that Ruby's ecosystem is so polished.
- nickjj 5y ago
  Yeah that is an interesting point. It's really easy to drop in an initializer, make your patch, let it ride in production for a while and then contribute your patch back to the main project than it is to set up a fork and all of the ceremony involved with keeping it up to date. It's a good example of removing barriers of entry or an idea of Ruby's "sharp knives". Great for when you need it but can also be a disaster when abused.
  - brightball 5y ago
    Yep. Definitely a great power, great responsibility tool. It's the one thing that keeps Ruby closer to true Aspect Oriented Programming than any other language (in my experience at least). It's probably the biggest thing that I miss from Ruby when working in other languages, even ones that I love like Elixir.

Aleksdev 5y ago

Nothing wrong with making a profit. Used sidekiq on all the rails projects I worked on. Great little gem :-)

bdcravens 5y ago

Been a happy Sidekiq Pro customer for years.

axelfontaine 5y ago

This is really an amazing achievement! It was Mike's IndieHackers interview (https://www.indiehackers.com/interview/how-charging-money-for-pro-features-allowed-me-quit-my-job-6e71309457 https://www.indiehackers.com/interview/how-charging-money-fo...) that really pushed me over the edge to go open-core with Flyway in 2017 and give it a sustainable future. Thanks again Mike, if you're reading this!

jakswa 5y ago

Sidekiq has stayed with me for most of my rails projects. Happy to read this. And I bet I'll continue to remember it as I run into projects that were inspired by it, like Shoryuken (which tries to be "sidekiq for AWS SNS/SQS" https://github.com/ruby-shoryuken/shoryuken https://github.com/ruby-shoryuken/shoryuken).

sickcodebruh 5y ago

Sidekiq is one of my first answers to the question, “Why would I choose Ruby in 2022?” It is a fantastic library and Mike is a remarkable developer. Congratulations on hitting 10 years and thank you for everything!

mtalantikite 5y ago

Just wanted to say the Prehistory section brought up a lot of nostalgia for me. Those early Ruby projects and articles that Ezra wrote led me to working at his company Engine Yard back in the beginning of 2008. Lots of memories of late nights trying to fix GitHub's deployment (GFS2 nightmares) as they were growing in the community. I remember when Resque came out and we started using it for so many of our customers. Tons of really smart hackers hanging out at the office/apartment on South Park. Definitely wouldn't have my career without all those early Ruby hackers working together in community. Makes me wonder where in the OSS world that community is thriving just like back then.

alberth 5y ago

Off topic: is “open-core” any different than a “Freemium commercial product you can self-host”? I don’t want this question to sidetrack into a heated debate about open source vs commercial. But I am genuinely curious if there is a difference between the two.

Murkin 5y ago

Amazing project, only sad note is the monetization side is a bit neglected. We were ready to pay for Sidekiq, but the pricing model was just off and we just used an OSS lib to patch the missing part. Mike if you are here, might be worth to invest some time in pricing research, you could probably double your revenue.

mperham 5y ago
I don’t really understand what you mean but I’d like to hear details.
- Murkin 5y ago
  We needed the unique jobs feature, which only part of the enterprise plan. And since (at that stage) our budget was low, the enterprise plan was too expensive. I reached out to ask if we can do any other payment scheme, or at least get startup discount to start using, but were declined. We ended up using an OSS library for that part and even now what we grown, still don't pay for sidekiq. If it was part of Pro, or we had a 1 year discount for startups, we prob just kept on paying even now (that we can afford it).
ed 5y ago
> the pricing model was just off Out of curiosity (I also sell enterprise products) what would make the purchase a no-brainer? Quarterly payments? E.g. I’ve heard that ~$300 (ballpark) is the max amount an individual can spend without a review process. Is that what you mean?
- Murkin 5y ago
  There were 3 plans offered, community, pro and enterprise. We only needed one feature that was sadly in the "enterprise" bucket. Sadly that plan was very expensive, so ended up not paying anything and just getting the feature from OSS. No the pricing, but rather the feature coupling to plans and no willingness to do any dynamic pricing.