4 ms·
Wasteful and the wrong place to do it. Also, Sidekiq is slow, old, and commercial crippleware. Generate and send SQL (wrapped in transactions) from C bindings
by tinglymintyfrsh 3y ago
Wasteful and the wrong place to do it. Also, Sidekiq is slow, old, and commercial crippleware.
Generate and send SQL (wrapped in transactions) from C bindings via a proper background job processor like beanstalkd in batches of say 50k. Getting Ruby involved in millions of records is asking for trouble.
- peteforde 3y agoFeel free to do things however you like, but Sidekiq is excellent. It's extremely well-designed and battle-hardened code, and it recently had a major release. Most of the non-enterprise customers pay because we want to support Mike Perham, not because we need paid-tier features.
- tinglymintyfrsh 3y ago"Battle-hardened" where besides tiny startups and 8 companies? Beanstalkd is simple, scales, and is platform agnostic. At scale, simple wins.
- peteforde 3y ago8 companies? Where do you get that [absurdly incorrect] number? Mike has actual thousands of enterprise customers. I don't think anyone here suggested that Beanstalkd is bad or should not be used, if it fits your use case like a glove. Speed-wise, let's not forget that Redis is also written in C. Bigger picture, folks outside of the Rails community love to sidestep the simple fact that something like 75% of the total value created by YC-backed companies was created by the subset who built with Rails. Clearly, we're connecting the dots in a way others are not.