4 ms·
Yep, I wished I had had access to this back when I was working on something in the Rails side. Ingesting large amounts of data from Shopify across multiple oaut
by hosh 10y ago
Yep, I wished I had had access to this back when I was working on something in the Rails side. Ingesting large amounts of data from Shopify across multiple oauth access points while being limited to the PostgreSQL backend and Shopify rate limit restrictions was a pain to setup with Redis and Sidekiq. I ended up forking Sidekiq to accomplish that -- essentially using Sidekiq as a concurrency framework to build in a different set of behavior. Reading through the GenStage document, I could have implemented something better.
The way I implemented it (with Ruby) left gaps of idle time, and huge amounts of data gets staged through the queues. I couldn't figure out how to make it better. Since I only had a week to come up with something while the infrastructure was melting down, at the time, it was acceptable. Looking at this though, the demand-based backpressure would work very well.
There are a couple problems I think I can use this in my current work -- this is great work!
- danvayn 10y agoInstall the Bullet gem and see if your queries look sharp!
- hosh 10y agoAlready did. That wasn't the issue. I get that you're trying to help, but you don't really understand the problem and why GenStage would help it. The issue isn't N+1 queries but in concurrency and reasoning with concurrency, something that Rails, and many Rails developers seem to avoid. See also: https://news.ycombinator.com/item?id=12074425 https://news.ycombinator.com/item?id=12074425