4 ms·

Agreed. Rails uses ActiveRecord and you'll most likely be trying to load millions of records through an ORM which will make you go insane trying to get it to wo

by hk1337 4y ago

Agreed. Rails uses ActiveRecord and you'll most likely be trying to load millions of records through an ORM which will make you go insane trying to get it to work efficiently. I have no problem with Rails or ActiveRecord but bulk upload of data through any ORM is a big no for me.

izietto 4y ago
> Rails uses ActiveRecord and you'll most likely be trying to load millions of records through an ORM which will make you go insane trying to get it to work efficiently. You just need to use batches (that `.in_batches` in the link), it's an easy-peasy approach which is way more acceptable for your database than a single update of 500m records. I've been using it for years and it has never raised a single issue.
- FrancoisBosun 4y ago
  The downcasing in done in Ruby-land, thus the records had to be loaded in memory, on the worker. If the downcasing was done on the DB itself, then the records would not have needed to travel all the way to the worker, only to be discarded 200ms later. Better to run an UPDATE query direct against the DB instance, with IDs to control concurrency: UPDATE addresses SET city = lower(city), state = lower(state) WHERE id BETWEEN a AND b
  - inferiorhuman 4y ago
    Yeah, one of the first things I learned (Ruby 1.8) was that Ruby is slow. If you can avoid creating a bunch of Ruby objects and push the logic into the database you'll typically see performance gains. The big exception was Mongo, where more "advanced" features got shoved off to the JS interpreter which was an expensive operation – at least until the 'aggregation framework' became a thing. Anyways. Before blindly doing string manipulation with postgres make sure that all the locale stuff is set correctly.
  - revscat 4y ago
    This is what the `in_batches` method already does. See [1], which is called by the `in_batches` method for update operations. [1] https://api.rubyonrails.org/classes/ActiveRecord/Relation.html#method-i-update_all https://api.rubyonrails.org/classes/ActiveRecord/Relation.ht...
    - strken 4y ago
      You could use `in_batches` that way, but in the code from the article it looks like each call to `maximum`/`minimum` is causing a round trip, plus the `map` inside `MyService` is loading each batch into memory.
  - izietto 4y ago
    I agree with you if you focus on this very example, but I think to it just as an example. When I read this article, I replace <lowercase> with <some business logic implemented server side>.

by hk1337 4y ago

izietto 4y ago

> Rails uses ActiveRecord and you'll most likely be trying to load millions of records through an ORM which will make you go insane trying to get it to work efficiently. You just need to use batches (that `.in_batches` in the link), it's an easy-peasy approach which is way more acceptable for your database than a single update of 500m records. I've been using it for years and it has never raised a single issue.

FrancoisBosun 4y ago
The downcasing in done in Ruby-land, thus the records had to be loaded in memory, on the worker. If the downcasing was done on the DB itself, then the records would not have needed to travel all the way to the worker, only to be discarded 200ms later. Better to run an UPDATE query direct against the DB instance, with IDs to control concurrency: UPDATE addresses SET city = lower(city), state = lower(state) WHERE id BETWEEN a AND b
- inferiorhuman 4y ago
  Yeah, one of the first things I learned (Ruby 1.8) was that Ruby is slow. If you can avoid creating a bunch of Ruby objects and push the logic into the database you'll typically see performance gains. The big exception was Mongo, where more "advanced" features got shoved off to the JS interpreter which was an expensive operation – at least until the 'aggregation framework' became a thing. Anyways. Before blindly doing string manipulation with postgres make sure that all the locale stuff is set correctly.
- revscat 4y ago
  This is what the `in_batches` method already does. See [1], which is called by the `in_batches` method for update operations. [1] https://api.rubyonrails.org/classes/ActiveRecord/Relation.html#method-i-update_all https://api.rubyonrails.org/classes/ActiveRecord/Relation.ht...
  - strken 4y ago
    You could use `in_batches` that way, but in the code from the article it looks like each call to `maximum`/`minimum` is causing a round trip, plus the `map` inside `MyService` is loading each batch into memory.
- izietto 4y ago
  I agree with you if you focus on this very example, but I think to it just as an example. When I read this article, I replace <lowercase> with <some business logic implemented server side>.