4 ms·

edit: to down voters: please educate me and not just down vote because you disagree with my question. --- Every tiny rails app that uses rspec ends up with +1

by itake 4y ago

edit: to down voters: please educate me and not just down vote because you disagree with my question. --- Every tiny rails app that uses rspec ends up with +1hr test times due to tight coupling between activemodel and the db connections. Do you know of any good blog posts the describe how to do rails testing at scale? > Companies doing grunt work style web development don't waste their time on flavor-of-the-month.js. Rails devs have to deal with DHH's-flavor-of-the-month.coffee (coffeescript, asset pipeline, webpacker, import_maps)

worker_person 4y ago
When you hit scale. Rails is not the concern. It's the datastore. All of the major performance problems at scale I've dealt with came down to a super giant table that keeps getting locked.
- itake 4y ago
  In another comment, it said Gitlab moved off ActiveRecord. The fundamental problem with rspec + ActiveRecord is for each test: you create db state (tons of inserts), run the test (more db reads and writes), and then tear down the state. This is very expensive when you have 100,000s of tests each taking 500ms. Rails/rspec does not make it easy to stub db state with ActiveRecord.
  - worker_person 4y ago
    http://underpop.online.fr/r/ruby/rails/tutorial/ruby-on-rails-7-12.html http://underpop.online.fr/r/ruby/rails/tutorial/ruby-on-rail... Use transactional fixtures as much as possible. Minimal db usage between tests.
    - itake 4y ago
      yes, that improves clean up time, but doesn't help with all of the db writes you have to make to create db state for each test case.
      - worker_person 4y ago
        Initial state is loaded via fixtures or whatever. (slow and expensive) Transaction is started. Run Test 1 Transaction rolled back. (quick) Test 2 can be run without the expensive fixtures. Repeat for all tests.
        itake 4y ago
        Does this mean if you only want to run the tests for `users_controller` during development, you still have to load the fixtures for all other tests with each run?
        worker_person 4y ago
        In the normal case. When you start a test it loads from fixtures into its own database. Any test can make use of what is in that database. So you tend to put a bunch of stuff that is useful for most tests. If a specific test needs an unusual setup, then it can do whatever is needed to load data. Generate data, load extra fixtures, etc. You can do this on a per test basis, or a group of tests, etc. An option is to reload everything between each test. Not recommended. Normal usage is to rollback any changes a test did, so that each test has a clean slate. This tends to be plenty fast. Parallelism can usually be done. If speed is still a concern, you can get fancy and load up multiple databases with fixtures and divide tests between them. Maybe have 2-8 sets of tests running at same time. It's been a few years, but I've inherited test suites that take 14 hours and been able to get them down to a few minutes with above techniques. Something like Guard can be setup to automatically run tests relevant to what you're working on. So you don't have to run full suite each time.
arcticfox 4y ago
> Every tiny rails app that uses rspec ends up with +1hr test times due to tight coupling between activemodel and the db connections. Wat? I mean test times aren't great but I've got several thousand specs for what I would call a medium-sized application that run in <1m using TurboTests or <3m using rspec alone. I can't imagine a "tiny" rails app that takes an hour to test. Maybe you're doing a lot of browser rendering or something? My app is API only.
- itake 4y ago
  I think the fundamental design problem for rspec is for each test you create db state (tons of writes), run the test (reads and writes), and then tear-down (truncate). Each test is at least adding an authorized user (and then removing it) at the simplest and at the worst creating complicated db state to support relational models that must be added and torn down with each test (database_cleaner). Gems like Fabricator or FactoryBot also make the "create db state" even more excessive, because developers would be lazy and use factory methods that create more than they actually need for the test. For example, one project I stored a tree-structure that could be modified by api calls. Each test required re-creating the 15 node tree + each node's relational data (think node = user and all users must have a profile, authorization scheme, etc.). I never figured out how to avoid resetting the db state for each test case.
  - jamie_ca 4y ago
    This isn't an rspec problem, it's a Factories-for-ActiveRecord problem. The two main options to mitigate are: - just work with in-memory objects (FactoryBot using build or build_stubbed, or just MyModel.new) and stub/mock finders as needed, - or there's this thing people like to hate that was introduced in Rails 1 called test fixtures, which allow you to prepopulate the database with some initial data and then your test just runs in a transaction and rolls back. The best suggestion I've heard recently (though I can't recall where to give credit) for managing complexity w/ fixtures is to write an extremely minimal fixture set aiming for 2 instances per model with minimal data required to be valid, and then customize it as part of your test. For a blog example, that means you have two Posts in your fixtures and would write a test around drafts by updating one of them to draft status (be it state, toggle, or publication date) and then test your query, or controller, or whatever. This keeps the data churn inside each test minimal, while keeping the out-of-scope data minimal as well -- if there's only 2 blog posts, all your tests can assume there will be one you care about and one you don't, which is useful for testing filtering/search/visibility/authorization/etc while remaining pretty consistent.
gepardi 4y ago
I don’t think anyone has to use coffee script. Our company has a large code base. Tests run in about 15 minutes in CI.
- itake 4y ago
  What testing tools are you using? rspec+database_cleaner+FactorBot/Fabricator? How many tests do you run?
mscccc 4y ago
I worked on GitHub's Rails app. The main way this was solved was by running tests in parallel. It's now available in default Rails: https://edgeguides.rubyonrails.org/testing.html#parallel-testing https://edgeguides.rubyonrails.org/testing.html#parallel-tes... In addition to that, the test suite performance was regularly monitored. And like any performance issue, we'd instrument and then fix issues as they came up.
- itake 4y ago
  (duplicating another comment I wrote to hear your thoughts) The fundamental design problem for rspec is for each test you create db state (tons of writes), run the test (reads and writes), and then tear-down (truncate). Each test is at least adding an authorized user (and then removing it) at the simplest and at the worst creating complicated db state to support relational models that must be added and torn down with each test (database_cleaner). Gems like Fabricator or FactoryBot also make the "create db state" even more excessive, because developers would be lazy and use factory methods that create more than they actually need for the test.
  - mscccc 4y ago
    GitHub uses minitest, but same situation. I don't exactly agree that it's a design problem. I think there is a lot of value in testing at a high level (controller tests) and having them touch the full stack (down to the DB). Gives a lot of confidence that the code will work in production. Throwing money at it (running the tests in parallel) pretty much solves it. I have seen FactoryBot use get out of control (creating 8x as many records as you'd expect). That's a really easy way to slow down a test suite :). One way I've found to fix that is by adding tests for the factories, and asserting they are only creating what you want them to. On model tests, another thing GitHub did well was encourage using `.build` when possible to avoid writing the the DB.
    - itake 4y ago
      > I think there is a lot of value in testing at a high level (controller tests) and having them touch the full stack (down to the DB). There is a lot of wasted work. Each (similar) test would insert and then delete the same rows, but make a little tweak. Other languages and frameworks (java, python or golang) don't need to test the db and run super quick. Most db systems do not actually need to be tested. e2e testing can be done manually via curl or qa. > Throwing money at it (running the tests in parallel) pretty much solves it. Tests still are slow locally. > One way I've found to fix that is by adding tests for the factories That is interesting. I had a meta-test that would limit the number of sql queries a test could trigger.

by itake 4y ago

worker_person 4y ago

When you hit scale. Rails is not the concern. It's the datastore. All of the major performance problems at scale I've dealt with came down to a super giant table that keeps getting locked.

itake 4y ago
In another comment, it said Gitlab moved off ActiveRecord. The fundamental problem with rspec + ActiveRecord is for each test: you create db state (tons of inserts), run the test (more db reads and writes), and then tear down the state. This is very expensive when you have 100,000s of tests each taking 500ms. Rails/rspec does not make it easy to stub db state with ActiveRecord.
- worker_person 4y ago
  http://underpop.online.fr/r/ruby/rails/tutorial/ruby-on-rails-7-12.html http://underpop.online.fr/r/ruby/rails/tutorial/ruby-on-rail... Use transactional fixtures as much as possible. Minimal db usage between tests.
  - itake 4y ago
    yes, that improves clean up time, but doesn't help with all of the db writes you have to make to create db state for each test case.
    - worker_person 4y ago
      Initial state is loaded via fixtures or whatever. (slow and expensive) Transaction is started. Run Test 1 Transaction rolled back. (quick) Test 2 can be run without the expensive fixtures. Repeat for all tests.
      - itake 4y ago
        Does this mean if you only want to run the tests for `users_controller` during development, you still have to load the fixtures for all other tests with each run?
        worker_person 4y ago
        In the normal case. When you start a test it loads from fixtures into its own database. Any test can make use of what is in that database. So you tend to put a bunch of stuff that is useful for most tests. If a specific test needs an unusual setup, then it can do whatever is needed to load data. Generate data, load extra fixtures, etc. You can do this on a per test basis, or a group of tests, etc. An option is to reload everything between each test. Not recommended. Normal usage is to rollback any changes a test did, so that each test has a clean slate. This tends to be plenty fast. Parallelism can usually be done. If speed is still a concern, you can get fancy and load up multiple databases with fixtures and divide tests between them. Maybe have 2-8 sets of tests running at same time. It's been a few years, but I've inherited test suites that take 14 hours and been able to get them down to a few minutes with above techniques. Something like Guard can be setup to automatically run tests relevant to what you're working on. So you don't have to run full suite each time.

arcticfox 4y ago

> Every tiny rails app that uses rspec ends up with +1hr test times due to tight coupling between activemodel and the db connections. Wat? I mean test times aren't great but I've got several thousand specs for what I would call a medium-sized application that run in <1m using TurboTests or <3m using rspec alone. I can't imagine a "tiny" rails app that takes an hour to test. Maybe you're doing a lot of browser rendering or something? My app is API only.

itake 4y ago
I think the fundamental design problem for rspec is for each test you create db state (tons of writes), run the test (reads and writes), and then tear-down (truncate). Each test is at least adding an authorized user (and then removing it) at the simplest and at the worst creating complicated db state to support relational models that must be added and torn down with each test (database_cleaner). Gems like Fabricator or FactoryBot also make the "create db state" even more excessive, because developers would be lazy and use factory methods that create more than they actually need for the test. For example, one project I stored a tree-structure that could be modified by api calls. Each test required re-creating the 15 node tree + each node's relational data (think node = user and all users must have a profile, authorization scheme, etc.). I never figured out how to avoid resetting the db state for each test case.
- jamie_ca 4y ago
  This isn't an rspec problem, it's a Factories-for-ActiveRecord problem. The two main options to mitigate are: - just work with in-memory objects (FactoryBot using build or build_stubbed, or just MyModel.new) and stub/mock finders as needed, - or there's this thing people like to hate that was introduced in Rails 1 called test fixtures, which allow you to prepopulate the database with some initial data and then your test just runs in a transaction and rolls back. The best suggestion I've heard recently (though I can't recall where to give credit) for managing complexity w/ fixtures is to write an extremely minimal fixture set aiming for 2 instances per model with minimal data required to be valid, and then customize it as part of your test. For a blog example, that means you have two Posts in your fixtures and would write a test around drafts by updating one of them to draft status (be it state, toggle, or publication date) and then test your query, or controller, or whatever. This keeps the data churn inside each test minimal, while keeping the out-of-scope data minimal as well -- if there's only 2 blog posts, all your tests can assume there will be one you care about and one you don't, which is useful for testing filtering/search/visibility/authorization/etc while remaining pretty consistent.

gepardi 4y ago

I don’t think anyone has to use coffee script. Our company has a large code base. Tests run in about 15 minutes in CI.

itake 4y ago
What testing tools are you using? rspec+database_cleaner+FactorBot/Fabricator? How many tests do you run?

mscccc 4y ago

I worked on GitHub's Rails app. The main way this was solved was by running tests in parallel. It's now available in default Rails: https://edgeguides.rubyonrails.org/testing.html#parallel-testing https://edgeguides.rubyonrails.org/testing.html#parallel-tes... In addition to that, the test suite performance was regularly monitored. And like any performance issue, we'd instrument and then fix issues as they came up.

itake 4y ago
(duplicating another comment I wrote to hear your thoughts) The fundamental design problem for rspec is for each test you create db state (tons of writes), run the test (reads and writes), and then tear-down (truncate). Each test is at least adding an authorized user (and then removing it) at the simplest and at the worst creating complicated db state to support relational models that must be added and torn down with each test (database_cleaner). Gems like Fabricator or FactoryBot also make the "create db state" even more excessive, because developers would be lazy and use factory methods that create more than they actually need for the test.
- mscccc 4y ago
  GitHub uses minitest, but same situation. I don't exactly agree that it's a design problem. I think there is a lot of value in testing at a high level (controller tests) and having them touch the full stack (down to the DB). Gives a lot of confidence that the code will work in production. Throwing money at it (running the tests in parallel) pretty much solves it. I have seen FactoryBot use get out of control (creating 8x as many records as you'd expect). That's a really easy way to slow down a test suite :). One way I've found to fix that is by adding tests for the factories, and asserting they are only creating what you want them to. On model tests, another thing GitHub did well was encourage using `.build` when possible to avoid writing the the DB.
  - itake 4y ago
    > I think there is a lot of value in testing at a high level (controller tests) and having them touch the full stack (down to the DB). There is a lot of wasted work. Each (similar) test would insert and then delete the same rows, but make a little tweak. Other languages and frameworks (java, python or golang) don't need to test the db and run super quick. Most db systems do not actually need to be tested. e2e testing can be done manually via curl or qa. > Throwing money at it (running the tests in parallel) pretty much solves it. Tests still are slow locally. > One way I've found to fix that is by adding tests for the factories That is interesting. I had a meta-test that would limit the number of sql queries a test could trigger.