9 ms·

So basically their biggest problem was that node.js machine-gun hammers the filesystem with stat calls when looking for modules. This was made worse by running

So basically their biggest problem was that node.js machine-gun hammers the filesystem with stat calls when looking for modules. This was made worse by running on a shared folder. Vagrant has a mode called rsync-auto which will synchronize directories into a local vm. Or you could just always have the code local to the node process. It makes more sense to have your editor work with remote files. Next biggest problem was relational database. I think that tests requiring a SQL database are usually a bad sign, and the statement "we'd like to use transactions but our orm doesn't support them" makes me sad. To paraphrase Ben Franklin, "those who would give up transactions for a framework deserve neither". Here's some ideas for speeding up your tests, in order from best to worst. - Make most business code rely on a service or interface. Swap that our during tests for a trivial in-memory implementation. - Use sqlite :memory: for tests - TRUNCATE TABLE - Set your database to running with scissors mode (disable fsync) [edit] I just took a look at the ORM used, waterline https://github.com/balderdashy/waterline https://github.com/balderdashy/waterline . Why wasn't step one moving to sails-memory for running tests?

pyre 11y ago
> Use sqlite :memory: for tests Then you run into issues where the functionality works on SQLite, but not on your real database.
- bliti 11y ago
  May not work. Given the use of ORMs these days, this is a smaller issue. Though it still requires attention. Having the tests run quicker allows for better use of the programmers time
  - pyre 11y ago
    > Given the use of ORMs these days, this is a smaller issue The ORM can't abstract away all differences between databases. For example, SQLite doesn't have Numeric type, so all of the Numeric(10, 2) fields on (e.g.) my SQLAlchemy (a Python ORM) models will trigger warnings when saving to a SQLite database.
    - bliti 11y ago
      Hence the sentence that follows after the one quoted. It does require attention. Just a bit less. In a sea of issues, reducing the importance of any might probably ease development in some way.
- sbov 11y ago
  Where I work, tests have two modes: quick and long. Quick mode gets in memory treatment. Long mode doesn't. Anything that goes live will undergo a long mode test beforehand.
  - hinkley 11y ago
    We would run those tests a couple times a week, and the memory tests idealy on every successful build (we chained the builds so we never bothered to run the DB tests if the unit tests had already failed). Strictly speaking it's also possible to get subtle differences in behavior across OSes. Especially if you have a race condition you don't know about yet. I always tend to encourage more diversity (carefully spent).
- Osiris 11y ago
  I run a script that mounts a RAM drive and copies my postgresql database into the drive. I use tablespace to force Postgresql to look at the RAM drive for the database data. That way I am testing my queries against the same database engine but get the benefit of avoiding disk IO.
- curun1r 11y ago
  SQLite might be the easiest to configure, but some other RDBMSs have an option to run in memory too (e.g. MySQL's MEMORY storage engine). Also, as a last resort, you can always configure the DB to write to an in-memory file system. This strategy is, in essence, mocking the FS layer in DB tests.
  - TylerE 11y ago
    Often those modes don't have the same semantics, which is dangerous for testing.
    - coldtea 11y ago
      After some point it's diminishing returns though. Obviously nothing can be 100% tested.
  - TylerE 11y ago
    Often those modes don't have the same semantics, which is dangerous for testing.
  - TylerE 11y ago
    Often those modes don't have the same semantics, which is dangerous for testing.
- pilif 11y ago
  you assume that SQLite supports all the querying features I'm using (hint: it doesn't).
  - pyre 11y ago
    Well, those sorts of things cause the tests to immediately fail, and you realize the error of your ways. It's more deadly when you tests pass, but behaviour on production is different.
EugeneOZ 11y ago
Disagree about DB tests. If you want to test queries, then it's bad idea to mock DB. If you don't want to test queries, don't call your tests coverage full.
- narrator 11y ago
  The best way to do db testing is with an empty db and in a specialized dao test. Try to make these dao tests non-overlapping so that the dao test is the only test that hits that part of the schema. Also, if your dao tests are the only thing that hit I/O you can safely do all the other non-integration tests concurrently. Service layer tests should have mocks of the dao and web service calls. You should do most of your testing here and with unit tests for tricky algorithms. Integration tests should hit the full stack, but those should run on a continuous integration server asynchronously and also take place in their own empty database. The key to dao and integration testing is the empty database. Don't let it fill up with test crud between tests. PostgreSQL is pretty fast when your test db only has a couple of rows.
  - raducu 11y ago
    This is by far my favorite way to test; not something I invented but something I picked up start with an EMPTY DATABASE (with tables setup), then every test: begins a transaction,asserts the db is empty, sets up the data (via factories and DAOs), performs its queries (via DAOs) rolls back the transaction for the next test.
    - lmm 11y ago
      It's a fast approach that gives some test coverage, but some of the most important things to test are around the interaction of multiple transactions, which you can't do if you're using a single big transaction to run your test.
  - mattmanser 11y ago
    An empty DB? Some major bugs are caused by people not testing their code with enough data.
kevinburke 11y ago
Hi Shane, thanks for the tip about sails-memory. We've seen a lot of inconsistencies in the way Waterline behaves, so we're hesitant to run it against a different backend than the one we use in production. We'd also lose all of the uniqueness constraints that we use in Postgres. In general we're trying to write and run more tests that only create and manipulate objects in memory, especially for testing business logic. Unfortunately we still have to have some level of integration with the database to test things at a higher system level. Re: transactions, most of our database code uses the ORM, so we thought other test speedups would give a bigger benefit. We recently added an interface for getting/querying with a `pg` connection directly and we're using that to implement transactions in a few critical places. I'm planning to release the code soon - hopefully in a blog post in a few weeks.
deleted 11y ago
[deleted]
MichaelGG 11y ago
There should be some optimizer for npm modules. The current system is silly. Popcorn Time installs around 5000 files as part of its node_modules. How is that useful? There should be some way to combine things into a single module/file or otherwise glob it all together, while removing pointless test, docs, and Adventure Time animated gifs from your deps folder.
- jessaustin 11y ago
  Apparently imminent npm v3 "flattens" node_modules/, which should help with this problem. At the same time I worry that version compatibility might become undecidable... Although I'm sure there are module maintainers shipping AT gifs, we can't really blame npm for that.
  - 131hn 11y ago
    By its design _browserify_ can be very powerfull on the node space (there are dedicated options in this way). We use it a lot (with grunt & co) to pack & deploy production code (with excellent result)
afarrell 11y ago
Django also issues lots to stat()s when starting up their test framework.
elliotf 11y ago
Running against a local mysql instance running on a ramdisk and emptying the DB tables between tests yields "fast enough" performance while avoiding test pollution: ~300 full-stack scenarios in ~5sec start to finish.
pekk 11y ago
In an alternate history, what would have been a good way for node.js to reduce the number of stat calls? What would the tradeoff be?
- kevinburke 11y ago
  May want to check out the patch - after a syscall in a directory failed because the directory didn't exist, node would issue 4-5 additional stat() calls for the other filetypes (.js, .index.json, .json, .coffee, etc) even though the directory didn't exist. Solution was to abort after the first failed stat(). https://github.com/nodejs/io.js/pull/1920 https://github.com/nodejs/io.js/pull/1920
kevinburke 11y ago
Should add, a problem with `rsync-auto` is some files in node_modules (bcrypt, among others) are compiled at runtime, and produce different outputs for Mac/Linux - you can't just copy them from Mac to Linux and expect it to work. Probably some things we could do to hack around this but it didn't seem worth it.
perlgeek 11y ago
Other options for databases: - Run all your tests in a transaction, roll it back in the end (also avoids writes to disk in most scenarios) - Put the DB file system on a RAM disk

pyre 11y ago

> Use sqlite :memory: for tests Then you run into issues where the functionality works on SQLite, but not on your real database.

bliti 11y ago
May not work. Given the use of ORMs these days, this is a smaller issue. Though it still requires attention. Having the tests run quicker allows for better use of the programmers time
- pyre 11y ago
  > Given the use of ORMs these days, this is a smaller issue The ORM can't abstract away all differences between databases. For example, SQLite doesn't have Numeric type, so all of the Numeric(10, 2) fields on (e.g.) my SQLAlchemy (a Python ORM) models will trigger warnings when saving to a SQLite database.
  - bliti 11y ago
    Hence the sentence that follows after the one quoted. It does require attention. Just a bit less. In a sea of issues, reducing the importance of any might probably ease development in some way.
sbov 11y ago
Where I work, tests have two modes: quick and long. Quick mode gets in memory treatment. Long mode doesn't. Anything that goes live will undergo a long mode test beforehand.
- hinkley 11y ago
  We would run those tests a couple times a week, and the memory tests idealy on every successful build (we chained the builds so we never bothered to run the DB tests if the unit tests had already failed). Strictly speaking it's also possible to get subtle differences in behavior across OSes. Especially if you have a race condition you don't know about yet. I always tend to encourage more diversity (carefully spent).
Osiris 11y ago
I run a script that mounts a RAM drive and copies my postgresql database into the drive. I use tablespace to force Postgresql to look at the RAM drive for the database data. That way I am testing my queries against the same database engine but get the benefit of avoiding disk IO.
curun1r 11y ago
SQLite might be the easiest to configure, but some other RDBMSs have an option to run in memory too (e.g. MySQL's MEMORY storage engine). Also, as a last resort, you can always configure the DB to write to an in-memory file system. This strategy is, in essence, mocking the FS layer in DB tests.
- TylerE 11y ago
  Often those modes don't have the same semantics, which is dangerous for testing.
  - coldtea 11y ago
    After some point it's diminishing returns though. Obviously nothing can be 100% tested.
- TylerE 11y ago
  Often those modes don't have the same semantics, which is dangerous for testing.
- TylerE 11y ago
  Often those modes don't have the same semantics, which is dangerous for testing.
pilif 11y ago
you assume that SQLite supports all the querying features I'm using (hint: it doesn't).
- pyre 11y ago
  Well, those sorts of things cause the tests to immediately fail, and you realize the error of your ways. It's more deadly when you tests pass, but behaviour on production is different.

EugeneOZ 11y ago

Disagree about DB tests. If you want to test queries, then it's bad idea to mock DB. If you don't want to test queries, don't call your tests coverage full.

narrator 11y ago
The best way to do db testing is with an empty db and in a specialized dao test. Try to make these dao tests non-overlapping so that the dao test is the only test that hits that part of the schema. Also, if your dao tests are the only thing that hit I/O you can safely do all the other non-integration tests concurrently. Service layer tests should have mocks of the dao and web service calls. You should do most of your testing here and with unit tests for tricky algorithms. Integration tests should hit the full stack, but those should run on a continuous integration server asynchronously and also take place in their own empty database. The key to dao and integration testing is the empty database. Don't let it fill up with test crud between tests. PostgreSQL is pretty fast when your test db only has a couple of rows.
- raducu 11y ago
  This is by far my favorite way to test; not something I invented but something I picked up start with an EMPTY DATABASE (with tables setup), then every test: begins a transaction,asserts the db is empty, sets up the data (via factories and DAOs), performs its queries (via DAOs) rolls back the transaction for the next test.
  - lmm 11y ago
    It's a fast approach that gives some test coverage, but some of the most important things to test are around the interaction of multiple transactions, which you can't do if you're using a single big transaction to run your test.
- mattmanser 11y ago
  An empty DB? Some major bugs are caused by people not testing their code with enough data.

kevinburke 11y ago

Hi Shane, thanks for the tip about sails-memory. We've seen a lot of inconsistencies in the way Waterline behaves, so we're hesitant to run it against a different backend than the one we use in production. We'd also lose all of the uniqueness constraints that we use in Postgres. In general we're trying to write and run more tests that only create and manipulate objects in memory, especially for testing business logic. Unfortunately we still have to have some level of integration with the database to test things at a higher system level. Re: transactions, most of our database code uses the ORM, so we thought other test speedups would give a bigger benefit. We recently added an interface for getting/querying with a `pg` connection directly and we're using that to implement transactions in a few critical places. I'm planning to release the code soon - hopefully in a blog post in a few weeks.

deleted 11y ago

[deleted]

MichaelGG 11y ago

There should be some optimizer for npm modules. The current system is silly. Popcorn Time installs around 5000 files as part of its node_modules. How is that useful? There should be some way to combine things into a single module/file or otherwise glob it all together, while removing pointless test, docs, and Adventure Time animated gifs from your deps folder.

jessaustin 11y ago
Apparently imminent npm v3 "flattens" node_modules/, which should help with this problem. At the same time I worry that version compatibility might become undecidable... Although I'm sure there are module maintainers shipping AT gifs, we can't really blame npm for that.
- 131hn 11y ago
  By its design _browserify_ can be very powerfull on the node space (there are dedicated options in this way). We use it a lot (with grunt & co) to pack & deploy production code (with excellent result)

afarrell 11y ago

Django also issues lots to stat()s when starting up their test framework.

elliotf 11y ago

Running against a local mysql instance running on a ramdisk and emptying the DB tables between tests yields "fast enough" performance while avoiding test pollution: ~300 full-stack scenarios in ~5sec start to finish.

pekk 11y ago

In an alternate history, what would have been a good way for node.js to reduce the number of stat calls? What would the tradeoff be?

kevinburke 11y ago
May want to check out the patch - after a syscall in a directory failed because the directory didn't exist, node would issue 4-5 additional stat() calls for the other filetypes (.js, .index.json, .json, .coffee, etc) even though the directory didn't exist. Solution was to abort after the first failed stat(). https://github.com/nodejs/io.js/pull/1920 https://github.com/nodejs/io.js/pull/1920

kevinburke 11y ago

Should add, a problem with `rsync-auto` is some files in node_modules (bcrypt, among others) are compiled at runtime, and produce different outputs for Mac/Linux - you can't just copy them from Mac to Linux and expect it to work. Probably some things we could do to hack around this but it didn't seem worth it.

perlgeek 11y ago

Other options for databases: - Run all your tests in a transaction, roll it back in the end (also avoids writes to disk in most scenarios) - Put the DB file system on a RAM disk