3 ms·

> ...you should be using mock external services and databases to keep things running fast. I've become skeptical of this advice, of late. If the database is ju

> ...you should be using mock external services and databases to keep things running fast. I've become skeptical of this advice, of late. If the database is just a dumb object store, then I suppose it's fine. But I've also seen a fair bit of rework resulting from tests not catching a bug that was only reproducible when working against a real database. You then end up needing additional tests to cover that sort of stuff, which takes extra time to develop and maintain. Between that and the whole "test-induced design damage" thing, there's some amount of test slowness I'll accept in return for reducing the development effort needed to maintain the tests. On the general principle that computers' time is way, way cheaper than humans' time. As far as keeping tests that rely on stuff like the database running quickly goes, my new preferred plan of attack is configure those components to use RAM disk in the local dev and CI environments.

bluGill 8y ago
In my experience a real database is not slow. The real database is slow only if the only one you can use is the production one (it should be obvious why this is a bad idea) If you copy the schema and import it into sqlite with just the data you need for the test it is fast. Mocks do have their place, but people get confused about the place. The purpose of a test is to assert "This will never change". That is a very strong statement. With a mock you are asserting that you will always call that function with those parameters - this is rarely what you want.
- bunderbunder 8y ago
  Substituting a different database for testing purposes is right up there with trying to mock the database, in my book. It places big constraints on your development, because you get stuck limiting yourself to the subset of SQL that is supported by both databases. That usually means trapping yourself in the 1990s, which, given everything that been implemented in the major RDBMSes over the past 20-25 years, implies accepting an artificially low ceiling on the performance, and possibly the semantics, that you can get out of your data store. Second, even if you're fine with sticking to a small subset of your database's features, you're still not guaranteed that the two DBMSes will behave the same way given the same input. This is particularly true if one of them is SQLite, which is dynamically typed. With the possible exception of key/value stores, databases generally exhibit such complex behavior that, if you're looking to conclusively verify that logic involving your database actually works, you probably shouldn't accept any substitutes.
  - bluGill 8y ago
    I would argue that sticking to 1990's sql is good anyway as it means the business can switch to a different engine. There are several great databases out there with support, if you limit yourself to standard sql you can throw our your current vendor if they don't give you enough value (which could be support, speed, or even the salesman beats you in golf) I understand sometimes the extensions are worth using, but good engineering is to limit those uses to where it actually is significant.
    - bunderbunder 8y ago
      If you're looking for the ability to switch data storage technologies, you'd do better to just make sure your data storage layer is cleanly isolated. Sticking to 1990s SQL gives you the ability to swap among 5 or 6 different databases, all of which have been around for a day or two, none of which have seen a whole lot of change in their relative strengths and weaknesses in a good long while. It's just not super likely that you'll find a compelling reason to switch among them, especially if the people who made the original choice did their homework first. Keeping a well-isolated data layer, on the other hand, gives you a much wider universe of things you can attainably switch to - document stores, triplestores, graph DBs, Cassandra, Hadoop/Hive, stuff that hasn't been invented yet, etc. I've lived through my fair share of database technology migrations. So far, every single one of them has been this kind, not a SQL -> SQL migration. So, while you're at it, might as well get your money's worth out of the SQL DB.
- pytester 8y ago
  >The purpose of a test is to assert "This will never change". IMO the purpose of test is to assert "this is behavior that will likely happen in production", not "this will never change". I try to use data that's as real as possible within resource constraints (e.g. slashing out boring data to get dump times down from 1 hr to 5 mins). >If you copy the schema and import it into sqlite with just the data you need omg no. I've had undiscovered bugs caused by using a different version of postgres locally in test to the one in prod, never mind using an entirely different database.
- kerng 8y ago
  Some good points, but I'm worried engineers take this as advice to always test with real customer data and even connect to prod from their dev boxes and such. Not a good way to handle PII.

bluGill 8y ago

In my experience a real database is not slow. The real database is slow only if the only one you can use is the production one (it should be obvious why this is a bad idea) If you copy the schema and import it into sqlite with just the data you need for the test it is fast. Mocks do have their place, but people get confused about the place. The purpose of a test is to assert "This will never change". That is a very strong statement. With a mock you are asserting that you will always call that function with those parameters - this is rarely what you want.

bunderbunder 8y ago
Substituting a different database for testing purposes is right up there with trying to mock the database, in my book. It places big constraints on your development, because you get stuck limiting yourself to the subset of SQL that is supported by both databases. That usually means trapping yourself in the 1990s, which, given everything that been implemented in the major RDBMSes over the past 20-25 years, implies accepting an artificially low ceiling on the performance, and possibly the semantics, that you can get out of your data store. Second, even if you're fine with sticking to a small subset of your database's features, you're still not guaranteed that the two DBMSes will behave the same way given the same input. This is particularly true if one of them is SQLite, which is dynamically typed. With the possible exception of key/value stores, databases generally exhibit such complex behavior that, if you're looking to conclusively verify that logic involving your database actually works, you probably shouldn't accept any substitutes.
- bluGill 8y ago
  I would argue that sticking to 1990's sql is good anyway as it means the business can switch to a different engine. There are several great databases out there with support, if you limit yourself to standard sql you can throw our your current vendor if they don't give you enough value (which could be support, speed, or even the salesman beats you in golf) I understand sometimes the extensions are worth using, but good engineering is to limit those uses to where it actually is significant.
  - bunderbunder 8y ago
    If you're looking for the ability to switch data storage technologies, you'd do better to just make sure your data storage layer is cleanly isolated. Sticking to 1990s SQL gives you the ability to swap among 5 or 6 different databases, all of which have been around for a day or two, none of which have seen a whole lot of change in their relative strengths and weaknesses in a good long while. It's just not super likely that you'll find a compelling reason to switch among them, especially if the people who made the original choice did their homework first. Keeping a well-isolated data layer, on the other hand, gives you a much wider universe of things you can attainably switch to - document stores, triplestores, graph DBs, Cassandra, Hadoop/Hive, stuff that hasn't been invented yet, etc. I've lived through my fair share of database technology migrations. So far, every single one of them has been this kind, not a SQL -> SQL migration. So, while you're at it, might as well get your money's worth out of the SQL DB.
pytester 8y ago
>The purpose of a test is to assert "This will never change". IMO the purpose of test is to assert "this is behavior that will likely happen in production", not "this will never change". I try to use data that's as real as possible within resource constraints (e.g. slashing out boring data to get dump times down from 1 hr to 5 mins). >If you copy the schema and import it into sqlite with just the data you need omg no. I've had undiscovered bugs caused by using a different version of postgres locally in test to the one in prod, never mind using an entirely different database.
kerng 8y ago
Some good points, but I'm worried engineers take this as advice to always test with real customer data and even connect to prod from their dev boxes and such. Not a good way to handle PII.