3 ms·

But then you need to have an actual database running. I find the ability to mock the DB entirely very useful to reduce the amount of scaffolding an average deve

But then you need to have an actual database running. I find the ability to mock the DB entirely very useful to reduce the amount of scaffolding an average developer needs to have in order to run the test suite.

thwarted 6y ago
Maintaining the mock is scaffolding, just of a different type.
- ithkuil 6y ago
  Sure, done by a different persona; and that's crucial: Let's imagine (I think) a common scenario: a simple Go project, composed of a number of modules owned by different team members or small groups, and shared packages etc. Now, I need to add some stateful component and I reason whether to use, say, MySQL or something else. If I choose MySQL and I need a real MySQL to run tests against it, now I need to prepare some docker-compose or equivalent scaffolding and shove it down ever other team member's throat. I want them to be able to run all tests without having to stop and think what tests their change (possibly in a shared package) might affect. Sure, CI will catch things. But deferring all tests failure detection to the CI stage adds latency, and often troubleshooting issues that happen on CI is hard if it's hard to rerun the same thing locally. I witnessed a "pressure" towards preferring pure-go solutions so that the team doesn't have to switch to a more "complex" build/test harness. Granted, this is only a problem if you managed so far to do all you needed to do with the pure Go build system (which I have to say, I like very much and I do need to have a pretty good reason before I abandon/"upgrade" to something else).
  - jrockway 6y ago
    I think the scaffolding is worth it. You ask people on your team to run "docker run --restart always --name mysql -e MYSQL_ROOT_PASSWORD=foobar -p 3306:3306 -d mysql:5.whatever", and then you never think about it again. (Also be sure to firewall that off so only localhost can get to your database with a predictable root password -- lots of bots out there that will "hack" this thing in 24 hours. Try it and see!) Compared to maintaining mocks that don't actually detect issues like invalid queries, or missing defaults, or mapping semantics between language types and database types, this is a lot simpler. You just need one thing running, and you only need to set it up once. Every feature that is available to production code is now available to your tests. You can also have your tests launch a database container for you. I found this slow, and that the setup overhead of one instruction in the README was worthwhile. Finally, you might not need the database for every test. If you have Service 1 that depends on the database and Service 2 that only depends on Service 1, writing fakes for Service 1 for the Service 2 tests to use to avoid the database is productive. Service 2 will want to test the error cases for Service 1 anyway, so you will have to have the provision for things like "make the next call to service 1 hang indefinitely", etc. so you will be writing that anyway. Some people like all integration tests to go all the way to the bottom of the stack, but I prefer testing only the boundary in depth and making simpler end-to-end tests. YMMV.
    - ithkuil 6y ago
      Then you change from MySQL to postgres and you have to tell everybody to start another container. Then you have 5 projects, are they all using the same password? Here we're talking about record/replay mocks, which are based on a real database. So, of course somebody will have to run that container with MySQL so that the tests will be run in record mode (e.g. when developing the tests). But, then if your teammates don't develop those tests, they don't technically need a real database. You still want them to run your tests because those tests might depend on some course your teammates touch (e.g. some common library code)
    - deleted 6y ago
      [deleted]
GordonS 6y ago
Compared to just spinning up a DB in a container, I find the amount of setup/scaffolding required to mock the DB to be far greater. For years I took the mocking route, but it means there is a whole class of bugs that your tests won't catch, and the scaffolding was often fragile. Few years back I switched to just running a real DB in a container - very happy with this way.
Nullabillity 6y ago
Not sure about how viable it is in Go, but our Python tests automatically `initdb` and start Postgres as part of the test suite setup, and burns it all down during teardown. Each test inside the suite gets an isolated database inside the cluster. Nothing to manage for the regular application/test developer, no persistent background services, nothing exposed outside of the local machine, no dependency on Docker. You just need to have Postgres on your $PATH, and even that is handled for you by Nix. We use the same approach to transparently also run the same tests against MSSQL, and it is mostly transparent (except for all the ugly hacks involved to run an isolated MSSQL instance outside of Docker container). Yes, there is some scaffolding involved, but it contains zero model logic, and ~never has to be touched again once it works.

thwarted 6y ago

Maintaining the mock is scaffolding, just of a different type.

ithkuil 6y ago
Sure, done by a different persona; and that's crucial: Let's imagine (I think) a common scenario: a simple Go project, composed of a number of modules owned by different team members or small groups, and shared packages etc. Now, I need to add some stateful component and I reason whether to use, say, MySQL or something else. If I choose MySQL and I need a real MySQL to run tests against it, now I need to prepare some docker-compose or equivalent scaffolding and shove it down ever other team member's throat. I want them to be able to run all tests without having to stop and think what tests their change (possibly in a shared package) might affect. Sure, CI will catch things. But deferring all tests failure detection to the CI stage adds latency, and often troubleshooting issues that happen on CI is hard if it's hard to rerun the same thing locally. I witnessed a "pressure" towards preferring pure-go solutions so that the team doesn't have to switch to a more "complex" build/test harness. Granted, this is only a problem if you managed so far to do all you needed to do with the pure Go build system (which I have to say, I like very much and I do need to have a pretty good reason before I abandon/"upgrade" to something else).
- jrockway 6y ago
  I think the scaffolding is worth it. You ask people on your team to run "docker run --restart always --name mysql -e MYSQL_ROOT_PASSWORD=foobar -p 3306:3306 -d mysql:5.whatever", and then you never think about it again. (Also be sure to firewall that off so only localhost can get to your database with a predictable root password -- lots of bots out there that will "hack" this thing in 24 hours. Try it and see!) Compared to maintaining mocks that don't actually detect issues like invalid queries, or missing defaults, or mapping semantics between language types and database types, this is a lot simpler. You just need one thing running, and you only need to set it up once. Every feature that is available to production code is now available to your tests. You can also have your tests launch a database container for you. I found this slow, and that the setup overhead of one instruction in the README was worthwhile. Finally, you might not need the database for every test. If you have Service 1 that depends on the database and Service 2 that only depends on Service 1, writing fakes for Service 1 for the Service 2 tests to use to avoid the database is productive. Service 2 will want to test the error cases for Service 1 anyway, so you will have to have the provision for things like "make the next call to service 1 hang indefinitely", etc. so you will be writing that anyway. Some people like all integration tests to go all the way to the bottom of the stack, but I prefer testing only the boundary in depth and making simpler end-to-end tests. YMMV.
  - ithkuil 6y ago
    Then you change from MySQL to postgres and you have to tell everybody to start another container. Then you have 5 projects, are they all using the same password? Here we're talking about record/replay mocks, which are based on a real database. So, of course somebody will have to run that container with MySQL so that the tests will be run in record mode (e.g. when developing the tests). But, then if your teammates don't develop those tests, they don't technically need a real database. You still want them to run your tests because those tests might depend on some course your teammates touch (e.g. some common library code)
  - deleted 6y ago
    [deleted]

GordonS 6y ago

Compared to just spinning up a DB in a container, I find the amount of setup/scaffolding required to mock the DB to be far greater. For years I took the mocking route, but it means there is a whole class of bugs that your tests won't catch, and the scaffolding was often fragile. Few years back I switched to just running a real DB in a container - very happy with this way.

Nullabillity 6y ago

Not sure about how viable it is in Go, but our Python tests automatically `initdb` and start Postgres as part of the test suite setup, and burns it all down during teardown. Each test inside the suite gets an isolated database inside the cluster. Nothing to manage for the regular application/test developer, no persistent background services, nothing exposed outside of the local machine, no dependency on Docker. You just need to have Postgres on your $PATH, and even that is handled for you by Nix. We use the same approach to transparently also run the same tests against MSSQL, and it is mostly transparent (except for all the ugly hacks involved to run an isolated MSSQL instance outside of Docker container). Yes, there is some scaffolding involved, but it contains zero model logic, and ~never has to be touched again once it works.