5 ms·

This post touches on "innovation tokens". While I agree with the premise of "choose boring technology", it feels like a process smell, particularly of a startup

This post touches on "innovation tokens". While I agree with the premise of "choose boring technology", it feels like a process smell, particularly of a startup whose goal is to innovate a techology. Feels demotivating as an engineer if management says our team can only innovate an arbitrary N times.

marcinzm 6y ago
I mean, what do you propose in terms of getting a diverse group of engineers to trade off innovation versus choosing boring technologies? Keeping in mind that how much risk the company is willing to take is not the decision of engineers but the executive team. Innovation tokens convey the level of risk the executive team is willing to take in terms of technologies. The alternative I've often seen is a dictatorial CTO (or VP of Eng) who simply says NO a lot which is a lot more demotivating. A large company may do detailed risk analyses but those are too cumbersome for a startup.
crawshaw 6y ago
It is a tricky tradeoff for startups. On the one hand, a startup has very limited resources and so has to focus on the business. On the other hand, a startup has to experiment to find the business. I don't think there's an easy answer. In our case, the control plane data store really should be as boring as possible. It was real stretch using anything other than MySQL. We tried to lay out the arguments in the post, but the most compelling was we had lots of unit tests that spun up a DB and shut it down quickly. Maybe a hundred tests whose total execution time was 1.5s. The "boring" options made that surprisingly difficult. (Tailscaler and blog post co-author)
- rubenv 6y ago
  For PostgreSQL and Go, here's a package to spin up a temp in-mem PostgreSQL: https://github.com/rubenv/pgtest/ https://github.com/rubenv/pgtest/
  - aidos 6y ago
    From memory, in Postgres, you could also have a copy of your base database and then copy it in for each test, which is seem to recall being fairly fast. It includes the data too. Create database test_db template initial_db;
- marcinzm 6y ago
  I'm used to Scala for these things and it seemed fairly easy to do. Docker DB container would be spun up at the start of testing (using a package such as testcontainers-scala). Then the DB would be setup using the migration scripts (using Flyway) so it has a known state. Every test (or set of tests if they're linked) would have a pre-hook which nuked the DB to a clean state. Then the docker container would be shut down at the end of testing. I'm guessing there's even some way to have this run concurrently with multiple DBs (one per thread) but we never did that. Is Java's ecosystem for this type of tooling just that much better?
  - bradfitz 6y ago
    I wrote about that in the article. Search for "dockertest". We considered that option but didn't like it. It still wasn't fast enough, and placed onerous (or at least annoying) dependencies on future employees.
    - mwcampbell 6y ago
      > It still wasn't fast enough, and placed onerous (or at least annoying) dependencies on future employees. Did you configure the Postgres (or MySQL) database to be entirely in memory, e.g. by using a tmpfs Docker volume? As for being onerous or annoying for new employees, which is worse: having to set up a Docker environment, or using a relatively obscure data store in a way that nobody else does?
      - bradfitz 6y ago
        Empirically, the former. We've since hired many employees who just learned about our database today from this blog post but had been happily testing against it on their laptops for months. 3-4 of us know about it, and that's sufficient.
- eatonphil 6y ago
  One solution is to use a standard subset of sql so you can use sqlite in unit tests and mysql/postgres in prod. Many languages also have in-memory SQL implementations that are also a more convenient substitute for sqlite. Of course the benefit of what you did, even if I wouldn't have done it, is that you're _not_ using a different system in dev vs prod.
  - jrockway 6y ago
    I would not go down the road of figuring out what subset of SQL various database understand. You will always be surprised, and you'll be surprised in production because "the thing" you developed against sqlite doesn't work in postgres. I used to do this and stopped when I noticed that sqlite and postgres treat booleans differently; postgres accepts 't' as true, but SQLite stores a literal 't' in the boolean-typed field. This means you get different results when you read things back out. All in all, not a rabbit hole you want to go down. Personally, I just create a new database for each test against a postgres server on localhost. The startup time is nearly zero, and the accuracy compared to production is nearly 100%.
  - bradfitz 6y ago
    Exactly. That's why I wrote: "No Docker, no mocks, testing what we’d actually use in production."
sbierwagen 6y ago
What is the purpose of a startup? Is it to put keywords on your resume, or is it to create a product?
- ZephyrBlu 6y ago
  Depends if you have funding or not.
rhizome 6y ago
Every decision to increase platform spread should be justified in detail, as the implementation and support overhead is essentially unbounded. Be careful you don't buy a pig in a poke.

marcinzm 6y ago

I mean, what do you propose in terms of getting a diverse group of engineers to trade off innovation versus choosing boring technologies? Keeping in mind that how much risk the company is willing to take is not the decision of engineers but the executive team. Innovation tokens convey the level of risk the executive team is willing to take in terms of technologies. The alternative I've often seen is a dictatorial CTO (or VP of Eng) who simply says NO a lot which is a lot more demotivating. A large company may do detailed risk analyses but those are too cumbersome for a startup.

crawshaw 6y ago

It is a tricky tradeoff for startups. On the one hand, a startup has very limited resources and so has to focus on the business. On the other hand, a startup has to experiment to find the business. I don't think there's an easy answer. In our case, the control plane data store really should be as boring as possible. It was real stretch using anything other than MySQL. We tried to lay out the arguments in the post, but the most compelling was we had lots of unit tests that spun up a DB and shut it down quickly. Maybe a hundred tests whose total execution time was 1.5s. The "boring" options made that surprisingly difficult. (Tailscaler and blog post co-author)

rubenv 6y ago
For PostgreSQL and Go, here's a package to spin up a temp in-mem PostgreSQL: https://github.com/rubenv/pgtest/ https://github.com/rubenv/pgtest/
- aidos 6y ago
  From memory, in Postgres, you could also have a copy of your base database and then copy it in for each test, which is seem to recall being fairly fast. It includes the data too. Create database test_db template initial_db;
marcinzm 6y ago
I'm used to Scala for these things and it seemed fairly easy to do. Docker DB container would be spun up at the start of testing (using a package such as testcontainers-scala). Then the DB would be setup using the migration scripts (using Flyway) so it has a known state. Every test (or set of tests if they're linked) would have a pre-hook which nuked the DB to a clean state. Then the docker container would be shut down at the end of testing. I'm guessing there's even some way to have this run concurrently with multiple DBs (one per thread) but we never did that. Is Java's ecosystem for this type of tooling just that much better?
- bradfitz 6y ago
  I wrote about that in the article. Search for "dockertest". We considered that option but didn't like it. It still wasn't fast enough, and placed onerous (or at least annoying) dependencies on future employees.
  - mwcampbell 6y ago
    > It still wasn't fast enough, and placed onerous (or at least annoying) dependencies on future employees. Did you configure the Postgres (or MySQL) database to be entirely in memory, e.g. by using a tmpfs Docker volume? As for being onerous or annoying for new employees, which is worse: having to set up a Docker environment, or using a relatively obscure data store in a way that nobody else does?
    - bradfitz 6y ago
      Empirically, the former. We've since hired many employees who just learned about our database today from this blog post but had been happily testing against it on their laptops for months. 3-4 of us know about it, and that's sufficient.
eatonphil 6y ago
One solution is to use a standard subset of sql so you can use sqlite in unit tests and mysql/postgres in prod. Many languages also have in-memory SQL implementations that are also a more convenient substitute for sqlite. Of course the benefit of what you did, even if I wouldn't have done it, is that you're _not_ using a different system in dev vs prod.
- jrockway 6y ago
  I would not go down the road of figuring out what subset of SQL various database understand. You will always be surprised, and you'll be surprised in production because "the thing" you developed against sqlite doesn't work in postgres. I used to do this and stopped when I noticed that sqlite and postgres treat booleans differently; postgres accepts 't' as true, but SQLite stores a literal 't' in the boolean-typed field. This means you get different results when you read things back out. All in all, not a rabbit hole you want to go down. Personally, I just create a new database for each test against a postgres server on localhost. The startup time is nearly zero, and the accuracy compared to production is nearly 100%.
- bradfitz 6y ago
  Exactly. That's why I wrote: "No Docker, no mocks, testing what we’d actually use in production."

sbierwagen 6y ago

What is the purpose of a startup? Is it to put keywords on your resume, or is it to create a product?

ZephyrBlu 6y ago
Depends if you have funding or not.

rhizome 6y ago

Every decision to increase platform spread should be justified in detail, as the implementation and support overhead is essentially unbounded. Be careful you don't buy a pig in a poke.