8 ms·

The obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience

The obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience running these databases and didn’t enjoy the prospect of wrestling with the ops overhead of making live replication work and behave well. Other databases like CockroachDB looked very tempting, but we had zero experience with it. And we didn’t want to lock ourselves into a cloud provider with a managed product like Spanner. What about managed mysql/posgresql? no lock-in and installing them locally is trivial.

butlerm 5y ago
If you install it locally, that is pretty much the definition of an unmanaged database. Other than by the user of course.
- alexjurkiewicz 5y ago
  You can use a managed version of the database in production, and an "unmanaged" local Docker image during development.
mrkurt 5y ago
Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.
- jeffffff 5y ago
  they've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.
  - tptacek 5y ago
    "Questionable technology decisions". You're trying to dunk, but I don't think you understand where the hoop is. Their technology decisions have panned out Tailscale well. We should all be so fortunate. This isn't Twitter with the "fail whale"; the only reason you know about any of this stuff is because they wrote about it. They ran their entire service with a JSON file backend for 18 months, and switched from it to etcd without you even knowing about it.
    - jeffffff 5y ago
      i really don't know a ton about this product or team but it sounds like if they had used aurora mysql or aurora postgres in the first place then there would be nothing to write a blog post about because it would've just worked and kept working. they say they want to avoid vendor lock-in but if the vendor became a real issue they'd be doing their first migration instead of being on v3 already. additionally, their bespoke solution relies on s3, which is also a vendor-specific technology, so it seems like they haven't avoided vendor lock in? i've seen many cases of developers doing more work to avoid vendor lock-in than it would take to replatform if it ended up being a necessity, and this really feels like that looking at it from the outside. i'd understand this better if mysql or postgres couldn't solve their problem, but that is not the case here, and i can't wrap my head around a company who is ok with their devs reinventing a very good wheel 3 times when the obvious choice would've worked fine the first time. it seems like they are successful in spite of these decisions, not because of these decisions. https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology
      - tptacek 5y ago
        You should start by learning more about the product, and then tell them they should use Aurora for all their backing store.
        jeffffff 5y ago
- Winsaucerer 5y ago
  I'm having a really difficult time migrating my managed postgresql instance in Google Cloud just to get it to a newer version. I'm down to my last option: shut down everything, export, re-import somewhere else again. I did briefly have logical replication working, and so had my near zero downtime solution in hand. But the replication broke after a few days because I delayed cutting over, and then I couldn't get it going again. I couldn't find out how to debug this either. The lesson I've learned here is that I'll be avoiding managed DB products whenever I can. It takes away control that turns out I sometimes really need.

butlerm 5y ago

If you install it locally, that is pretty much the definition of an unmanaged database. Other than by the user of course.

alexjurkiewicz 5y ago
You can use a managed version of the database in production, and an "unmanaged" local Docker image during development.

mrkurt 5y ago

Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.

jeffffff 5y ago
they've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.
- tptacek 5y ago
  "Questionable technology decisions". You're trying to dunk, but I don't think you understand where the hoop is. Their technology decisions have panned out Tailscale well. We should all be so fortunate. This isn't Twitter with the "fail whale"; the only reason you know about any of this stuff is because they wrote about it. They ran their entire service with a JSON file backend for 18 months, and switched from it to etcd without you even knowing about it.
  - jeffffff 5y ago
    i really don't know a ton about this product or team but it sounds like if they had used aurora mysql or aurora postgres in the first place then there would be nothing to write a blog post about because it would've just worked and kept working. they say they want to avoid vendor lock-in but if the vendor became a real issue they'd be doing their first migration instead of being on v3 already. additionally, their bespoke solution relies on s3, which is also a vendor-specific technology, so it seems like they haven't avoided vendor lock in? i've seen many cases of developers doing more work to avoid vendor lock-in than it would take to replatform if it ended up being a necessity, and this really feels like that looking at it from the outside. i'd understand this better if mysql or postgres couldn't solve their problem, but that is not the case here, and i can't wrap my head around a company who is ok with their devs reinventing a very good wheel 3 times when the obvious choice would've worked fine the first time. it seems like they are successful in spite of these decisions, not because of these decisions. https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology
    - tptacek 5y ago
      You should start by learning more about the product, and then tell them they should use Aurora for all their backing store.
      - jeffffff 5y ago
Winsaucerer 5y ago
I'm having a really difficult time migrating my managed postgresql instance in Google Cloud just to get it to a newer version. I'm down to my last option: shut down everything, export, re-import somewhere else again. I did briefly have logical replication working, and so had my near zero downtime solution in hand. But the replication broke after a few days because I delayed cutting over, and then I couldn't get it going again. I couldn't find out how to debug this either. The lesson I've learned here is that I'll be avoiding managed DB products whenever I can. It takes away control that turns out I sometimes really need.