3 ms·

What is good for the goose is not necessarily good for the gander. Obviously for startups, you're 100% right. Just announce a brief downtime and/or do migratio

What is good for the goose is not necessarily good for the gander. Obviously for startups, you're 100% right. Just announce a brief downtime and/or do migrations after-hours. Keep it simple, no one will care if their requests timeout once every week for 30 seconds. If your company has hundreds of developers making changes across every timezone and downtime (or developers being blocked waiting for scheduled merge windows) costs real money or creates real problems other than optics, something like this or Vitess (MySQL) is definitely worth it. Engineering should not be a "one-size-fits-all" type of job, and while I do love postgres, my main gripe with the community is that the "keep it simple stupid" mentality persists well beyond its sell-by date in many cases.

ltbarcly3 2y ago
The bigger and more important your company is, the less you should rely on a tool like this. You have more budget to invest in operations and less tolerance for being down for 3 days when a tool like this has a bug that takes your site down. You should hire DBA's and operations staff that understand how to apply db migrations in a safe way, and have them review them before and/or apply them during deployments. It's not hard to do manually, just a little bit of extra work (not that much extra). With a little bit of feedback engineers will learn the basics of how to write migrations that are safe and then the use case for this product is dramatically reduced anyway. You keep things simple because you need to be able to understand what is going on to work with it later. pgroll is inherently complex and poorly designed, but even if it wasn't it is still bad to use something that you can't reasonably correct the problems it causes when it breaks.
- erulabs 2y ago
  You may be right about pgroll specifically, I haven't looked at it closely. However, you can't really say "just do db migrations in a safe way". The way your DBA staff would "apply migrations safely" would be to use ghost-tables and views and triggers and locks - ie: they would write pt-online-schema-change or gh-ost or VReplication or, well, pgroll. These tools were born at places like Facebook or Github by the team responsible for applying migrations safely. The argument that "all software has bugs" applies to both the database itself as well as the software you're writing on top. Hence why "reversible" is the 2nd selling-point here.
  - thayne 2y ago
    And if you don't use a tool like that (whether off the shelf, or home built), then your DBAs will be a bottleneck to any schema changes, which will lead to them being overworked and stressed, which will lead to them making more mistakes.
    - ltbarcly3 2y ago
      I think you are way out of your realm of experience here. Nope, schema changes are not that common compared to other tasks and will only account for a small fraction of your DBA's day. I can say this from copious (over 20 years) experience. They will make FAR fewer mistakes than some random engineer. It takes less time (like a fraction, 1/20 or less) to validate and give feedback on schema changes than it does to untangle the mess that gets made otherwise, restore data from backups, go to meetings to explain outages to executives, etc etc so if you want to keep your DBA's happy and able to sign out of work at a reasonable hour, make sure they get to approve schema changes before they go out. Not to mention that very few engineers have a good understanding of schema design or how to write performant schemas, how to index data, etc. Getting a good migration the first time is just so much less painful and time consuming than trying to deal with bad ones.
      - thayne 2y ago
        > It takes less time (like a fraction, 1/20 or less) to validate and give feedback on schema changes But that's the thing, with tools like this, pt-online-schema-change etc., it is reasonable for a DBA to review changes written by other engineers. Without them, when you have extremely large data sets, it is necessary to do magic with triggers, views, etc. to safely make certain kinds of changes. And doing that is beyond the experience of most non-dba engineers, and has a lot of nuance to it. I'm not saying DBAs shouldn't have to approve schema changes. I'm saying they shouldn't have to write complicated migration scripts by hand. > I think you are way out of your realm of experience here. Or maybe my experience is different than yours.
        ltbarcly3 2y ago
emmelaich 2y ago
I've found in practice, over many years in many industries that downtime is actually much easier to schedule and makes many things far far easier.

ltbarcly3 2y ago

The bigger and more important your company is, the less you should rely on a tool like this. You have more budget to invest in operations and less tolerance for being down for 3 days when a tool like this has a bug that takes your site down. You should hire DBA's and operations staff that understand how to apply db migrations in a safe way, and have them review them before and/or apply them during deployments. It's not hard to do manually, just a little bit of extra work (not that much extra). With a little bit of feedback engineers will learn the basics of how to write migrations that are safe and then the use case for this product is dramatically reduced anyway. You keep things simple because you need to be able to understand what is going on to work with it later. pgroll is inherently complex and poorly designed, but even if it wasn't it is still bad to use something that you can't reasonably correct the problems it causes when it breaks.

erulabs 2y ago
You may be right about pgroll specifically, I haven't looked at it closely. However, you can't really say "just do db migrations in a safe way". The way your DBA staff would "apply migrations safely" would be to use ghost-tables and views and triggers and locks - ie: they would write pt-online-schema-change or gh-ost or VReplication or, well, pgroll. These tools were born at places like Facebook or Github by the team responsible for applying migrations safely. The argument that "all software has bugs" applies to both the database itself as well as the software you're writing on top. Hence why "reversible" is the 2nd selling-point here.
- thayne 2y ago
  And if you don't use a tool like that (whether off the shelf, or home built), then your DBAs will be a bottleneck to any schema changes, which will lead to them being overworked and stressed, which will lead to them making more mistakes.
  - ltbarcly3 2y ago
    I think you are way out of your realm of experience here. Nope, schema changes are not that common compared to other tasks and will only account for a small fraction of your DBA's day. I can say this from copious (over 20 years) experience. They will make FAR fewer mistakes than some random engineer. It takes less time (like a fraction, 1/20 or less) to validate and give feedback on schema changes than it does to untangle the mess that gets made otherwise, restore data from backups, go to meetings to explain outages to executives, etc etc so if you want to keep your DBA's happy and able to sign out of work at a reasonable hour, make sure they get to approve schema changes before they go out. Not to mention that very few engineers have a good understanding of schema design or how to write performant schemas, how to index data, etc. Getting a good migration the first time is just so much less painful and time consuming than trying to deal with bad ones.
    - thayne 2y ago
      > It takes less time (like a fraction, 1/20 or less) to validate and give feedback on schema changes But that's the thing, with tools like this, pt-online-schema-change etc., it is reasonable for a DBA to review changes written by other engineers. Without them, when you have extremely large data sets, it is necessary to do magic with triggers, views, etc. to safely make certain kinds of changes. And doing that is beyond the experience of most non-dba engineers, and has a lot of nuance to it. I'm not saying DBAs shouldn't have to approve schema changes. I'm saying they shouldn't have to write complicated migration scripts by hand. > I think you are way out of your realm of experience here. Or maybe my experience is different than yours.
      - ltbarcly3 2y ago

emmelaich 2y ago

I've found in practice, over many years in many industries that downtime is actually much easier to schedule and makes many things far far easier.