4 ms·

In the old methodology, any schema change requires the entire application to be shut down while a team of experts (or one very overworked DBA) modifies the sche

In the old methodology, any schema change requires the entire application to be shut down while a team of experts (or one very overworked DBA) modifies the schema manually. WHAT ? I'm working pretty much in an environment described in the article. Big bank, complicated database setup with multiple replicates and databases up to a size of ~25TB. On a (pre defined) release weekend there are essentially two entities to be released: The application (obviously) and the database changes. For any DB change a patch is created and tested through an environment of half a dozen instances. If bugs are found on already deployed db patches they are never fixed in the patch, but a subsequent patch (also tested through all environments) is created. Needles to say that anything provided, which applies database changes is version controlled. The idea to apply database patches manually is so damn outlandish to beggar believe. Anybody doing this on a production database without a strictly controlled (and tested!) break-fix should be fired on the spot.

edoceo 8y ago
Maybe training rather than termination?
- C1sc0cat 8y ago
  They should have never been put in that position in the first place.
- CaptainZapp 8y ago
  I'd consider this such a gross case of malpractice on the level of a surgeon forgetting some instruments in patient, before sewing her shut. Errors and mistakes happen, no question about that. And any ol' mistake certainly shouldn't be a firing offense. But this behaviour is so grossly incompetent that I don't believe that it can't be helped with any training.
  - geofft 8y ago
    Surgeons are generally trained with a grueling multi-year academic and practical program about how to conduct surgery. DBAs aren't. If you're lucky you have someone who has been trained to be a DBA professionally - but I've never worked with one, all the SQL I've interacted with has been run by people with unrelated job titles who learned how to operate databases by Googling when necessary. You're welcome to fire people if you think it's going to help. I'm just curious how you think a skilled, trained DBA is going to show up at your desk when that happens.
    - C1sc0cat 8y ago
      You pay them money :-) though think the guy that was our DBA at BT has now retired - his first boss was Diskaja at NPL labs
  - icedchai 8y ago
    At a bank, yes. However, for a smaller startup firm, no. (The truth is, for most of this stuff, most of the time, it just doesn't matter.)
    - C1sc0cat 8y ago
      It just doesn't matter until it does.
      - 0xFACEFEED 8y ago
        Which is fine if failing is cheap. Depends on the risk profile.
- LaGrange 8y ago
  Maybe training, but definitely not keeping the position. And, to be frank, if they had any experience, I'd consider that a sign of negligence.
bsg75 8y ago
> In the old methodology, any schema change requires the entire application to be shut down MySQL perhaps, when even the addition of nullable columns without defaults or constraints requires a full table lock and copy? That seems to be a common reason for stating online schema migrations are difficult (not impossible), but not a problem with many other RDBMS engines.
- dub 8y ago
  While it's clear that the author of the linked article isn't familiar with MySQL administration, it's worth noting that the most recent versions of MySQL finally support instant column adds: https://mysqlserverteam.com/mysql-8-0-innodb-now-supports-instant-add-column/ https://mysqlserverteam.com/mysql-8-0-innodb-now-supports-in... That won't save us from needing extra tooling like pt-online-schema-change or gh-ost for other types of schema changes in high availability environments, but it's a step in the right direction.
- VintageCool 8y ago
  MySQL has supported algorithm=online since the release of MySQL 5.6 in 2013. MySQL builds a shadow copy off the new table in the background, then swaps the new table in for the old table once the copy is complete.
unethical_ban 8y ago
There are some banks that won't have DBAs pretty soon if you're in charge, for better or worse.
- Novashi 8y ago
  And medical records software In fact, the entire hospital had plans for multi-hour downtimes due to releases, not to mention department-specific contingency plans if the downtime took longer than expected.
- devonkim 8y ago
  Almost every DBA (both administrator and architect) I’ve worked with was opposed to the workflow that most developers like to use - schema migration tools like Liquibase and FlywayDB along with continuous deployment. I have problems with this approach only if there’s poor testing around database changes and this is the default in most companies. For example, adding an index to a column can write lock an entire table and cause a site outage as a result. On most developer test databases this is accomplished within a few seconds usually in which time most test suites won’t notice the issue. And because most development environments don’t look anything like production for cost reasons, you’re incurring a risk there. Instead of fixing the technical root of why there’s so much risk, most companies add the most possibly expensive resource out there to mitigate or understand it - a person. The more dysfunctional, the more people get piled on while nothing actually is fixed (this is a larger cultural enterprise problem that goes beyond database changes). Most decent data tier engineers I know don’t like to babysit DB changes yet almost everyone I’ve worked with had that as a large percentage of their time spent. Such a waste of time and talent. But then there’s a few DBAs I’ve seen that basically don’t do anything productive by making everything that goes wrong a developer problem, refusing to help troubleshoot (even when the DB in question is not accessible to developers), and other not very devops-y behaviors (hostility-first operations that is). It’d be one thing if the DBs had some huge changes happen, but usually I’ve seen these folks hardly do anything visible to me as another infrastructure guy and the troubleshooting I wind up doing on behalf of powerless developers is something that would have taken someone more knowledgeable less than 5 minutes with a quick trace of DB queries in flight and some historical profiling. But being change review Charlie is something that a lot of people do that, while necessary to some degree, becomes their entire job. That’s just frighteningly dull work to me.
justjash 8y ago
In my first job out of college I was required to do the after hours prod releases with another guy (we were the 2 youngest on the team...). We did quarterly releases most of the time and did everything manually from building/packaging the web app to running the DB change scripts, etc. Needless to say, when the other guy left for a different job I was left alone to do these releases and left a month or two later.
manish_gill 8y ago
> Anybody doing this on a production database without a strictly controlled (and tested!) break-fix should be fired on the spot. I agree with almost everything you said except this line. Usually, people with production access are (or should be) sensible enough not to touch the prod db with anything untested that might mutate things. If they're not sensible enough for that, they shouldn't have had access in the first place, and instead of placing blame and firing people, I'd start thinking about /why/ that happened. Points to a more fundamental flaw in the Engineering. Firing won't fix anything. Also note that the statement in question talks about an old methodology. As for the article, good stuff. I enjoyed reading it.
deleted 8y ago
[deleted]

edoceo 8y ago

Maybe training rather than termination?

C1sc0cat 8y ago
They should have never been put in that position in the first place.
CaptainZapp 8y ago
I'd consider this such a gross case of malpractice on the level of a surgeon forgetting some instruments in patient, before sewing her shut. Errors and mistakes happen, no question about that. And any ol' mistake certainly shouldn't be a firing offense. But this behaviour is so grossly incompetent that I don't believe that it can't be helped with any training.
- geofft 8y ago
  Surgeons are generally trained with a grueling multi-year academic and practical program about how to conduct surgery. DBAs aren't. If you're lucky you have someone who has been trained to be a DBA professionally - but I've never worked with one, all the SQL I've interacted with has been run by people with unrelated job titles who learned how to operate databases by Googling when necessary. You're welcome to fire people if you think it's going to help. I'm just curious how you think a skilled, trained DBA is going to show up at your desk when that happens.
  - C1sc0cat 8y ago
    You pay them money :-) though think the guy that was our DBA at BT has now retired - his first boss was Diskaja at NPL labs
- icedchai 8y ago
  At a bank, yes. However, for a smaller startup firm, no. (The truth is, for most of this stuff, most of the time, it just doesn't matter.)
  - C1sc0cat 8y ago
    It just doesn't matter until it does.
    - 0xFACEFEED 8y ago
      Which is fine if failing is cheap. Depends on the risk profile.
LaGrange 8y ago
Maybe training, but definitely not keeping the position. And, to be frank, if they had any experience, I'd consider that a sign of negligence.

bsg75 8y ago

> In the old methodology, any schema change requires the entire application to be shut down MySQL perhaps, when even the addition of nullable columns without defaults or constraints requires a full table lock and copy? That seems to be a common reason for stating online schema migrations are difficult (not impossible), but not a problem with many other RDBMS engines.

dub 8y ago
While it's clear that the author of the linked article isn't familiar with MySQL administration, it's worth noting that the most recent versions of MySQL finally support instant column adds: https://mysqlserverteam.com/mysql-8-0-innodb-now-supports-instant-add-column/ https://mysqlserverteam.com/mysql-8-0-innodb-now-supports-in... That won't save us from needing extra tooling like pt-online-schema-change or gh-ost for other types of schema changes in high availability environments, but it's a step in the right direction.
VintageCool 8y ago
MySQL has supported algorithm=online since the release of MySQL 5.6 in 2013. MySQL builds a shadow copy off the new table in the background, then swaps the new table in for the old table once the copy is complete.

unethical_ban 8y ago

There are some banks that won't have DBAs pretty soon if you're in charge, for better or worse.

Novashi 8y ago
And medical records software In fact, the entire hospital had plans for multi-hour downtimes due to releases, not to mention department-specific contingency plans if the downtime took longer than expected.
devonkim 8y ago
Almost every DBA (both administrator and architect) I’ve worked with was opposed to the workflow that most developers like to use - schema migration tools like Liquibase and FlywayDB along with continuous deployment. I have problems with this approach only if there’s poor testing around database changes and this is the default in most companies. For example, adding an index to a column can write lock an entire table and cause a site outage as a result. On most developer test databases this is accomplished within a few seconds usually in which time most test suites won’t notice the issue. And because most development environments don’t look anything like production for cost reasons, you’re incurring a risk there. Instead of fixing the technical root of why there’s so much risk, most companies add the most possibly expensive resource out there to mitigate or understand it - a person. The more dysfunctional, the more people get piled on while nothing actually is fixed (this is a larger cultural enterprise problem that goes beyond database changes). Most decent data tier engineers I know don’t like to babysit DB changes yet almost everyone I’ve worked with had that as a large percentage of their time spent. Such a waste of time and talent. But then there’s a few DBAs I’ve seen that basically don’t do anything productive by making everything that goes wrong a developer problem, refusing to help troubleshoot (even when the DB in question is not accessible to developers), and other not very devops-y behaviors (hostility-first operations that is). It’d be one thing if the DBs had some huge changes happen, but usually I’ve seen these folks hardly do anything visible to me as another infrastructure guy and the troubleshooting I wind up doing on behalf of powerless developers is something that would have taken someone more knowledgeable less than 5 minutes with a quick trace of DB queries in flight and some historical profiling. But being change review Charlie is something that a lot of people do that, while necessary to some degree, becomes their entire job. That’s just frighteningly dull work to me.

justjash 8y ago

In my first job out of college I was required to do the after hours prod releases with another guy (we were the 2 youngest on the team...). We did quarterly releases most of the time and did everything manually from building/packaging the web app to running the DB change scripts, etc. Needless to say, when the other guy left for a different job I was left alone to do these releases and left a month or two later.

manish_gill 8y ago

> Anybody doing this on a production database without a strictly controlled (and tested!) break-fix should be fired on the spot. I agree with almost everything you said except this line. Usually, people with production access are (or should be) sensible enough not to touch the prod db with anything untested that might mutate things. If they're not sensible enough for that, they shouldn't have had access in the first place, and instead of placing blame and firing people, I'd start thinking about /why/ that happened. Points to a more fundamental flaw in the Engineering. Firing won't fix anything. Also note that the statement in question talks about an old methodology. As for the article, good stuff. I enjoyed reading it.

deleted 8y ago

[deleted]