5 ms·

Why wouldn't you do that in the dev database, then push to test, then push to prod?

by gnu8 9y ago

LandR 9y ago
We dont' have a dev / test / live database. All dev and testing is dont against the live database. YOu just have to be very, very, very careful. It's something we are trying to implement now, honestly. It's on the TODO list. The problem I'm having is where do you start? How do you simulate the data flowing through live in the test / dev enviornment.
- thehardsphere 9y ago
  Here's two ideas: 1. Set up replication, with your test db the replication slave of your production database, such that changes in production are mirrored in testing nearly instantly. 2. Take your production database backups (you do have those, right?) and restore them to the test database. This is less fresh than a replication setup but if you're just doing functional testing it should be enough to say "Well, okay, this little change isn't going to blow everything up."
- j_s 9y ago
  Upvotes for honest anecdata and asking for advice! > How do you simulate the data flowing through live in the test / dev enviornment. Most don't, rolling with the schema and 'minimum viable' data flow instead. Make darn sure your backups work by testing restores regularly. Restoring a production backup elsewhere may be an option if it's possible to avoid sensitive data (scrubbing afterward introduces risk of missing something, and should be scripted/repeatable). Exactly matching production data flow is not as important as avoiding accidentally hosing the prod db.
- EGreg 9y ago
  Let me tell you from my experience. One time I was editing code on the live server using SFTP, before I knew about version control. And the whole thing was deleted, with no way to get it back. It doesn't take that long to put some safeguards. Here is the thing: WHENEVER you say "you have to be VERY careful" or blame someone for making a mistake, that's a place you need to put more checks. The computer should check routine things, not people. "The problem I'm having is where do you start? How do you simulate the data flowing through live in the test / dev enviornment." I am guessing you use MySQL so simply set up MYSQL REPLICATION and test all your stuff on the replica database first. Actually you should periodically dump the replica and test on a snapshot, so you can mess up the snapshot. Main db -> Replica -> Snapshot.
  - LandR 9y ago
    It's a SQLSERVER, which at this point is about 200GB. Can I say just replicate the schema and then run a script to populate some test data. Or just move over a subset of the main DB? Got any good articles on how to setup test environments. I would then need all the dev code to point to the test one, not the live db and any Web APIs to be able to switch to point to the test db (which must mean running 2 web apis, a live and a test one...) Seems like it gets very complicated, but probably worth figuring out. We don't even have CI yet.
    - thehardsphere 9y ago
      Yes, typically when testing web applications, you have a second instance of the application running on a different machine than the production server, configured to run against a different database server as well. The exact details of how to do this well depend on how your application is written and its requirements. If all you want to do is functional testing of the application by yourself, then it is OK to deploy a copy of the software on your desktop along with a test database. I do this for the application I work on, and when I need to test against MSSQL, the Express edition is good enough. If you need to do load testing or integration testing or other more complicated forms of testing, you usually try to duplicated the expected production environment as well as you can. This may not be possible, so you may have to make decisions about what's most relevant and simply accept those tradeoffs. EDIT: Don't worry about CI or anything else that fancy and complicated for now. You may not even need it; that depends on your team and how often your application changes. Just for now take the simplest steps you can until you get this basic stuff figured out first.
btschaegg 9y ago
Not to excuse anything here, but you wouldn't believe how _insane_ some business environments are when it comes to databases. And yes, by that I mean those systems that you wouldn't believe anyone would ever want to risk losing. Depending on what you're working on, chances are there even are people who use _security_ as an excuse to basically prevent any halfway useful emergency strategy one could think of when handling data impossible to pull off (oh the irony).
- gnu8 9y ago
  Availability is one leg of the CIA triad. Security is the justification for having backups and a recovery strategy, not an excuse for ignoring it.

Why wouldn't you do that in the dev database, then push to test, then push to prod?

by gnu8 9y ago

Why wouldn't you do that in the dev database, then push to test, then push to prod?

We dont' have a dev / test / live database. All dev and testing is dont against the live database. YOu just have to be very, very, very careful. It's something we are trying to implement now, honestly. It's on the TODO list. The problem I'm having is where do you start? How do you simulate the data flowing through live in the test / dev enviornment.

thehardsphere 9y ago
Here's two ideas: 1. Set up replication, with your test db the replication slave of your production database, such that changes in production are mirrored in testing nearly instantly. 2. Take your production database backups (you do have those, right?) and restore them to the test database. This is less fresh than a replication setup but if you're just doing functional testing it should be enough to say "Well, okay, this little change isn't going to blow everything up."
j_s 9y ago
Upvotes for honest anecdata and asking for advice! > How do you simulate the data flowing through live in the test / dev enviornment. Most don't, rolling with the schema and 'minimum viable' data flow instead. Make darn sure your backups work by testing restores regularly. Restoring a production backup elsewhere may be an option if it's possible to avoid sensitive data (scrubbing afterward introduces risk of missing something, and should be scripted/repeatable). Exactly matching production data flow is not as important as avoiding accidentally hosing the prod db.
EGreg 9y ago
Let me tell you from my experience. One time I was editing code on the live server using SFTP, before I knew about version control. And the whole thing was deleted, with no way to get it back. It doesn't take that long to put some safeguards. Here is the thing: WHENEVER you say "you have to be VERY careful" or blame someone for making a mistake, that's a place you need to put more checks. The computer should check routine things, not people. "The problem I'm having is where do you start? How do you simulate the data flowing through live in the test / dev enviornment." I am guessing you use MySQL so simply set up MYSQL REPLICATION and test all your stuff on the replica database first. Actually you should periodically dump the replica and test on a snapshot, so you can mess up the snapshot. Main db -> Replica -> Snapshot.
- LandR 9y ago
  It's a SQLSERVER, which at this point is about 200GB. Can I say just replicate the schema and then run a script to populate some test data. Or just move over a subset of the main DB? Got any good articles on how to setup test environments. I would then need all the dev code to point to the test one, not the live db and any Web APIs to be able to switch to point to the test db (which must mean running 2 web apis, a live and a test one...) Seems like it gets very complicated, but probably worth figuring out. We don't even have CI yet.
  - thehardsphere 9y ago
    Yes, typically when testing web applications, you have a second instance of the application running on a different machine than the production server, configured to run against a different database server as well. The exact details of how to do this well depend on how your application is written and its requirements. If all you want to do is functional testing of the application by yourself, then it is OK to deploy a copy of the software on your desktop along with a test database. I do this for the application I work on, and when I need to test against MSSQL, the Express edition is good enough. If you need to do load testing or integration testing or other more complicated forms of testing, you usually try to duplicated the expected production environment as well as you can. This may not be possible, so you may have to make decisions about what's most relevant and simply accept those tradeoffs. EDIT: Don't worry about CI or anything else that fancy and complicated for now. You may not even need it; that depends on your team and how often your application changes. Just for now take the simplest steps you can until you get this basic stuff figured out first.

btschaegg 9y ago

Not to excuse anything here, but you wouldn't believe how _insane_ some business environments are when it comes to databases. And yes, by that I mean those systems that you wouldn't believe anyone would ever want to risk losing. Depending on what you're working on, chances are there even are people who use _security_ as an excuse to basically prevent any halfway useful emergency strategy one could think of when handling data impossible to pull off (oh the irony).

gnu8 9y ago
Availability is one leg of the CIA triad. Security is the justification for having backups and a recovery strategy, not an excuse for ignoring it.