4 ms·

Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).

by seanmcq 16y ago

nettdata 16y ago
In my environment, our Dev's (individuals or environments/subnets) don't have access to PROD or QA, and our CIT boxes are in DEV. Likewise, QA and PROD only have access to their own environments. We have a build master that promotes a reviewed deployment package to QA and/or PROD environments, where the appropriate QA or PROD operations folks do the actual deployment. It's a luxury to have the resources available for this, but it's a life saver, because it really is stupidly easy to make a simple mistake and totally screw things up. The last time something similar happened to me, it happened to be at the end of a REALLY long day. And what do you know... that day was then made 24 hours longer, interspersed with the occasional cat nap while backups were being restored and verified. Fun times. Not.
- smokinn 16y ago
  I'm not so sure it's a luxury. Maybe it is if your startup is servicing a group of techies who have knowledge of what problems lay in the background but most clients don't know and don't care. It's not that hard to keep up really. It's a question of a day or two of setup and then the hardish part of constant discipline to not take the "easy way out" and poke holes into the segregation you've set up. Mainly it takes a single team lead or CTO or whatever to be really explicit that you just don't break the steps and you'll avoid a LOT of problems. You'll still have problems, problems are inevitable, but in general you'll have mitigated them and with a proper backup and merge procedure you'll minimize downtime.
  - nettdata 16y ago
    Agreed, but to be clear, the luxury I was referring to was specifically having the DEV and PROD staff being totally separate teams/individuals, not just the separation of environments. In other words, nobody on the DEV team did any PROD operations, except in the case of bug investigation, tuning advice, etc.
jrockway 16y ago
Interesting idea. I wonder if it would be worthwhile to store server information in LDAP, add a field for environment type, and then teach your server components to act on that information. Then when the dev app server connects to the prod database, the database checks the server name, sees that it's dev, and refuses the connection.
michael_dorfman 16y ago
Nice, but still too specific. Lesson: don't let anything talk to your production servers. Make it as difficult as possible for anyone to log in. Treat production as if it were a loaded gun. You shouldn't be touching it unless you absolutely have to, and even then, you need to be vigilantly aware of the consequences of your actions.
arethuza 16y ago
Maybe I should go and put in a reference to this thread in yesterday's thread about how you don't really need firewalls...

Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).

by seanmcq 16y ago

Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).

In my environment, our Dev's (individuals or environments/subnets) don't have access to PROD or QA, and our CIT boxes are in DEV. Likewise, QA and PROD only have access to their own environments. We have a build master that promotes a reviewed deployment package to QA and/or PROD environments, where the appropriate QA or PROD operations folks do the actual deployment. It's a luxury to have the resources available for this, but it's a life saver, because it really is stupidly easy to make a simple mistake and totally screw things up. The last time something similar happened to me, it happened to be at the end of a REALLY long day. And what do you know... that day was then made 24 hours longer, interspersed with the occasional cat nap while backups were being restored and verified. Fun times. Not.

smokinn 16y ago
I'm not so sure it's a luxury. Maybe it is if your startup is servicing a group of techies who have knowledge of what problems lay in the background but most clients don't know and don't care. It's not that hard to keep up really. It's a question of a day or two of setup and then the hardish part of constant discipline to not take the "easy way out" and poke holes into the segregation you've set up. Mainly it takes a single team lead or CTO or whatever to be really explicit that you just don't break the steps and you'll avoid a LOT of problems. You'll still have problems, problems are inevitable, but in general you'll have mitigated them and with a proper backup and merge procedure you'll minimize downtime.
- nettdata 16y ago
  Agreed, but to be clear, the luxury I was referring to was specifically having the DEV and PROD staff being totally separate teams/individuals, not just the separation of environments. In other words, nobody on the DEV team did any PROD operations, except in the case of bug investigation, tuning advice, etc.

jrockway 16y ago

Interesting idea. I wonder if it would be worthwhile to store server information in LDAP, add a field for environment type, and then teach your server components to act on that information. Then when the dev app server connects to the prod database, the database checks the server name, sees that it's dev, and refuses the connection.

michael_dorfman 16y ago

Nice, but still too specific. Lesson: don't let anything talk to your production servers. Make it as difficult as possible for anyone to log in. Treat production as if it were a loaded gun. You shouldn't be touching it unless you absolutely have to, and even then, you need to be vigilantly aware of the consequences of your actions.

arethuza 16y ago

Maybe I should go and put in a reference to this thread in yesterday's thread about how you don't really need firewalls...