11 ms·

Slightly off topic but articles like this make me weep as I look at my infrastructure and think about all the best practices that we by all means should be foll

Slightly off topic but articles like this make me weep as I look at my infrastructure and think about all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages). I'd give my first born child for immutable servers. Consistency in database systems refers to the requirement that any given database transaction must change affected data only in allowed ways. Any data written to the database must be valid according to all defined rules I would LOVE for this to be a thing. Just as soon as I can convince people that maybe storing my.cnf in VCS is a better idea than updating it in prod. Live. Sighs.

FLUX-YOU 9y ago
>I would LOVE for this to be a thing. It is a thing, just not at that company. Seriously, get a year or two, then look for escape options. There's a Mt. Everest climb's worth of distance and effort between "it works" and "best practices", and like successfully climbing Everest, very few companies are doing all of the best practices, and I suspect your dude going the opposite direction by digging with a spoon.
- iamdave 9y ago
  Seriously, get a year or two, then look for escape options. Been a year and a half. Have a 2nd round interview lined up for Thursday to get off this crazy train. I also happen to be reading The Phoenix Project. Which is...strangely therapeutic ;)
  - hinkley 9y ago
    There was a job when I was still green where they had fired an architect and they were bringing in more hands on people with a hope someone would step up. Someone opined that architects have a half-life of about 12 months. I ended up splitting lead duties with another person and she and I split the architectural oversight. Since then the same patterns (alas, minus the diversity) have worked out a number of times and I’m adamant that adding another lead is far better than consolidating that under one person, at least until you have half a dozen leads. In fact my most useful mentor and boss said to me look, I trust you guys to sort this out. I want to hear the plans and ask questions but I only need to be involved when you can’t come to an agreement. And then I can be your tie breaker. I think a lot of things would go better if folks in these positions would audit and break ties and worry about bigger picture stuff most of the time.
    - iamdave 9y ago
      In fact my most useful mentor and boss said to me look, I trust you guys to sort this out. I want to hear the plans and ask questions but I only need to be involved when you can’t come to an agreement. And then I can be your tie breaker. I had a boss like this and very much believed in that sort of autonomy, only getting involved-as you said to be the 'tie breaker', but still lead a great engineering team and was a great source of knowledge and mentorship. I say 'had' for a reason. That reason is mentioned in the post that started this comment chain. Now what's that tell you? Heh...heh :(
      - hinkley 9y ago
        I had a different boss who trusted us implicitly but essentially we were managing ourselves. Unlike Mr Mentor we didn’t talk shop and he didn’t challenge us very much or at least concretely to do better. In fact I was only ever aware of him doing about 20 hours of work a week and half of that was meetings (the other half administravia). Given the politics there I suspect the rest of his time was kissing butts or covering them. Since the end of the dotcom era I’ve never seen a more high minded bunch of Emperors New Clothes types all in one spot. Woof.
  - Terr_ 9y ago
    > I also happen to be reading The Phoenix Project. Which is...strangely therapeutic ;) Huh, I started reading that earlier today. Between the book and replaying levels in Infinifactory, the words "bottleneck" and "backpressure" loom large in my mind.
- hinkley 9y ago
  Some people get ahold of tunnel boring equipment and go as fast as possible in the wrong direction. With this mountain in the way we can’t tel if there’s anything cool on the other side, but damn are we hopeful.
gedrap 9y ago
>> storing my.cnf in VCS is a better idea than updating it in prod. Eh, 'but we can't store things/API keys/passwords in VCS! It's bad!' is a pet peeve of mine. No, let's not 'just update it manually when we need it'. No, if outsider getting your DB password (somehow) poses existential threat... Chances are that you have way more concerning problems.
- stouset 9y ago
  Storing passwords and other secrets in VCS is a phenomenally bad idea, particularly when your team size grows past two or three people. All it takes is one fuckup of accidentally committing to a public repo. Or firing someone on the team but not thinking to rotate secrets. Or hiring an outside contractor that now you implicitly give access to your production AWS credentials, etc. Secrets do not belong in source control, hard stop.
  - iamdave 9y ago
    It is wholly possible to store configuration data in vcs in a way that doesn't sacrifice depth of security when it comes to secrets and passwords. But otherwise yes, storing passwords in vcs is bad. That wasn't necessarily what I meant to imply by suggesting storing configuration files in vcs, however. I probably phrased that poorly.
  - breatheoften 9y ago
    I don’t understand why secrets should be excluded from source control. It seems like a perfectly fine place for them to be stored to me especially when talking about secrets which the developers require in order to develop the code or maintain a set of systems. If the secrets in source control are: (1) encrypted (2) never have decryption keys stored/loaded on any developer machine (3) never have stored decrypted representations (only in memory representation of decrypted forms allowed when required). If you follow these rules you won’t be more likely to accidentally commit unencrypted versions of the secrets and you’ll also by necessity have setup some (auditable) gatekeeper for logging of decryption events (via an aws kms or similar decryption as a service api). For the category of secrets which must never exist in any decrypted form on a developer machine — maybe I can see the argument that those should be left out of source control as this would represent a reduction in the surface area for offline attacks against the encrypted form of the secrets -— but I would guess that this actually represents a somewhat minor gain in practice? And also seems strictly unimportant to use cases where access to the decrypted representation of the secrets on the developers machine is mandatory ...
AFNobody 9y ago
> Slightly off topic but articles like this make me weep as I look at my infrastructure and think about all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages). I'd give my first born child for immutable servers. I know that feel although in my case its non-technical management who just want things to be frozen in time. Cest la vie at some companies unfortunately. Good luck with your escape plan. :)
btilly 9y ago
Sounds like said "senior engineer" is an example of what I discussed in response to https://news.ycombinator.com/item?id=15975130 https://news.ycombinator.com/item?id=15975130. Don't let it happen to you.
mikekchar 9y ago
Not really related to your main point, but speaking as a greying "old guy", one of the things that's really hard is to gracefully allow the next generation to contribute. I still vividly remember starting out and being incredulous about some of the stupid ideas the senior guys on the team had. I was proven right quite a lot of the time too. It's easy to build that inner narrative of the hero righting wrongs. But then fast forward to the back end of your career and you've got a kind of weird dilemma. The vast majority of ridiculous things that the young guys try to push is absolute crap. They read about it in a blog post somewhere and don't have nearly enough experience to understand the nuances. Or they have yet another stupid f-ing framework that they just know is going to revolutionise the industry. Yeah... been there, done that, my closet is overflowing with T-shirts, thank you very much. You sit there thinking, "I spent 20 or 30 years fighting my way through this crap and when I finally have a good handle on it, these new guys are just going to throw it all away". But... Dammit. You absolutely know that one of them has a great idea and you're not quite sure which one of them has it. Because they all think they have it sorted (even when none of them do). They feel they are completely sure that "this is the best way and only a moron couldn't see it". But you know that it's always more subtle than it appears. If you let them all run with their ideas, then 90% (99%???) will crash hard... Let's face it -- just like you did when you had that stupid idea that you were absolutely certain was "the way forward". Anyway, I belabour the point. From my vantage point, it's pretty difficult to let go. Some people just can't do it. In my position now, I have a lot more sympathy for the "old guard" than I did when I was younger. To be honest, I think it is technically impossible for me to have less sympathy than I did -- and I think this is pretty common in the industry. Probably both sides of the equation are in a similar situation -- neither side feels comfortable letting the other side make decisions. Just talking out of the hole in my head right now, but I think the main thing is that programmers generally like to program. They like to have freedom to try approaches that they think will be successful. As much as possible, it's probably best to allow your teammates, whoever they are, to have the freedom that they need. At the same time you should expect to have your own freedom. Getting the balance is not easy and sometimes you have to change jobs in order to get into a team where you are allowed the freedom to do your best. In my experience, teams that take this seriously are the best teams to work on (even if they sometimes do stupid things).
- 9y ago
lmm 9y ago
> all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages) Huh, that's exactly how I feel about the "must use a RDBMS for all data" senior architects I've worked with.
afpx 9y ago
What does the 1990s have to do with it? The 90s were all about data consistency (probably too much so).
fauigerzigerk 9y ago
This is ageism plain and simple. You clearly have no clue what was the predominant view in 1990s. Consistency wasn't just _a_ thing in the 1990s. It was a core dogma even beyond database systems. It was top of the agenda for a while until it sort of fizzled out in the early 2000s when distributed transactions hit a complexity wall.
- smaddox 9y ago
  How do you reduce transactional complexity by throwing out consistency? Genuine question. At the very least, it seems like you need to mark potentially inconsistent data as dirty.
  - fauigerzigerk 9y ago
    I didn't mean to imply that you could. But the simplicity that consistent distributed transactions can deliver in principle can easily get drowned out by accidential complexity created by a specific implementation. For instance, you have to get all participants in a distributed transaction to support the same standard protocol without too much bridging, wrapping, special interface description languages, tons of configuration metadata, etc. You have to agree on a small stable set of features that everyone implements in a robust way. But the major corporate sponsors of the competing approaches originating in the 1990s and early 2000s got disrupted before any one of them could gain enough traction to bludgeon everyone else into compliance. My point was merely that almost no one in the 1990s doubted the necessity of consistent transactions. Eventual consistency and all the rest only became popular with NoSQL and the general desire to imitate internet giants that didn't even exist in the 1990s.

FLUX-YOU 9y ago

>I would LOVE for this to be a thing. It is a thing, just not at that company. Seriously, get a year or two, then look for escape options. There's a Mt. Everest climb's worth of distance and effort between "it works" and "best practices", and like successfully climbing Everest, very few companies are doing all of the best practices, and I suspect your dude going the opposite direction by digging with a spoon.

iamdave 9y ago
Seriously, get a year or two, then look for escape options. Been a year and a half. Have a 2nd round interview lined up for Thursday to get off this crazy train. I also happen to be reading The Phoenix Project. Which is...strangely therapeutic ;)
- hinkley 9y ago
  There was a job when I was still green where they had fired an architect and they were bringing in more hands on people with a hope someone would step up. Someone opined that architects have a half-life of about 12 months. I ended up splitting lead duties with another person and she and I split the architectural oversight. Since then the same patterns (alas, minus the diversity) have worked out a number of times and I’m adamant that adding another lead is far better than consolidating that under one person, at least until you have half a dozen leads. In fact my most useful mentor and boss said to me look, I trust you guys to sort this out. I want to hear the plans and ask questions but I only need to be involved when you can’t come to an agreement. And then I can be your tie breaker. I think a lot of things would go better if folks in these positions would audit and break ties and worry about bigger picture stuff most of the time.
  - iamdave 9y ago
    In fact my most useful mentor and boss said to me look, I trust you guys to sort this out. I want to hear the plans and ask questions but I only need to be involved when you can’t come to an agreement. And then I can be your tie breaker. I had a boss like this and very much believed in that sort of autonomy, only getting involved-as you said to be the 'tie breaker', but still lead a great engineering team and was a great source of knowledge and mentorship. I say 'had' for a reason. That reason is mentioned in the post that started this comment chain. Now what's that tell you? Heh...heh :(
    - hinkley 9y ago
      I had a different boss who trusted us implicitly but essentially we were managing ourselves. Unlike Mr Mentor we didn’t talk shop and he didn’t challenge us very much or at least concretely to do better. In fact I was only ever aware of him doing about 20 hours of work a week and half of that was meetings (the other half administravia). Given the politics there I suspect the rest of his time was kissing butts or covering them. Since the end of the dotcom era I’ve never seen a more high minded bunch of Emperors New Clothes types all in one spot. Woof.
- Terr_ 9y ago
  > I also happen to be reading The Phoenix Project. Which is...strangely therapeutic ;) Huh, I started reading that earlier today. Between the book and replaying levels in Infinifactory, the words "bottleneck" and "backpressure" loom large in my mind.
hinkley 9y ago
Some people get ahold of tunnel boring equipment and go as fast as possible in the wrong direction. With this mountain in the way we can’t tel if there’s anything cool on the other side, but damn are we hopeful.

gedrap 9y ago

>> storing my.cnf in VCS is a better idea than updating it in prod. Eh, 'but we can't store things/API keys/passwords in VCS! It's bad!' is a pet peeve of mine. No, let's not 'just update it manually when we need it'. No, if outsider getting your DB password (somehow) poses existential threat... Chances are that you have way more concerning problems.

stouset 9y ago
Storing passwords and other secrets in VCS is a phenomenally bad idea, particularly when your team size grows past two or three people. All it takes is one fuckup of accidentally committing to a public repo. Or firing someone on the team but not thinking to rotate secrets. Or hiring an outside contractor that now you implicitly give access to your production AWS credentials, etc. Secrets do not belong in source control, hard stop.
- iamdave 9y ago
  It is wholly possible to store configuration data in vcs in a way that doesn't sacrifice depth of security when it comes to secrets and passwords. But otherwise yes, storing passwords in vcs is bad. That wasn't necessarily what I meant to imply by suggesting storing configuration files in vcs, however. I probably phrased that poorly.
- breatheoften 9y ago
  I don’t understand why secrets should be excluded from source control. It seems like a perfectly fine place for them to be stored to me especially when talking about secrets which the developers require in order to develop the code or maintain a set of systems. If the secrets in source control are: (1) encrypted (2) never have decryption keys stored/loaded on any developer machine (3) never have stored decrypted representations (only in memory representation of decrypted forms allowed when required). If you follow these rules you won’t be more likely to accidentally commit unencrypted versions of the secrets and you’ll also by necessity have setup some (auditable) gatekeeper for logging of decryption events (via an aws kms or similar decryption as a service api). For the category of secrets which must never exist in any decrypted form on a developer machine — maybe I can see the argument that those should be left out of source control as this would represent a reduction in the surface area for offline attacks against the encrypted form of the secrets -— but I would guess that this actually represents a somewhat minor gain in practice? And also seems strictly unimportant to use cases where access to the decrypted representation of the secrets on the developers machine is mandatory ...

AFNobody 9y ago

> Slightly off topic but articles like this make me weep as I look at my infrastructure and think about all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages). I'd give my first born child for immutable servers. I know that feel although in my case its non-technical management who just want things to be frozen in time. Cest la vie at some companies unfortunately. Good luck with your escape plan. :)

btilly 9y ago

Sounds like said "senior engineer" is an example of what I discussed in response to https://news.ycombinator.com/item?id=15975130 https://news.ycombinator.com/item?id=15975130. Don't let it happen to you.

mikekchar 9y ago

Not really related to your main point, but speaking as a greying "old guy", one of the things that's really hard is to gracefully allow the next generation to contribute. I still vividly remember starting out and being incredulous about some of the stupid ideas the senior guys on the team had. I was proven right quite a lot of the time too. It's easy to build that inner narrative of the hero righting wrongs. But then fast forward to the back end of your career and you've got a kind of weird dilemma. The vast majority of ridiculous things that the young guys try to push is absolute crap. They read about it in a blog post somewhere and don't have nearly enough experience to understand the nuances. Or they have yet another stupid f-ing framework that they just know is going to revolutionise the industry. Yeah... been there, done that, my closet is overflowing with T-shirts, thank you very much. You sit there thinking, "I spent 20 or 30 years fighting my way through this crap and when I finally have a good handle on it, these new guys are just going to throw it all away". But... Dammit. You absolutely know that one of them has a great idea and you're not quite sure which one of them has it. Because they all think they have it sorted (even when none of them do). They feel they are completely sure that "this is the best way and only a moron couldn't see it". But you know that it's always more subtle than it appears. If you let them all run with their ideas, then 90% (99%???) will crash hard... Let's face it -- just like you did when you had that stupid idea that you were absolutely certain was "the way forward". Anyway, I belabour the point. From my vantage point, it's pretty difficult to let go. Some people just can't do it. In my position now, I have a lot more sympathy for the "old guard" than I did when I was younger. To be honest, I think it is technically impossible for me to have less sympathy than I did -- and I think this is pretty common in the industry. Probably both sides of the equation are in a similar situation -- neither side feels comfortable letting the other side make decisions. Just talking out of the hole in my head right now, but I think the main thing is that programmers generally like to program. They like to have freedom to try approaches that they think will be successful. As much as possible, it's probably best to allow your teammates, whoever they are, to have the freedom that they need. At the same time you should expect to have your own freedom. Getting the balance is not easy and sometimes you have to change jobs in order to get into a team where you are allowed the freedom to do your best. In my experience, teams that take this seriously are the best teams to work on (even if they sometimes do stupid things).

9y ago

lmm 9y ago

> all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages) Huh, that's exactly how I feel about the "must use a RDBMS for all data" senior architects I've worked with.

afpx 9y ago

What does the 1990s have to do with it? The 90s were all about data consistency (probably too much so).

fauigerzigerk 9y ago

This is ageism plain and simple. You clearly have no clue what was the predominant view in 1990s. Consistency wasn't just _a_ thing in the 1990s. It was a core dogma even beyond database systems. It was top of the agenda for a while until it sort of fizzled out in the early 2000s when distributed transactions hit a complexity wall.

smaddox 9y ago
How do you reduce transactional complexity by throwing out consistency? Genuine question. At the very least, it seems like you need to mark potentially inconsistent data as dirty.
- fauigerzigerk 9y ago
  I didn't mean to imply that you could. But the simplicity that consistent distributed transactions can deliver in principle can easily get drowned out by accidential complexity created by a specific implementation. For instance, you have to get all participants in a distributed transaction to support the same standard protocol without too much bridging, wrapping, special interface description languages, tons of configuration metadata, etc. You have to agree on a small stable set of features that everyone implements in a robust way. But the major corporate sponsors of the competing approaches originating in the 1990s and early 2000s got disrupted before any one of them could gain enough traction to bludgeon everyone else into compliance. My point was merely that almost no one in the 1990s doubted the necessity of consistent transactions. Eventual consistency and all the rest only became popular with NoSQL and the general desire to imitate internet giants that didn't even exist in the 1990s.