4 ms·

That seems like an extremely good idea actually, since if you dogfood your own releasing service then you can't fix it anymore if you accidentally bring down th

by WJW 6y ago

That seems like an extremely good idea actually, since if you dogfood your own releasing service then you can't fix it anymore if you accidentally bring down the service.

xxpor 6y ago
That's usually solved with a parallel stack deployment, use the other stack if something is broken
- paxys 6y ago
  If the "other stack" isn't regularly used then you can assume it will be broken when needed
  - cpascal 6y ago
    You just run the previous version of the production stack in your "dogfood/operations" stack. Once you've fully rolled out production and have vetted it, you can upgrade the other one to match production.
    - epidemian 6y ago
      With large codebases maintained by many people, sometimes it's difficult to "just" do things like that. It's a bit weird to think that no one within that large group of professional developers hasn't thought of doing a simple obvious solution. It's probably not that simple.
      - cutemonster 6y ago
        Lots of comments here around that say "just do it like this: ...", or "why don't you: ... it'd be good because ..."
    - marcinzm 6y ago
      And then you get hit with a subtle bug that utterly nukes your system when the year changes from 2020 to 2021. Now you can't deploy anything because both systems are down. Given Github's scale and number of engineers it's basically guaranteed that they'll hit some sort of bug like that eventually. Not all bugs act immediately.
Xorlev 6y ago
That also means when it does go wrong, it takes much longer to fix. Good operational practice is to decrease MTTR, not make it worse.
notwhereyouare 6y ago
I did a short stint at wayfair and about 1-2 months in, there was a deploy that somehow got passed the test flow and when deployed took down their entire site. So badly that they couldn't even deploy the fix
- cutemonster 6y ago
  How did they solve that in the end?
Existenceblinks 6y ago
It sounds like a good idea, but is very bad one. I worked on a git deploy product and dogfood was very appealing, turned out it's confusing as fuck. And When things get wrong it's even more confusing, it's like horror time travel movie when you can't find origin to fix consequences.

by WJW 6y ago

That seems like an extremely good idea actually, since if you dogfood your own releasing service then you can't fix it anymore if you accidentally bring down the service.

xxpor 6y ago

That's usually solved with a parallel stack deployment, use the other stack if something is broken

paxys 6y ago
If the "other stack" isn't regularly used then you can assume it will be broken when needed
- cpascal 6y ago
  You just run the previous version of the production stack in your "dogfood/operations" stack. Once you've fully rolled out production and have vetted it, you can upgrade the other one to match production.
  - epidemian 6y ago
    With large codebases maintained by many people, sometimes it's difficult to "just" do things like that. It's a bit weird to think that no one within that large group of professional developers hasn't thought of doing a simple obvious solution. It's probably not that simple.
    - cutemonster 6y ago
      Lots of comments here around that say "just do it like this: ...", or "why don't you: ... it'd be good because ..."
  - marcinzm 6y ago
    And then you get hit with a subtle bug that utterly nukes your system when the year changes from 2020 to 2021. Now you can't deploy anything because both systems are down. Given Github's scale and number of engineers it's basically guaranteed that they'll hit some sort of bug like that eventually. Not all bugs act immediately.

Xorlev 6y ago

That also means when it does go wrong, it takes much longer to fix. Good operational practice is to decrease MTTR, not make it worse.

notwhereyouare 6y ago

I did a short stint at wayfair and about 1-2 months in, there was a deploy that somehow got passed the test flow and when deployed took down their entire site. So badly that they couldn't even deploy the fix

cutemonster 6y ago
How did they solve that in the end?

Existenceblinks 6y ago

It sounds like a good idea, but is very bad one. I worked on a git deploy product and dogfood was very appealing, turned out it's confusing as fuck. And When things get wrong it's even more confusing, it's like horror time travel movie when you can't find origin to fix consequences.