5 ms·

I am trying to understand why the new merge method needed to be tested online via experiment. Both correctness and performance of the new merge method could hav

by smg 11y ago

I am trying to understand why the new merge method needed to be tested online via experiment. Both correctness and performance of the new merge method could have been tested offline working with snapshots (backups) of repos. Could a github engineer shed more light here?

hueving 11y ago
maybe they just do everything in production. "move fast and fix things"
abritishguy 11y ago
That is harder than just running the experiment online.
kevan 11y ago
Speculation, but if they already have the infrastructure to run the test online then it was probably easier than building one-time-use tools to test backups.
tanoku 11y ago
Author here. 5 years ago I would have agreed with you and logged e.g. 10 million merge requests to replay them offline. But one thing I've found over the years (which may seem obvious in retrospect) is that staging environment are not identical to production. Particularly not when it comes to finding sneaky bugs and performance regressions -- the code doesn't run on the same exact environment it will run when it is deployed (it has different input behaviors, and most importantly, it has different load and performance characteristics). The question then becomes "why would you run these experiments offline when you can run them online?". So we simply do. I personally feel it's a game changer.
- solutionyogi 11y ago
  Couldn't agree more. It is almost impossible to have a staging environment which is EXACTLY like the Production environment. A slightest difference can introduce bug in Production which will get missed out in staging. And if you can run your experiment in Production without any downside, why not?
- humanrebar 11y ago
  > why would you run these experiments offline when you can run them online? It probably doesn't apply in this case, but many bugs cannot be replicated in the production environment. At least, you don't want to break things in production to see if your software does the right thing in adverse scenarios.
  - allannienhuis 11y ago
    Some would say that's a really good strategy. Keeps you on your toes... http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
- jdmichal 11y ago
  TL;DR: In theory, theory and practice are the same. In practice, they are not.
jerf 11y ago
If you read about what they're doing, they basically are doing that. The tests are run independently of the production code, and production is just providing the test cases.

by smg 11y ago

hueving 11y ago

maybe they just do everything in production. "move fast and fix things"

abritishguy 11y ago

That is harder than just running the experiment online.

kevan 11y ago

Speculation, but if they already have the infrastructure to run the test online then it was probably easier than building one-time-use tools to test backups.

tanoku 11y ago

Author here. 5 years ago I would have agreed with you and logged e.g. 10 million merge requests to replay them offline. But one thing I've found over the years (which may seem obvious in retrospect) is that staging environment are not identical to production. Particularly not when it comes to finding sneaky bugs and performance regressions -- the code doesn't run on the same exact environment it will run when it is deployed (it has different input behaviors, and most importantly, it has different load and performance characteristics). The question then becomes "why would you run these experiments offline when you can run them online?". So we simply do. I personally feel it's a game changer.

solutionyogi 11y ago
Couldn't agree more. It is almost impossible to have a staging environment which is EXACTLY like the Production environment. A slightest difference can introduce bug in Production which will get missed out in staging. And if you can run your experiment in Production without any downside, why not?
humanrebar 11y ago
> why would you run these experiments offline when you can run them online? It probably doesn't apply in this case, but many bugs cannot be replicated in the production environment. At least, you don't want to break things in production to see if your software does the right thing in adverse scenarios.
- allannienhuis 11y ago
  Some would say that's a really good strategy. Keeps you on your toes... http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
jdmichal 11y ago
TL;DR: In theory, theory and practice are the same. In practice, they are not.

jerf 11y ago

If you read about what they're doing, they basically are doing that. The tests are run independently of the production code, and production is just providing the test cases.