3 ms·

Same experience here for the most part. We're working on migrating away from an internal tool which has a lot of problems: flags can change in the middle of use

Same experience here for the most part. We're working on migrating away from an internal tool which has a lot of problems: flags can change in the middle of user sessions, limited targeting criteria, changes to flags require changes to code, no distinction between feature flags and experiments, experiments often target populations that vary greatly, experiments are "running" for months and in some cases years... Our approach to fixing these problems starts with having a golden path for running an experiment which essentially fits the OP. It's still going to take some work to educate everyone but the whole "golden path" culture makes it easier.

withinboredom 3y ago
When we started working on the internal platform, this was exactly the problems we had. When we were finally deleting the old code, we found a couple of experiments that had been running for nearly half a decade. For giggles, we ran an analysis on those experiments: no difference between a & b. That's usually the best result you can get, honestly. It means you get to make a decision of whether to go with a or b. You can pick the one you like better.
- travisjungroth 3y ago
  That's a great outcome for a do-no-harm test. Horrible outcome when you're expecting a positive effect.
  - withinboredom 3y ago
    It’s an experiment, you shouldn’t be “expecting” anything. You hypothesize an effect, but that doesn’t mean it will be there and if you prove it wrong, you continue to iterate.
    - travisjungroth 3y ago
      > you shouldn’t be “expecting” anything This is the biggest lie in experimentation. Of course you expect something. Why are you running this test over all other tests? What I'm challenging is that if a team has spent three months building a feature, you a/b test it and find no effect, that is not a good outcome. Having a tie where you get to choose anything is worse than having a winner that forces your hand. At least you have the option to improve your product.
      - withinboredom 3y ago
        > What I'm challenging is that if a team has spent three months building a feature, you a/b test it and find no effect, that is not a good outcome. That's a great outcome. At one company we spent a few months building a feature only for it to fail the test, now that was a bad outcome. The feature's code was so good, we ended up refactoring it to look like the old feature and switching to that. So there was a silver lining, I guess. The key takeaway was to never a/b test a feature that big again. Instead we would spend a few weeks to build something that didn't need to scale nor feature complete. (IOW, an MVP/POC shitty code). If it had come out that there was no difference, we would have gone with the new version code because it was so well built -- alternatively, if the code was shit, we probably would have thrown it out. That's why its the best result. You can write shitty POC code and toss it out -- or keep it if you really want.
      - withinboredom 3y ago
        > Of course you expect something. Why are you running this test over all other tests? Because it has the best chance to prove/disprove your hypothesis. That's it. Even if it doesn't, all that means is that the metrics you're measuring are not connected to what you're doing. There is more to learn and explore. So, you can hope that it will prove or disprove your hypothesis, but there is no rational reason to expect it to go either way.
        travisjungroth 3y ago
        But why this hypothesis? Sometimes people do tests just to learn as much as they can, but 95%+ of the time they’re trying to improve their product. > there is no rational reason to expect it to go either way. Flipping a coin has the same probability of heads during a new moon as during a full moon. I’m going to jump ahead and expect that you agree with that statement. If I phrase that as a hypothesis and do an experiment, suddenly there’s no rational reason to expect it to go either way? Of course there is. The universe didn’t come into being when I started my experiment. Null hypothesis testing is a mental hack. A very effective one, but a hack. There is no null. Even assuming zero knowledge isn’t the most rational thing. But, the hack is that history has shown that when people try to act like they know nothing, they end up with better results. People are so overconfident that pretending they knew nothing improved things! This doesn’t mean it’s the truth, or even the best option.

withinboredom 3y ago

When we started working on the internal platform, this was exactly the problems we had. When we were finally deleting the old code, we found a couple of experiments that had been running for nearly half a decade. For giggles, we ran an analysis on those experiments: no difference between a & b. That's usually the best result you can get, honestly. It means you get to make a decision of whether to go with a or b. You can pick the one you like better.

travisjungroth 3y ago
That's a great outcome for a do-no-harm test. Horrible outcome when you're expecting a positive effect.
- withinboredom 3y ago
  It’s an experiment, you shouldn’t be “expecting” anything. You hypothesize an effect, but that doesn’t mean it will be there and if you prove it wrong, you continue to iterate.
  - travisjungroth 3y ago
    > you shouldn’t be “expecting” anything This is the biggest lie in experimentation. Of course you expect something. Why are you running this test over all other tests? What I'm challenging is that if a team has spent three months building a feature, you a/b test it and find no effect, that is not a good outcome. Having a tie where you get to choose anything is worse than having a winner that forces your hand. At least you have the option to improve your product.
    - withinboredom 3y ago
      > What I'm challenging is that if a team has spent three months building a feature, you a/b test it and find no effect, that is not a good outcome. That's a great outcome. At one company we spent a few months building a feature only for it to fail the test, now that was a bad outcome. The feature's code was so good, we ended up refactoring it to look like the old feature and switching to that. So there was a silver lining, I guess. The key takeaway was to never a/b test a feature that big again. Instead we would spend a few weeks to build something that didn't need to scale nor feature complete. (IOW, an MVP/POC shitty code). If it had come out that there was no difference, we would have gone with the new version code because it was so well built -- alternatively, if the code was shit, we probably would have thrown it out. That's why its the best result. You can write shitty POC code and toss it out -- or keep it if you really want.
    - withinboredom 3y ago
      > Of course you expect something. Why are you running this test over all other tests? Because it has the best chance to prove/disprove your hypothesis. That's it. Even if it doesn't, all that means is that the metrics you're measuring are not connected to what you're doing. There is more to learn and explore. So, you can hope that it will prove or disprove your hypothesis, but there is no rational reason to expect it to go either way.
      - travisjungroth 3y ago
        But why this hypothesis? Sometimes people do tests just to learn as much as they can, but 95%+ of the time they’re trying to improve their product. > there is no rational reason to expect it to go either way. Flipping a coin has the same probability of heads during a new moon as during a full moon. I’m going to jump ahead and expect that you agree with that statement. If I phrase that as a hypothesis and do an experiment, suddenly there’s no rational reason to expect it to go either way? Of course there is. The universe didn’t come into being when I started my experiment. Null hypothesis testing is a mental hack. A very effective one, but a hack. There is no null. Even assuming zero knowledge isn’t the most rational thing. But, the hack is that history has shown that when people try to act like they know nothing, they end up with better results. People are so overconfident that pretending they knew nothing improved things! This doesn’t mean it’s the truth, or even the best option.