4 ms·

If I read the first mistake correctly, then getFeatureFlag() has the side-effect to count how often it was called and uses this to calculate the outcome of the

If I read the first mistake correctly, then getFeatureFlag() has the side-effect to count how often it was called and uses this to calculate the outcome of the experiment? Wow. I don't know what to say....

pjm331 3y ago
This is indeed the case. Have run into a few surprising things like this when implementing posthog experiments recently
sometimes_all 3y ago
Yeah I felt that way too. Initially I thought I wasn't sure what I was missing, since the only difference is that the order of the checks is switched, and the function will still return the same true/false in both cases. Then I thought about side effects and it felt icky.
willsmith72 3y ago
Yep gross...
alsiola 3y ago
Writing an article about developer mistakes is easier than redesigning your rubbish API though.
xp84 3y ago
That's how every one of these tools works, that’s the whole point of using them: you only call them when you’re going to actually show the variation to the user. If you’re running a test that modifies the homepage only, you shouldn’t be calling that decision method in, say, your global navigation code that you show everyone. Or, for instance, if your test only affects how the header looks for subscribers, you have to put an outer if statement “if subscriber“ before the “if test variation.“ How else would it correctly know exactly who saw the test?
dyeje 3y ago
When you call the feature flag, it’s going to put the user into one of the groups. The article is saying you don’t want to add irrelevant users (in the example, ones that had already done the action they were testing) because it’s going to skew your results.
- willsmith72 3y ago
  The point is from an api design perspective something like "posthog.getFeatureFlag('experiment-key')" doesn't look like it's actually performing a mutation.

4 ms·

If I read the first mistake correctly, then getFeatureFlag() has the side-effect to count how often it was called and uses this to calculate the outcome of the

by rmetzler 3y ago

pjm331 3y ago
This is indeed the case. Have run into a few surprising things like this when implementing posthog experiments recently
sometimes_all 3y ago
Yeah I felt that way too. Initially I thought I wasn't sure what I was missing, since the only difference is that the order of the checks is switched, and the function will still return the same true/false in both cases. Then I thought about side effects and it felt icky.
willsmith72 3y ago
Yep gross...
alsiola 3y ago
Writing an article about developer mistakes is easier than redesigning your rubbish API though.
xp84 3y ago
That's how every one of these tools works, that’s the whole point of using them: you only call them when you’re going to actually show the variation to the user. If you’re running a test that modifies the homepage only, you shouldn’t be calling that decision method in, say, your global navigation code that you show everyone. Or, for instance, if your test only affects how the header looks for subscribers, you have to put an outer if statement “if subscriber“ before the “if test variation.“ How else would it correctly know exactly who saw the test?
dyeje 3y ago
When you call the feature flag, it’s going to put the user into one of the groups. The article is saying you don’t want to add irrelevant users (in the example, ones that had already done the action they were testing) because it’s going to skew your results.
- willsmith72 3y ago
  The point is from an api design perspective something like "posthog.getFeatureFlag('experiment-key')" doesn't look like it's actually performing a mutation.