4 ms·

If I’m understanding correctly, this switches your profile on some interval between A and B, whereas a proper A/B test will randomly bucket a user to experiment

If I’m understanding correctly, this switches your profile on some interval between A and B, whereas a proper A/B test will randomly bucket a user to experiment A or B. Not that it matters - this solution is probably the right way to go without building something into Twitter itself - but the more data/stats oriented folks may be confused or irked by calling this “A/B testing”

maximedupre 4y ago
That is correct - it switched your profile version at a regular interval. Indeed, that's the only way I could do it. The Twitter API has its limitations :D I didn't know there was a specific definition of A/B testing. I'll see if I get more complaints about the terms I use ^^. To me, that's still A/B testing - that is I'm testing a version A and a version B and then report on which one does better. I guess the way I'm doing it is different :D
- e_i_pi_2 4y ago
  Definitely agree that this is a good solution given the limitations, I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results. That said I think the chances of that are super low and A/B testing is only so accurate anyway. Great idea and nice website!
  - maximedupre 4y ago
    Thank you! > I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results. That is definitely true. I'm about to start alternating the versions every 5m to mitigate this. The closer I can get to 0m, the more accurate the results are. This way even if you get a followers spike (let's say you get a viral tweet), the followers will be properly distributed between each version.
    - yeahboats 4y ago
      I think what you really want to do is randomize A or B within each 30m (or 5m) interval. This is basically switchback testing. Door dash has a nice write-up here: https://doordash.engineering/2018/02/13/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash/ https://doordash.engineering/2018/02/13/switchback-tests-and...
      - maximedupre 4y ago
        Oh right, that's actually a better technique! Thanks for the idea.
      - shay_ker 4y ago
        Yep - we've run many switchback tests so I'm happy to chat more about it. It's a lot more akin to what you're building here, from a stats point of view.
    - kqr 4y ago
      Then you get the opposite problem: users are more likely to see multiple versions of the same profile.
      - maximedupre 4y ago
        I feel like that's less of a problem because the user is more likely to convert on the version that he likes more, which would still provide accurate data. But yes, there is no perfect solution with the limitations of the API :D
- daniel-cussen 4y ago
  That's cool man apologize for what is for you to apologize, and nothing else. Nothing else. And do your thing in the semblance of your vision of your ideas, all the way and show the world. And the world won't like it. You gotta be sure of what you know. Be sure and be wrong, better than always being unsure.
  - maximedupre 4y ago
    Thanks Daniel, I guess xD That was quite cryptic, but I think I get it haha. Cheers!
- shay_ker 4y ago
  I hear ya - just giving the data/stats perspective. It's an audience that cares a lot about these details.
laweijfmvo 4y ago
Why is bucketing users the better approach? To me it seems like bucketing would ignore "all other factors" that also changed, whereas dynamic (or periodic) switching seems like it would normalize those "other factors" across both A/B (ideally).
- maximedupre 4y ago
  I suppose that the issue is that a single user could see both versions, which could skew the data. I'm not sure I totally understand why, because if a user follows you after seeing the other profile version, it might be because he preferred this other version. But conceptually it makes sense to eliminate as many variables as you can to isolate the components of the test.

maximedupre 4y ago

That is correct - it switched your profile version at a regular interval. Indeed, that's the only way I could do it. The Twitter API has its limitations :D I didn't know there was a specific definition of A/B testing. I'll see if I get more complaints about the terms I use ^^. To me, that's still A/B testing - that is I'm testing a version A and a version B and then report on which one does better. I guess the way I'm doing it is different :D

e_i_pi_2 4y ago
Definitely agree that this is a good solution given the limitations, I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results. That said I think the chances of that are super low and A/B testing is only so accurate anyway. Great idea and nice website!
- maximedupre 4y ago
  Thank you! > I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results. That is definitely true. I'm about to start alternating the versions every 5m to mitigate this. The closer I can get to 0m, the more accurate the results are. This way even if you get a followers spike (let's say you get a viral tweet), the followers will be properly distributed between each version.
  - yeahboats 4y ago
    I think what you really want to do is randomize A or B within each 30m (or 5m) interval. This is basically switchback testing. Door dash has a nice write-up here: https://doordash.engineering/2018/02/13/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash/ https://doordash.engineering/2018/02/13/switchback-tests-and...
    - maximedupre 4y ago
      Oh right, that's actually a better technique! Thanks for the idea.
    - shay_ker 4y ago
      Yep - we've run many switchback tests so I'm happy to chat more about it. It's a lot more akin to what you're building here, from a stats point of view.
  - kqr 4y ago
    Then you get the opposite problem: users are more likely to see multiple versions of the same profile.
    - maximedupre 4y ago
      I feel like that's less of a problem because the user is more likely to convert on the version that he likes more, which would still provide accurate data. But yes, there is no perfect solution with the limitations of the API :D
daniel-cussen 4y ago
That's cool man apologize for what is for you to apologize, and nothing else. Nothing else. And do your thing in the semblance of your vision of your ideas, all the way and show the world. And the world won't like it. You gotta be sure of what you know. Be sure and be wrong, better than always being unsure.
- maximedupre 4y ago
  Thanks Daniel, I guess xD That was quite cryptic, but I think I get it haha. Cheers!
shay_ker 4y ago
I hear ya - just giving the data/stats perspective. It's an audience that cares a lot about these details.

laweijfmvo 4y ago

Why is bucketing users the better approach? To me it seems like bucketing would ignore "all other factors" that also changed, whereas dynamic (or periodic) switching seems like it would normalize those "other factors" across both A/B (ideally).

maximedupre 4y ago
I suppose that the issue is that a single user could see both versions, which could skew the data. I'm not sure I totally understand why, because if a user follows you after seeing the other profile version, it might be because he preferred this other version. But conceptually it makes sense to eliminate as many variables as you can to isolate the components of the test.