2 ms·

test A - before test B - after what are you talking about ?

• shermantanktop 1y ago
  “A/B in time” suffers from inability to control for other factors that might vary over time. In this case, that could be the economy or other transit policies. But sometimes it’s the only possible approach.
• shadowgovt 1y ago
  Generally, that's considered to introduce counfounding factors on the time axis ("did we see improvement because we changed something or because flu season hit and people stayed home") that you'd prefer to mitigate by running your A and B simultaneously. But in the absence of the ability to run them simultaneously, "A is before and B is after" can be a fine proxy. Of course, if B is worse, it'd be nice if you could only subject, say, 5% of your population to it before you just slam the slider to 100% and hit everyone with it.
  • Calwestjobs 1y ago
    yes, but how the hell he proposes to make A/B testing of "whole Manhattan policy"? build another Manhattan just for test? makes no sense. whole manhattan is important. not 5%. so no 5%. a/b test can be done only for things which affect one person, like for example GUI etc, big group under test but effect on individuals, in such big scale a/b test is tool to deceive, not to get to right conclusion
    • shadowgovt 1y ago
      It is, indeed, much easier to do A/B testing online in environments you control than IRL. (Purely hypothetically: one could identify 10% of the island as operating under the new rules and compare outcomes. This is politically fraught on multiple levels and also gives messy spatial results.)
• Ntrails 1y ago
  "before" and "after" introduces a large axis of noise The problem is that for A/B testing to really work you need independent groups outcomes. As soon as there is any bias in group selection or cross group effect it's very hard to unpick.