13 ms·

We already had A/B testing of congestion pricing. The A test was without congestion pricing in NYC, and has been tested for decades.

bunderbunder 1y ago
That's not an A/B test because it has no way of controlling for broader economic trends over time. How do you figure out if what you're seeing is because of that one thing that changed, or the enormous list of other things that also changed around the same time? A more valid design would be randomly assigning some cities to institute congestion pricing, and other cities to not have it. Obviously not feasible in practice, but that's at least the kind of thing to strive toward when designing these kinds of studies.
- jannyfer 1y ago
  That would be a bad design for an A/B study (and NYC congestion pricing is not a “study” anyway), because cities are few and not alike and have an enormous list of other things that are different. What NYC equivalent would you pick? In any case, not every policy change needs to be an academic exercise.
  - bunderbunder 1y ago
    Yup, that is indeed a part of the problem. You'll notice I did say, "Obviously not feasible in practice." I've got a textbook on field experiments that refers to these kinds of questions as FUQ - acronym for "Fundamentally Unanswerable Questions". You can collect suggestive evidence, but firmly establishing cause and effect is something you've just got to let go of.
- JumpCrisscross 1y ago
  > randomly assigning some cities to institute congestion pricing, and other cities to not have it Cities are stupidly heterogenous. These data wouldn't be more meaningful than comparing cities with congestion pricing to those without. (And comparing them from their congestion eras.)
  - bunderbunder 1y ago
    What you're telling me here is that you aren't aware of what the randomization is for in randomized controlled trials. "Our treatment units are stupidly heterogeneous" is exactly the problem it solves. A century's worth of developing increasingly sophisticated statistical techniques for making do without random assignment has thus far failed to accomplish anything than provisional mitigations that are notoriously easy to use incorrectly in practice.
- sorcerer-mar 1y ago
  Everyone knows how you can conduct good experiments in a land of frictionless spherical cows.
s1artibartfast 1y ago
An important part of testing is establishing assessment criteria and collecting data. I wish more laws would pre-state what their intended outcome and success would look like.
tonymet 1y ago
you haven't described the observations or the sample
- ericpauley 1y ago
  TFA describes this extensively. The observations are traffic speed, bus timeliness, and over a dozen other metrics. The samples are sub-areas of NYC.
  - tonymet 1y ago
    Anything negative ?

bunderbunder 1y ago

That's not an A/B test because it has no way of controlling for broader economic trends over time. How do you figure out if what you're seeing is because of that one thing that changed, or the enormous list of other things that also changed around the same time? A more valid design would be randomly assigning some cities to institute congestion pricing, and other cities to not have it. Obviously not feasible in practice, but that's at least the kind of thing to strive toward when designing these kinds of studies.

jannyfer 1y ago
That would be a bad design for an A/B study (and NYC congestion pricing is not a “study” anyway), because cities are few and not alike and have an enormous list of other things that are different. What NYC equivalent would you pick? In any case, not every policy change needs to be an academic exercise.
- bunderbunder 1y ago
  Yup, that is indeed a part of the problem. You'll notice I did say, "Obviously not feasible in practice." I've got a textbook on field experiments that refers to these kinds of questions as FUQ - acronym for "Fundamentally Unanswerable Questions". You can collect suggestive evidence, but firmly establishing cause and effect is something you've just got to let go of.
JumpCrisscross 1y ago
> randomly assigning some cities to institute congestion pricing, and other cities to not have it Cities are stupidly heterogenous. These data wouldn't be more meaningful than comparing cities with congestion pricing to those without. (And comparing them from their congestion eras.)
- bunderbunder 1y ago
  What you're telling me here is that you aren't aware of what the randomization is for in randomized controlled trials. "Our treatment units are stupidly heterogeneous" is exactly the problem it solves. A century's worth of developing increasingly sophisticated statistical techniques for making do without random assignment has thus far failed to accomplish anything than provisional mitigations that are notoriously easy to use incorrectly in practice.
sorcerer-mar 1y ago
Everyone knows how you can conduct good experiments in a land of frictionless spherical cows.

s1artibartfast 1y ago

An important part of testing is establishing assessment criteria and collecting data. I wish more laws would pre-state what their intended outcome and success would look like.

tonymet 1y ago

you haven't described the observations or the sample

ericpauley 1y ago
TFA describes this extensively. The observations are traffic speed, bus timeliness, and over a dozen other metrics. The samples are sub-areas of NYC.
- tonymet 1y ago
  Anything negative ?