3 ms·
> With a valid measure I would expect a roughly even distribution over time between underestimates and overestimates This is a valid hypothesis. It’s wrong, an
by JumpCrisscross 8mo ago
> With a valid measure I would expect a roughly even distribution over time between underestimates and overestimates
This is a valid hypothesis. It’s wrong, and I’ll explain why. (It’s a bad and invalid thing to conclude.)
If measurement errors were iid, you’d be correct. But they’re not. They’re well documented for not being so. Earlier survey results are biased by directional response bias inasmuch as the employers with the lease changes respond first. So the earliest releases tend to match whatever was going on before. Then the employers who had to do paperwork respond. And then, finally, someone gets around to calling the folks who never got back. Some of them aren’t around anymore.
So yeah, the directional tendency in revisions is well documented. And for a long time, the early releases were appreciated. But maybe American statistical and media literacy is such that only final releases should be released, which would mean we’d always be working with data 6 months to a year out of date.
- _heimdall 8mo agoThat's all well and good in theory, but job reports data over recent years have noticeably shifted towards downward monthly revisions. Prior to the pandemic response, the graph [1] looks much more balanced with regards to positive and negative decisions. [1] https://www.apmresearchlab.org/blog/how-abnormal-are-the-revisions-in-this-months-jobs-report https://www.apmresearchlab.org/blog/how-abnormal-are-the-rev...
- jeffbee 8mo agoSure, but it's totally ridiculous to post about that without discussing the survey response rate, which is the cause of that drift. People are attributing it to political meddling, and that is baseless. Naturally all of this metadata about the BLS surveys is available for free from the BLS, so you can just go look at it.
- _heimdall 8mo agoInteresting that you're claiming this is baseless without providing any sources for your alternative. How do you know that (a) the response rate is down meaningfully and (b) that data shows a strong correlation or causation between the two?
- JumpCrisscross 8mo ago> but job reports data over recent years have noticeably shifted towards downward monthly revisions. Prior to the pandemic response, the graph [1] looks much more balanced with regards to positive and negative decisions Yes. The reasons for this are well documented. Changing methodology for the preview estimates is rigorous. That means our published estimates lag best estimates, something the primary sources note in every release if one gets past the headlines. Also, if you have one year of massive job gains and four years of flat and falling, you’ll spend most of your epoch biased one way. Again, not a sign of methodological problems. Just a predictable methodological artifact that folks are supposed to be able to incorporate before using, much less emotionally reacting to, the data.
- _heimdall 8mo agoWhy would the shift to a new methodology bias the estimates to one end? I would expect a new methodology to make comparisons of data between the two systems to potentially be unhelpful, but I wouldn't expect a valid methodology to bias one way or another. Related, I wouldn't expect past data to bias a current estimate. If 6 or 12 months of positive growth biases the next prediction it falls into the hot hands fallacy. It isn't predicting based on current predictions, its predicting based on recent past behavior and extrapolating forward. This only makes sense to do if the data is not yet available, and even then the extrapolation isn't a useful estimate of current conditions.
- JumpCrisscross 8mo ago> If 6 or 12 months of positive growth biases the next prediction it falls into the hot hands fallacy It’s a sample of a sample. The full sample is the final release. The early results are the preliminary releases. When firms change things they take longer to respond. So whichever way the economy is moving, there will be bias in that direction. If the economy is turning, you won’t know direction. If it’s accelerating or slowing down you don’t know magnitude. Sometimes context clues can help. Sometimes they can’t. There is no known statistical treatment for intuiting the missing data before one has it11