5 ms·

Study: Packages Sealed with ‘Atheist’ Tape 10 Times More Likely to Disappear

75 points by bado 14y ago

RexRollman 14y ago
Sad.
ghshephard 14y ago
I'd like to see the actual survey data. I'm not even sure how to interpret this: "Thus the ad hoc study: The company says it sent two packages each to 89 people (178 packages total) canvassing nearly every U.S. state — one package with the Atheist-branded tape, one without. And this is where the results suggest something fishy: According to Atheist Shoes, company-branded packages took on average three days longer to reach their destination and were 10 times more likely to disappear outright." What does "10 times more likely to disappear outright" mean? Are they saying that 10 of the 89 Atheist packages disappeared, and only 1 of the 89 non-atheist packages disappeared?
- simonsez 14y ago
  Agreed - the sample size is also really small to be interpreting slower arrival times as well, if only 1 or 2 packages took a long time it's likely they just found an outlier.
  - jlgreco 14y ago
    Yeah, that 3 days could either be damning, or absolutely nothing. Until we can examine the data this is a non-story despite how juicy we may want it to be.
  - christopheraden 14y ago
    Sample size is something that must be interpreted in the presence of power. You can make a solid conclusion with a very small sample size if the true difference in arrival times is very large, given that the assumptions of the hypothesis test hold (t-test can be a little ridiculous with some of its assumptions sometimes). In the original article, one of the footnotes mentioned that they tested the data using Wilcoxon's Signed-Rank test, which mitigates a lot of the impact of single outliers. I'd love to see the raw data though, to see an even less-sensitive method to outliers (sign test). If the difference between the groups is as large as the article would lead us to believe, the loss of power should not present any problem.
  - A1kmm 14y ago
    According to http://www.atheistberlin.com/study http://www.atheistberlin.com/study they got a significant Wilcoxon Signed Rank Test at p < 0.01. The Wilcoxon Signed Rank Test is nonparametric (i.e. sacrifices some statistical power in order to not make assumptions about the underlying distribution of the data) so is meaningful even if there is a long tail of packages that take longer due to circumstances outside of the study variable.
- pavanky 14y ago
  Yep pretty much. In the original article (not from time.com) it was 9 and 1. So I am not sure where the number came from.
- deleted 14y ago
  [deleted]
- citricsquid 14y ago
  I can't find the previous submission but someone pointed out that one of the Atheist labelled packages was lost for almost 40 days: that alone dragged the Atheist package average down so much it can almost account for the entire difference. edit: I misremembered about the difference it made, here's the submission: https://news.ycombinator.com/item?id=5442728 https://news.ycombinator.com/item?id=5442728
- codezero 14y ago
  Check out the comments on their own web site: http://www.atheistberlin.com/study http://www.atheistberlin.com/study Not sure it will completely sate you but it's a start.
  - ghshephard 14y ago
    Thanks - their comments (below the infographic) are awesome - very level headed and it looks like they are going to publish a peer-reviewed paper.
- DanBC 14y ago
  (https://news.ycombinator.com/item?id=5442728 https://news.ycombinator.com/item?id=5442728) (http://www.atheistberlin.com/study http://www.atheistberlin.com/study)
xaritas 14y ago
This could be sabotage, but maybe somebody is just doing it for kicks. Either way, it's really too bad about all these lost soles.
- nealabq 14y ago
  The heels responsible should foot the bill and be given the boot.
  - mixmax 14y ago
    Almost as witty as the article headline announcing British politician Michael Foot being placed in charge of a committee on decommissioning nuclear weapons "Foot heads arms body"
tokenadult 14y ago
My all-time favorite link to share in Hacker News discussions about a preliminary research finding is the article "Warning Signs in Experimental Design and Interpretation" by Peter Norvig, LISP hacker and director of research at Google, on how to interpret scientific research. http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html As other comments have already pointed out, it's not clear from such a small sample size that the observed results can be generalized as a description of reality. If people are worried about this issue (and it does seem like a legitimate concern), try running lots of replication studies, with variations in study methodology and variations in experimental conditions, and see what happens. If I randomly receive a package, I'm suspicious in general, but maybe other people are glad to get surprises in the mail.
- Bud 14y ago
  I agree, in terms of the science in question. However, I will still give you 5:1 odds that this particular phenomenon is real and not due to a small sample size. Let me know if you want to take that action. :)
- leeny 14y ago
  poor experimental design/interpretation. great PR. if only the shoes themselves somehow harkened back to the rebellious brand image that this press created...
- shmageggy 14y ago
  Commenters on the blog pointed out a pretty glaring problem in the experimental setup; their control boxes contained tape with no writing on it at all, which opens up the possibility that package handlers were responding to the presence of text, rather than the content of it. A human handler might be concerned about terrorism or something, and an OCR might just get confused. Commenters suggested labeling controls with a neutral word, and also trying labels such as "Allah" for comparison.
- A1kmm 14y ago
  I agree that there are some things they could have done wrong that aren't explicitly ruled out in http://www.atheistberlin.com/study http://www.atheistberlin.com/study: Were the packages the same weight or were some of them heavier? Were the packages given over to the same post office, or were some packages given to one office and some to another? What order were the packages handed over to the post office? Assuming that there were no errors like that, this does appear to be a well designed experiment, and they are using rigorous statistical methods that are appropriate to carry out the tests, so it is entirely valid to draw causal inference that packages of that weight delivered on that day to that post office were caused to be delivered later if they had the atheist tape on them. It would be hard to generalise that more to say that every package delivered from anywhere to the US on any day will be delayed. It is also not necessarily possible to jump from the valid causal inference above to "USPS employs bigots" without further experimentation - perhaps the difference occurred before the packages reached USPS? Perhaps USPS employees occasionally have to make a judgement call about the last packages to go into a crate, and they are more likely to put in a package that might be time sensitive than one with tape that implies it probably isn't (it would be interesting to see the impact of other labelled tapes, like Christian, Muslim, Toys, Business Documents, Baby Formula, Shoes and so on).
deleted 14y ago
[deleted]
robomartin 14y ago
If one had several thousand dollars to waste it'd be interesting to run a larger test with a range of deities as well as "atheist", "flying spaghetti monster", the little fish symbol and other religious and non-religious words and symbology. I'd throw in stuff like Mickey Mouse, Obama and George Bush images in there to have some fun. The results could be very interesting.
- ryguytilidie 14y ago
  Not really though. Does finding that many religious people in the US are insanely intolerant really prove anything?
  - saraid216 14y ago
    It proves that God is a third variable? Hidden. Unknowable. The connecting thread between two completely unrelated incidents.
panhandlr 14y ago
Study: Controversy is 10 times more likely to sell shoes
utopkara 14y ago
I don't care if I get downvotes for saying this, but I think this story is full of crap. Such a small sample size could only be considered acceptable if there was overwhelming consistence. Also, the USPS system is not a black box, it is well known that they optimize for cost, not speed. e.g., if the adresses happen to not have the 9-digit zip, the packages will have to go through manual sorting, probably understaffed. I'd say no real atheist would trumpet for this crap quasi-science, only those who pretend to be one just for making a quick buck would.
- rorrr 14y ago
  Your opinion is crap. 10X difference on such a sample size is statistically significant.
olalonde 14y ago
Previous discussion: https://news.ycombinator.com/item?id=5442728 https://news.ycombinator.com/item?id=5442728
JohnBooty 14y ago
I realize this is purely anecdotal, but my uncle worked for the USPS as a mail carrier in a major East Coast city for decades and saw people fired for really small violations. In one case, a guy with many years of seniority was fired because he stole some spam-mail coupons that a company had mailed to a non-existent address. I'm not saying the USPS is a model of efficiency and I'm sure there are regional variations on strictness and oversight, but it's pretty likely that any USPS who loses (or "loses") a greater-than-average number of packages is risking their job. I don't doubt that there are people who'd object to a package marked "ATHIEST" but how many USPS employees would be so offended they'd risk their jobs? USPS workers often work there for decades and a lot of USPS employees have an eye towards staying there until retirement. They also get good health benefits, a nice government pension, and you don't need a higher education diploma to get hired by the USPS in the first place. A USPS employee is unlikely to have a lot of comparable job opportunities after spending years with the USPS, so I have a hard time believing so many of them would be like, "ATHIEST? Screw this!" and simply punt a package they disliked off of a bridge.

75 points by bado 14y ago

Sad.

I'd like to see the actual survey data. I'm not even sure how to interpret this: "Thus the ad hoc study: The company says it sent two packages each to 89 people (178 packages total) canvassing nearly every U.S. state — one package with the Atheist-branded tape, one without. And this is where the results suggest something fishy: According to Atheist Shoes, company-branded packages took on average three days longer to reach their destination and were 10 times more likely to disappear outright." What does "10 times more likely to disappear outright" mean? Are they saying that 10 of the 89 Atheist packages disappeared, and only 1 of the 89 non-atheist packages disappeared?

simonsez 14y ago
Agreed - the sample size is also really small to be interpreting slower arrival times as well, if only 1 or 2 packages took a long time it's likely they just found an outlier.
- jlgreco 14y ago
  Yeah, that 3 days could either be damning, or absolutely nothing. Until we can examine the data this is a non-story despite how juicy we may want it to be.
- christopheraden 14y ago
  Sample size is something that must be interpreted in the presence of power. You can make a solid conclusion with a very small sample size if the true difference in arrival times is very large, given that the assumptions of the hypothesis test hold (t-test can be a little ridiculous with some of its assumptions sometimes). In the original article, one of the footnotes mentioned that they tested the data using Wilcoxon's Signed-Rank test, which mitigates a lot of the impact of single outliers. I'd love to see the raw data though, to see an even less-sensitive method to outliers (sign test). If the difference between the groups is as large as the article would lead us to believe, the loss of power should not present any problem.
- A1kmm 14y ago
  According to http://www.atheistberlin.com/study http://www.atheistberlin.com/study they got a significant Wilcoxon Signed Rank Test at p < 0.01. The Wilcoxon Signed Rank Test is nonparametric (i.e. sacrifices some statistical power in order to not make assumptions about the underlying distribution of the data) so is meaningful even if there is a long tail of packages that take longer due to circumstances outside of the study variable.
pavanky 14y ago
Yep pretty much. In the original article (not from time.com) it was 9 and 1. So I am not sure where the number came from.
deleted 14y ago
[deleted]
citricsquid 14y ago
I can't find the previous submission but someone pointed out that one of the Atheist labelled packages was lost for almost 40 days: that alone dragged the Atheist package average down so much it can almost account for the entire difference. edit: I misremembered about the difference it made, here's the submission: https://news.ycombinator.com/item?id=5442728 https://news.ycombinator.com/item?id=5442728
codezero 14y ago
Check out the comments on their own web site: http://www.atheistberlin.com/study http://www.atheistberlin.com/study Not sure it will completely sate you but it's a start.
- ghshephard 14y ago
  Thanks - their comments (below the infographic) are awesome - very level headed and it looks like they are going to publish a peer-reviewed paper.
DanBC 14y ago
(https://news.ycombinator.com/item?id=5442728 https://news.ycombinator.com/item?id=5442728) (http://www.atheistberlin.com/study http://www.atheistberlin.com/study)

xaritas 14y ago

This could be sabotage, but maybe somebody is just doing it for kicks. Either way, it's really too bad about all these lost soles.

nealabq 14y ago
The heels responsible should foot the bill and be given the boot.
- mixmax 14y ago
  Almost as witty as the article headline announcing British politician Michael Foot being placed in charge of a committee on decommissioning nuclear weapons "Foot heads arms body"

tokenadult 14y ago

My all-time favorite link to share in Hacker News discussions about a preliminary research finding is the article "Warning Signs in Experimental Design and Interpretation" by Peter Norvig, LISP hacker and director of research at Google, on how to interpret scientific research. http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html As other comments have already pointed out, it's not clear from such a small sample size that the observed results can be generalized as a description of reality. If people are worried about this issue (and it does seem like a legitimate concern), try running lots of replication studies, with variations in study methodology and variations in experimental conditions, and see what happens. If I randomly receive a package, I'm suspicious in general, but maybe other people are glad to get surprises in the mail.

Bud 14y ago
I agree, in terms of the science in question. However, I will still give you 5:1 odds that this particular phenomenon is real and not due to a small sample size. Let me know if you want to take that action. :)
leeny 14y ago
poor experimental design/interpretation. great PR. if only the shoes themselves somehow harkened back to the rebellious brand image that this press created...
shmageggy 14y ago
Commenters on the blog pointed out a pretty glaring problem in the experimental setup; their control boxes contained tape with no writing on it at all, which opens up the possibility that package handlers were responding to the presence of text, rather than the content of it. A human handler might be concerned about terrorism or something, and an OCR might just get confused. Commenters suggested labeling controls with a neutral word, and also trying labels such as "Allah" for comparison.
A1kmm 14y ago
I agree that there are some things they could have done wrong that aren't explicitly ruled out in http://www.atheistberlin.com/study http://www.atheistberlin.com/study: Were the packages the same weight or were some of them heavier? Were the packages given over to the same post office, or were some packages given to one office and some to another? What order were the packages handed over to the post office? Assuming that there were no errors like that, this does appear to be a well designed experiment, and they are using rigorous statistical methods that are appropriate to carry out the tests, so it is entirely valid to draw causal inference that packages of that weight delivered on that day to that post office were caused to be delivered later if they had the atheist tape on them. It would be hard to generalise that more to say that every package delivered from anywhere to the US on any day will be delayed. It is also not necessarily possible to jump from the valid causal inference above to "USPS employs bigots" without further experimentation - perhaps the difference occurred before the packages reached USPS? Perhaps USPS employees occasionally have to make a judgement call about the last packages to go into a crate, and they are more likely to put in a package that might be time sensitive than one with tape that implies it probably isn't (it would be interesting to see the impact of other labelled tapes, like Christian, Muslim, Toys, Business Documents, Baby Formula, Shoes and so on).

deleted 14y ago

[deleted]

robomartin 14y ago

If one had several thousand dollars to waste it'd be interesting to run a larger test with a range of deities as well as "atheist", "flying spaghetti monster", the little fish symbol and other religious and non-religious words and symbology. I'd throw in stuff like Mickey Mouse, Obama and George Bush images in there to have some fun. The results could be very interesting.

ryguytilidie 14y ago
Not really though. Does finding that many religious people in the US are insanely intolerant really prove anything?
- saraid216 14y ago
  It proves that God is a third variable? Hidden. Unknowable. The connecting thread between two completely unrelated incidents.

panhandlr 14y ago

Study: Controversy is 10 times more likely to sell shoes

utopkara 14y ago

I don't care if I get downvotes for saying this, but I think this story is full of crap. Such a small sample size could only be considered acceptable if there was overwhelming consistence. Also, the USPS system is not a black box, it is well known that they optimize for cost, not speed. e.g., if the adresses happen to not have the 9-digit zip, the packages will have to go through manual sorting, probably understaffed. I'd say no real atheist would trumpet for this crap quasi-science, only those who pretend to be one just for making a quick buck would.

rorrr 14y ago
Your opinion is crap. 10X difference on such a sample size is statistically significant.

olalonde 14y ago

Previous discussion: https://news.ycombinator.com/item?id=5442728 https://news.ycombinator.com/item?id=5442728

JohnBooty 14y ago

I realize this is purely anecdotal, but my uncle worked for the USPS as a mail carrier in a major East Coast city for decades and saw people fired for really small violations. In one case, a guy with many years of seniority was fired because he stole some spam-mail coupons that a company had mailed to a non-existent address. I'm not saying the USPS is a model of efficiency and I'm sure there are regional variations on strictness and oversight, but it's pretty likely that any USPS who loses (or "loses") a greater-than-average number of packages is risking their job. I don't doubt that there are people who'd object to a package marked "ATHIEST" but how many USPS employees would be so offended they'd risk their jobs? USPS workers often work there for decades and a lot of USPS employees have an eye towards staying there until retirement. They also get good health benefits, a nice government pension, and you don't need a higher education diploma to get hired by the USPS in the first place. A USPS employee is unlikely to have a lot of comparable job opportunities after spending years with the USPS, so I have a hard time believing so many of them would be like, "ATHIEST? Screw this!" and simply punt a package they disliked off of a bridge.