3 ms·

Selection bias exists, but it doesn't explain away numbers like "700,000" and "94%". The fact is that Facebook pissed off a huge number of their users, for see

by gamache 18y ago

Selection bias exists, but it doesn't explain away numbers like "700,000" and "94%". The fact is that Facebook pissed off a huge number of their users, for seemingly no good reason, as the previous design was quite popular and functional.

jfarmer 18y ago
First, 700,000 isn't "huge" for Facebook. They gain about that many new users in a day. Second, the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. The poll is designed in a way that practically guarantees bias: a self-selected sample, a hot-button issue where one side cares much more strongly than the other, and the viral hooks of Facebook to make sure it gets spread far and wide. It's worthless if you're looking to draw any real conclusions about how Facebook users feel about the new homepage. It's not worthless if you're looking to exploit people's confirmation bias and get an article all over the web. So, YMMV, I guess.
- fixie 18y ago
  Reference for ~700,000 new users per day http://www.insidefacebook.com/2008/12/16/facebook-now-growing-by-over-600000-users-a-day-and-new-engagement-stats/ http://www.insidefacebook.com/2008/12/16/facebook-now-growin...
  - gamache 18y ago
    Thank you for this link. It also reports 13 million users log in once per day. That, in the context of Facebook, is what I'd call an "active user". These, overwhelmingly, are the people who are taking this poll. 700,000 therefore represents about 5% of the core userbase.
    - teej 18y ago
      > 700,000 therefore represents about 5% of the core userbase. Your math here is really flawed. Facebook's daily uniques are a magnitude smaller than their weekly and monthly uniques. Since this survey has been up since the redesign, you would have to compare 700k to the number of uniques over that period of time. I can assure you that number is much greater than 13M.
      - gamache 18y ago
        I'd bet you a dollar that at least 3/4 of the poll-takers are dailies. These are the users who really are taking this personally.
        teej 18y ago
        The whole point of this argument has been over statistical invalidity. Your guesses are pointless without data.
      - Dauntless 18y ago
        What? Your logic is as if all users knew about this poll, when I guess that the number of people checking for new apps is extremely tiny compared to the entire user base. I don’t even knew about this app, and it's more likely that 700k is very close to how many actually look for new apps on facebook during this period.
- chrisbolt 18y ago
  Another point: of Facebook's 175 million active users, 700,000 represents 0.4%.
  - gamache 18y ago
    Here, "active user" means "registered account". It's not a good indicator of who is actually using the site on a regular basis.
    - jfarmer 18y ago
      Generally FB takes "active" to mean "logged on within the last 30 days." That includes new users and returning users, but not users who haven't been on for the last month.
- gamache 18y ago
  Please, if you're going to act as a proponent of responsible statistics, don't exaggerate like you did in your first line. (EDIT: wow, this is not an exaggeration. An adjacent link confirms it. Currently eating my words.) True, 700,000 is not a staggering number for Facebook, but is a statistically signifiant portion of their most active users. I wouldn't put money on the figure of 94%, but I think it correctly indicates a strong trend.
  - jfarmer 18y ago
    I didn't exaggerate anything. Two years ago they were growing at 200-300k users per day. Dave & Ami were using those numbers in public talks. So, let's be generous and say instead that 700k users represents two or three days worth of growth rather than one. It actually doesn't matter with respect to the more salient point, which is that talk of "statistical significance" is irrelevant because the calculations require an unbiased sample. As I outlined above, there are at least three strong reasons to suspect this poll is irreparably biased. Self-selection alone is typically enough to condemn a poll. Look up the history of Gallup if you want to see the real-world impact of self-selection in polling.
- andylei 18y ago
  selection bias is compounded by the fact the costs of voting. i like the redesign, and i was going to vote YES, but it's not a Facebook poll, it's an application. I'd have to sign up to use the app, allowing it access to my user data, just to vote. No thanks. I bet most people who like the redesign don't want to jump through all the extra hoops to vote yes.
- tvon 18y ago
  Yeah, people who are happy with the new layout (or don't care) are not likely to install an app just to respond to the poll.
  - unalone 18y ago
    I took a few days before adding the app to vote yes, and I just did that because I was pissed off at the people voting no. For the record: perhaps 10 of my friends have voted no. This is not a significant percentage. Of those 10, probably about 8 of them did it as an impulse vote and 2 actually got annoyed at the layout.
- tokenadult 18y ago
  the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. Correct. Ordinarily, the ABSOLUTE size of a sample is more crucial than what percentage of the whole a sample makes up, for determining whether the sample is adequately large, but that is only if the sample is unbiased. If that precondition is not met, don't talk about how large the sample is, talk about how badly self-selected "samples" can misrepresent the characteristics of a population. Thanks for reminding HN readers about this important issue.
arockwell 18y ago
The one thing you're forgetting is that this happens _every_ time there's a new website redesign for Facebook. Its actually very typical for this to happen in general. People despise change will complain about a new design even if the design is much better. In a couple of weeks no one will even notice or think much of it.
jsmcgd 18y ago
Everytime single time that Facebook has changed the design almost everyone hates it and everytime almost everyone grows to like it.
jackowayed 18y ago
>the previous design was quite popular and functional. But there was just the same outrage over that design for a good month. I probably got 10 group invites to groups like "new facebook sucks!" and "don't visit facebook on (some date) to show them how much we hate new facebook." I'm not sure if anyone took a poll, even a biased one, but I haven't noticed among my friends any more outrage over this redesign than the last one.
baddox 18y ago
Everyone hated the news feed, yet now it's hard to think of wtf Facebook would be if it didn't have the feed. Everyone hated the first major redesign too, but now I'm sure everyone's used to it. They should do a test, wherein they switch the layout back to a previous layout then have people vote. I have a feeling this would show that the vocal majority would dislike ANY major change. I'm pretty sure Facebook's team has a lot better designers than any of the 700,000 voters.

by gamache 18y ago

jfarmer 18y ago

First, 700,000 isn't "huge" for Facebook. They gain about that many new users in a day. Second, the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. The poll is designed in a way that practically guarantees bias: a self-selected sample, a hot-button issue where one side cares much more strongly than the other, and the viral hooks of Facebook to make sure it gets spread far and wide. It's worthless if you're looking to draw any real conclusions about how Facebook users feel about the new homepage. It's not worthless if you're looking to exploit people's confirmation bias and get an article all over the web. So, YMMV, I guess.

fixie 18y ago
Reference for ~700,000 new users per day http://www.insidefacebook.com/2008/12/16/facebook-now-growing-by-over-600000-users-a-day-and-new-engagement-stats/ http://www.insidefacebook.com/2008/12/16/facebook-now-growin...
- gamache 18y ago
  Thank you for this link. It also reports 13 million users log in once per day. That, in the context of Facebook, is what I'd call an "active user". These, overwhelmingly, are the people who are taking this poll. 700,000 therefore represents about 5% of the core userbase.
  - teej 18y ago
    > 700,000 therefore represents about 5% of the core userbase. Your math here is really flawed. Facebook's daily uniques are a magnitude smaller than their weekly and monthly uniques. Since this survey has been up since the redesign, you would have to compare 700k to the number of uniques over that period of time. I can assure you that number is much greater than 13M.
    - gamache 18y ago
      I'd bet you a dollar that at least 3/4 of the poll-takers are dailies. These are the users who really are taking this personally.
      - teej 18y ago
        The whole point of this argument has been over statistical invalidity. Your guesses are pointless without data.
    - Dauntless 18y ago
      What? Your logic is as if all users knew about this poll, when I guess that the number of people checking for new apps is extremely tiny compared to the entire user base. I don’t even knew about this app, and it's more likely that 700k is very close to how many actually look for new apps on facebook during this period.
chrisbolt 18y ago
Another point: of Facebook's 175 million active users, 700,000 represents 0.4%.
- gamache 18y ago
  Here, "active user" means "registered account". It's not a good indicator of who is actually using the site on a regular basis.
  - jfarmer 18y ago
    Generally FB takes "active" to mean "logged on within the last 30 days." That includes new users and returning users, but not users who haven't been on for the last month.
gamache 18y ago
Please, if you're going to act as a proponent of responsible statistics, don't exaggerate like you did in your first line. (EDIT: wow, this is not an exaggeration. An adjacent link confirms it. Currently eating my words.) True, 700,000 is not a staggering number for Facebook, but is a statistically signifiant portion of their most active users. I wouldn't put money on the figure of 94%, but I think it correctly indicates a strong trend.
- jfarmer 18y ago
  I didn't exaggerate anything. Two years ago they were growing at 200-300k users per day. Dave & Ami were using those numbers in public talks. So, let's be generous and say instead that 700k users represents two or three days worth of growth rather than one. It actually doesn't matter with respect to the more salient point, which is that talk of "statistical significance" is irrelevant because the calculations require an unbiased sample. As I outlined above, there are at least three strong reasons to suspect this poll is irreparably biased. Self-selection alone is typically enough to condemn a poll. Look up the history of Gallup if you want to see the real-world impact of self-selection in polling.
andylei 18y ago
selection bias is compounded by the fact the costs of voting. i like the redesign, and i was going to vote YES, but it's not a Facebook poll, it's an application. I'd have to sign up to use the app, allowing it access to my user data, just to vote. No thanks. I bet most people who like the redesign don't want to jump through all the extra hoops to vote yes.
tvon 18y ago
Yeah, people who are happy with the new layout (or don't care) are not likely to install an app just to respond to the poll.
- unalone 18y ago
  I took a few days before adding the app to vote yes, and I just did that because I was pissed off at the people voting no. For the record: perhaps 10 of my friends have voted no. This is not a significant percentage. Of those 10, probably about 8 of them did it as an impulse vote and 2 actually got annoyed at the layout.
tokenadult 18y ago
the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. Correct. Ordinarily, the ABSOLUTE size of a sample is more crucial than what percentage of the whole a sample makes up, for determining whether the sample is adequately large, but that is only if the sample is unbiased. If that precondition is not met, don't talk about how large the sample is, talk about how badly self-selected "samples" can misrepresent the characteristics of a population. Thanks for reminding HN readers about this important issue.

arockwell 18y ago

The one thing you're forgetting is that this happens _every_ time there's a new website redesign for Facebook. Its actually very typical for this to happen in general. People despise change will complain about a new design even if the design is much better. In a couple of weeks no one will even notice or think much of it.

jsmcgd 18y ago

Everytime single time that Facebook has changed the design almost everyone hates it and everytime almost everyone grows to like it.

jackowayed 18y ago

>the previous design was quite popular and functional. But there was just the same outrage over that design for a good month. I probably got 10 group invites to groups like "new facebook sucks!" and "don't visit facebook on (some date) to show them how much we hate new facebook." I'm not sure if anyone took a poll, even a biased one, but I haven't noticed among my friends any more outrage over this redesign than the last one.

baddox 18y ago

Everyone hated the news feed, yet now it's hard to think of wtf Facebook would be if it didn't have the feed. Everyone hated the first major redesign too, but now I'm sure everyone's used to it. They should do a test, wherein they switch the layout back to a previous layout then have people vote. I have a feeling this would show that the vocal majority would dislike ANY major change. I'm pretty sure Facebook's team has a lot better designers than any of the 700,000 voters.