3 ms·

First, 700,000 isn't "huge" for Facebook. They gain about that many new users in a day. Second, the number of respondees or the proportion responding a certai

by jfarmer 18y ago

First, 700,000 isn't "huge" for Facebook. They gain about that many new users in a day. Second, the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. The poll is designed in a way that practically guarantees bias: a self-selected sample, a hot-button issue where one side cares much more strongly than the other, and the viral hooks of Facebook to make sure it gets spread far and wide. It's worthless if you're looking to draw any real conclusions about how Facebook users feel about the new homepage. It's not worthless if you're looking to exploit people's confirmation bias and get an article all over the web. So, YMMV, I guess.

fixie 18y ago
Reference for ~700,000 new users per day http://www.insidefacebook.com/2008/12/16/facebook-now-growing-by-over-600000-users-a-day-and-new-engagement-stats/ http://www.insidefacebook.com/2008/12/16/facebook-now-growin...
- gamache 18y ago
  Thank you for this link. It also reports 13 million users log in once per day. That, in the context of Facebook, is what I'd call an "active user". These, overwhelmingly, are the people who are taking this poll. 700,000 therefore represents about 5% of the core userbase.
  - teej 18y ago
    > 700,000 therefore represents about 5% of the core userbase. Your math here is really flawed. Facebook's daily uniques are a magnitude smaller than their weekly and monthly uniques. Since this survey has been up since the redesign, you would have to compare 700k to the number of uniques over that period of time. I can assure you that number is much greater than 13M.
    - gamache 18y ago
      I'd bet you a dollar that at least 3/4 of the poll-takers are dailies. These are the users who really are taking this personally.
      - teej 18y ago
        The whole point of this argument has been over statistical invalidity. Your guesses are pointless without data.
    - Dauntless 18y ago
      What? Your logic is as if all users knew about this poll, when I guess that the number of people checking for new apps is extremely tiny compared to the entire user base. I don’t even knew about this app, and it's more likely that 700k is very close to how many actually look for new apps on facebook during this period.
chrisbolt 18y ago
Another point: of Facebook's 175 million active users, 700,000 represents 0.4%.
- gamache 18y ago
  Here, "active user" means "registered account". It's not a good indicator of who is actually using the site on a regular basis.
  - jfarmer 18y ago
    Generally FB takes "active" to mean "logged on within the last 30 days." That includes new users and returning users, but not users who haven't been on for the last month.
gamache 18y ago
Please, if you're going to act as a proponent of responsible statistics, don't exaggerate like you did in your first line. (EDIT: wow, this is not an exaggeration. An adjacent link confirms it. Currently eating my words.) True, 700,000 is not a staggering number for Facebook, but is a statistically signifiant portion of their most active users. I wouldn't put money on the figure of 94%, but I think it correctly indicates a strong trend.
- jfarmer 18y ago
  I didn't exaggerate anything. Two years ago they were growing at 200-300k users per day. Dave & Ami were using those numbers in public talks. So, let's be generous and say instead that 700k users represents two or three days worth of growth rather than one. It actually doesn't matter with respect to the more salient point, which is that talk of "statistical significance" is irrelevant because the calculations require an unbiased sample. As I outlined above, there are at least three strong reasons to suspect this poll is irreparably biased. Self-selection alone is typically enough to condemn a poll. Look up the history of Gallup if you want to see the real-world impact of self-selection in polling.
andylei 18y ago
selection bias is compounded by the fact the costs of voting. i like the redesign, and i was going to vote YES, but it's not a Facebook poll, it's an application. I'd have to sign up to use the app, allowing it access to my user data, just to vote. No thanks. I bet most people who like the redesign don't want to jump through all the extra hoops to vote yes.
tvon 18y ago
Yeah, people who are happy with the new layout (or don't care) are not likely to install an app just to respond to the poll.
- unalone 18y ago
  I took a few days before adding the app to vote yes, and I just did that because I was pissed off at the people voting no. For the record: perhaps 10 of my friends have voted no. This is not a significant percentage. Of those 10, probably about 8 of them did it as an impulse vote and 2 actually got annoyed at the layout.
tokenadult 18y ago
the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. Correct. Ordinarily, the ABSOLUTE size of a sample is more crucial than what percentage of the whole a sample makes up, for determining whether the sample is adequately large, but that is only if the sample is unbiased. If that precondition is not met, don't talk about how large the sample is, talk about how badly self-selected "samples" can misrepresent the characteristics of a population. Thanks for reminding HN readers about this important issue.

by jfarmer 18y ago

fixie 18y ago

Reference for ~700,000 new users per day http://www.insidefacebook.com/2008/12/16/facebook-now-growing-by-over-600000-users-a-day-and-new-engagement-stats/ http://www.insidefacebook.com/2008/12/16/facebook-now-growin...

gamache 18y ago
Thank you for this link. It also reports 13 million users log in once per day. That, in the context of Facebook, is what I'd call an "active user". These, overwhelmingly, are the people who are taking this poll. 700,000 therefore represents about 5% of the core userbase.
- teej 18y ago
  > 700,000 therefore represents about 5% of the core userbase. Your math here is really flawed. Facebook's daily uniques are a magnitude smaller than their weekly and monthly uniques. Since this survey has been up since the redesign, you would have to compare 700k to the number of uniques over that period of time. I can assure you that number is much greater than 13M.
  - gamache 18y ago
    I'd bet you a dollar that at least 3/4 of the poll-takers are dailies. These are the users who really are taking this personally.
    - teej 18y ago
      The whole point of this argument has been over statistical invalidity. Your guesses are pointless without data.
  - Dauntless 18y ago
    What? Your logic is as if all users knew about this poll, when I guess that the number of people checking for new apps is extremely tiny compared to the entire user base. I don’t even knew about this app, and it's more likely that 700k is very close to how many actually look for new apps on facebook during this period.

chrisbolt 18y ago

Another point: of Facebook's 175 million active users, 700,000 represents 0.4%.

gamache 18y ago
Here, "active user" means "registered account". It's not a good indicator of who is actually using the site on a regular basis.
- jfarmer 18y ago
  Generally FB takes "active" to mean "logged on within the last 30 days." That includes new users and returning users, but not users who haven't been on for the last month.

gamache 18y ago

Please, if you're going to act as a proponent of responsible statistics, don't exaggerate like you did in your first line. (EDIT: wow, this is not an exaggeration. An adjacent link confirms it. Currently eating my words.) True, 700,000 is not a staggering number for Facebook, but is a statistically signifiant portion of their most active users. I wouldn't put money on the figure of 94%, but I think it correctly indicates a strong trend.

jfarmer 18y ago
I didn't exaggerate anything. Two years ago they were growing at 200-300k users per day. Dave & Ami were using those numbers in public talks. So, let's be generous and say instead that 700k users represents two or three days worth of growth rather than one. It actually doesn't matter with respect to the more salient point, which is that talk of "statistical significance" is irrelevant because the calculations require an unbiased sample. As I outlined above, there are at least three strong reasons to suspect this poll is irreparably biased. Self-selection alone is typically enough to condemn a poll. Look up the history of Gallup if you want to see the real-world impact of self-selection in polling.

andylei 18y ago

selection bias is compounded by the fact the costs of voting. i like the redesign, and i was going to vote YES, but it's not a Facebook poll, it's an application. I'd have to sign up to use the app, allowing it access to my user data, just to vote. No thanks. I bet most people who like the redesign don't want to jump through all the extra hoops to vote yes.

tvon 18y ago

Yeah, people who are happy with the new layout (or don't care) are not likely to install an app just to respond to the poll.

unalone 18y ago
I took a few days before adding the app to vote yes, and I just did that because I was pissed off at the people voting no. For the record: perhaps 10 of my friends have voted no. This is not a significant percentage. Of those 10, probably about 8 of them did it as an impulse vote and 2 actually got annoyed at the layout.

tokenadult 18y ago

the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. Correct. Ordinarily, the ABSOLUTE size of a sample is more crucial than what percentage of the whole a sample makes up, for determining whether the sample is adequately large, but that is only if the sample is unbiased. If that precondition is not met, don't talk about how large the sample is, talk about how badly self-selected "samples" can misrepresent the characteristics of a population. Thanks for reminding HN readers about this important issue.