4 ms·

You are over-simplifying. Almost all of the metrics Facebook will feed you when you setup an account are on a view-through attribution basis. Yes, there will

You are over-simplifying. Almost all of the metrics Facebook will feed you when you setup an account are on a view-through attribution basis. Yes, there will be some clicks, but if you attribute revenue from Facebook by clicks only, it will look terrible. This is an old problem for anyone that has spent time with display ad platforms. So, you install a facebook pixel on your checkout, and if the person making a purchase saw your ad on Facebook, Facebook's reporting console will take credit for that. Thats view-through attribution and it highly overstates the credit Facebook should receive for making the sale, but there's not much you can do to measure it otherwise. So what do you do? The answer is, you setup a holdout test. Facebook has a black-box solution for helping with this, but it is a huge conflict of interest to let Facebook handle your test. The thing to do then is set up the test on your own platform. That is a massive undertaking, and any business that doesn't have a significantly sized engineering and analytic team doesn't stand a chance. So yea, people look at things like reach, and view through conversions, and it would be very hard not too or they might not have the means to do something more sophisticated. Fake accounts undeniably puff of numbers for these people, and they should be taken to court for not giving a shit about that fact.

disgruntledphd2 7y ago
Facebook are 100% not going to lie to you through their experimentation platform. The whole point of it is to help people accurately measure the results of their ads. In general, it will be extremely difficult to run accurate tests on FB (or indeed Google) ads without using some of their infrastructure, as these platforms can balance users appropriately in terms of likely response to your ads (for instance, ensuring that both ads are shown to people with approximately the same click probability). Without this, you run the risk of making bad decisions because one ad got served to users who were far more likely to convert. Of course you should definitely run your own tests, but I wouldn't completely discount the platform's tools as they do have advantages which are impossible for you to replicate (i.e. balancing users in terms of response rates in each condition).
- tjpaudio 7y ago
  Yea, but what are they using to analyze your results? You literally have no idea how robust (or not) their statistical methods are. It's been my experience that 3rd party solutions are notoriously shoddy in this department. They have no incentive to be robust.
  - disgruntledphd2 7y ago
    AFAIK (it's been over a year), they use intent to treat approach, and only remove users from the experiment after they have already been matched to an ad. This allows them to ensure that the users are matched in terms of expected conversion rate, which allows for valid results. I believe that there were rumours of using bayesian analysis in the future, but I believe it's a two-sided p-value based on the differences in your chosen outcome. Reasonably simple, and mostly effective.

disgruntledphd2 7y ago

Facebook are 100% not going to lie to you through their experimentation platform. The whole point of it is to help people accurately measure the results of their ads. In general, it will be extremely difficult to run accurate tests on FB (or indeed Google) ads without using some of their infrastructure, as these platforms can balance users appropriately in terms of likely response to your ads (for instance, ensuring that both ads are shown to people with approximately the same click probability). Without this, you run the risk of making bad decisions because one ad got served to users who were far more likely to convert. Of course you should definitely run your own tests, but I wouldn't completely discount the platform's tools as they do have advantages which are impossible for you to replicate (i.e. balancing users in terms of response rates in each condition).

tjpaudio 7y ago
Yea, but what are they using to analyze your results? You literally have no idea how robust (or not) their statistical methods are. It's been my experience that 3rd party solutions are notoriously shoddy in this department. They have no incentive to be robust.
- disgruntledphd2 7y ago
  AFAIK (it's been over a year), they use intent to treat approach, and only remove users from the experiment after they have already been matched to an ad. This allows them to ensure that the users are matched in terms of expected conversion rate, which allows for valid results. I believe that there were rumours of using bayesian analysis in the future, but I believe it's a two-sided p-value based on the differences in your chosen outcome. Reasonably simple, and mostly effective.