6 ms·

Facebook's Ads System Suffers Glitch Days Before Black Friday

78 points by mathattack 8y ago

puzzle 8y ago
Was this caused by changes rushed before a holiday code freeze? I know nothing about FB, but it's customary for companies to stop pushing new binaries in early-mid December and to leave ad systems alone a week before Black Friday or so.
- mathattack 8y ago
  Much more than a week!
  - seanyesmunt 8y ago
    Yep. When I was at Walmart the code freeze started in early November, with only critical bug fixes going in, then continued until the start of January.
- kenhwang 8y ago
  I work in adtech and we went into code freeze about 2 weeks ago because the holidays are crazy.
- xiphias2 8y ago
  At another big company we had a code freeze as well, but we found a bug in a new feature that we released. We were not allowed to fix the bug, so we were reverting binaries of the data processing pipeline. The problem is that the data was already created and not backwards compatible, so we just created new problems... strict code freeze together with a huge feature push is always a nightmare.
  - fma 8y ago
    My big company has code freezes and no planned deploys around holidays... Bug fixes are allowed, just need deviations.
  - Drakim 8y ago
    That's why you should first have a feature freeze, and after a while go into a code freeze.
    - xiphias2 8y ago
      It took weeks for the bug to be surfaced. I was working with conversions, which is rare and noisy data. The problem is that most policies are not adjusted for the subteams.
- googlryas 8y ago
  Does facebook follow this practice? I know at least some other FAANG companies do, but does this go against 'move fast and break things'?
stygiansonic 8y ago
Probably related the other outages on Tuesday? https://www.nbcnews.com/tech/tech-news/outages-hit-facebook-s-apps-including-instagram-whatsapp-n938506 https://www.nbcnews.com/tech/tech-news/outages-hit-facebook-...
cm2012 8y ago
I can speak for all FB advertisers when I say this was a huge pain in the ass.
- craftyguy 8y ago
  Perhaps, but it sounds like a rare win for actual users of facebook.
  - cm2012 8y ago
    Facebook users, as a whole, like the Facebook ads I run for my clients. They click, share, comment and buy the products or services or advertise, which are all pretty solid. There's almost no where else on the web where my products (SMB services, mostly) can be connected to my customers (business owners) at scale and at a reasonable cost, by the way.
    - acct1771 8y ago
      Got an email address?
  - jonathanjaeger 8y ago
    The users were still seeing ads once Facebook, as a site, was back up. Just because campaign managers for ads couldn't use the Ads Manager, doesn't mean users don't see ads. For example, I had campaigns spending thousands of dollars, but I just had to guesstimate the cost because Ads Manager wasn't showing the data. That means I could be underspending or overspending because I couldn't optimize. It's a real headache for ad buyers, but once again, users still see ads.
  - deleted 8y ago
    [deleted]
- type0 8y ago
  I would rather give money to the mob than fb, at least those guys are more honest. I can't understand why anyone still thinks it's a good idea to give money to this monstrosity of a website.
sudovancity 8y ago
Hot take I know, but this has not bee a good year for Facebook.
dudurocha 8y ago
I work in an Ad Tech company and this week is crazy. We also try to make a code freeze but the reality is many deploys in production to fix things thursday evening...
Rjevski 8y ago
Good, less cancer for Black Friday I guess.
rachelbythebay 8y ago
Fixers need good air cover.

78 points by mathattack 8y ago

Was this caused by changes rushed before a holiday code freeze? I know nothing about FB, but it's customary for companies to stop pushing new binaries in early-mid December and to leave ad systems alone a week before Black Friday or so.

mathattack 8y ago
Much more than a week!
- seanyesmunt 8y ago
  Yep. When I was at Walmart the code freeze started in early November, with only critical bug fixes going in, then continued until the start of January.
kenhwang 8y ago
I work in adtech and we went into code freeze about 2 weeks ago because the holidays are crazy.
xiphias2 8y ago
At another big company we had a code freeze as well, but we found a bug in a new feature that we released. We were not allowed to fix the bug, so we were reverting binaries of the data processing pipeline. The problem is that the data was already created and not backwards compatible, so we just created new problems... strict code freeze together with a huge feature push is always a nightmare.
- fma 8y ago
  My big company has code freezes and no planned deploys around holidays... Bug fixes are allowed, just need deviations.
- Drakim 8y ago
  That's why you should first have a feature freeze, and after a while go into a code freeze.
  - xiphias2 8y ago
    It took weeks for the bug to be surfaced. I was working with conversions, which is rare and noisy data. The problem is that most policies are not adjusted for the subteams.
googlryas 8y ago
Does facebook follow this practice? I know at least some other FAANG companies do, but does this go against 'move fast and break things'?

stygiansonic 8y ago

Probably related the other outages on Tuesday? https://www.nbcnews.com/tech/tech-news/outages-hit-facebook-s-apps-including-instagram-whatsapp-n938506 https://www.nbcnews.com/tech/tech-news/outages-hit-facebook-...

cm2012 8y ago

I can speak for all FB advertisers when I say this was a huge pain in the ass.

craftyguy 8y ago
Perhaps, but it sounds like a rare win for actual users of facebook.
- cm2012 8y ago
  Facebook users, as a whole, like the Facebook ads I run for my clients. They click, share, comment and buy the products or services or advertise, which are all pretty solid. There's almost no where else on the web where my products (SMB services, mostly) can be connected to my customers (business owners) at scale and at a reasonable cost, by the way.
  - acct1771 8y ago
    Got an email address?
- jonathanjaeger 8y ago
  The users were still seeing ads once Facebook, as a site, was back up. Just because campaign managers for ads couldn't use the Ads Manager, doesn't mean users don't see ads. For example, I had campaigns spending thousands of dollars, but I just had to guesstimate the cost because Ads Manager wasn't showing the data. That means I could be underspending or overspending because I couldn't optimize. It's a real headache for ad buyers, but once again, users still see ads.
- deleted 8y ago
  [deleted]
type0 8y ago
I would rather give money to the mob than fb, at least those guys are more honest. I can't understand why anyone still thinks it's a good idea to give money to this monstrosity of a website.

sudovancity 8y ago

Hot take I know, but this has not bee a good year for Facebook.

dudurocha 8y ago

I work in an Ad Tech company and this week is crazy. We also try to make a code freeze but the reality is many deploys in production to fix things thursday evening...

Rjevski 8y ago

Good, less cancer for Black Friday I guess.

rachelbythebay 8y ago

Fixers need good air cover.