18 ms·

If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, the

by Osiris 6y ago

If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, they still only had 5 people assigned to it while it increased in importance to 50% of revenue?

simonh 6y ago
I've seen this sort of thing happen many times. What usually happens is the original team that built it leave the company or move on to other teams, and the one engineer left actually has a different job as well now but is stuck supporting it part time as a favour. It's not that you couldn't assign other engineers to help support it, but they wouldn't have any idea how to do so because there's probably no documentation and only that half an engineer around to show them how it works.
- disgruntledphd2 6y ago
  It was because Facebook practiced open allocation for engineers at the time. Essentially, you had a choice of teams and the manager would need to convince the engineer to work on the project. At the time, ads was not a high status part of FB engineering, and as such, it was very difficult to get people to work on ads products.
  - ForHackernews 6y ago
    It's funny that people want to work at Facebook or Google but don't want to work on ads. In other companies, everyone wants to work near the profit center and avoid cost centers.
    - rvba 6y ago
      Because those projects are hard. It is much easier to make a greenfield project that will get 70% done and cancelled before anyone even asks to fix bugs.
      - username90 6y ago
        Can't have been that hard if half an engineer was enough to do the job.
        simonh 6y ago
        The half an engineer was just the last team member around, only doing essential maintenance to keep it working.
        username90 6y ago
        So why allocate more engineers to the project when it was doing fine with half a head?
        kelnos 6y ago
        The article states that they needed to rebuild most of it because the entire thing depended on a deprecated Chrome feature that could be removed at any time.
        rconti 6y ago
        Because the collections agent for that technical debt hadn't called _yet_
        simonh 6y ago
        To borrow a phrase from the investment industry, the fact that a ticking time bomb has never previously gone off is not always a reliable indication of future performance.
    - deleted 6y ago
      [deleted]
  - throwawayboise 6y ago
    Hard to imagine that anything responsible for 25% of revenue (billions of dollars, at FB scale) would not be seen as "high status" if not by the engineers, then by management.
    - shaneos 6y ago
      [author] It wasn't. The plan at the time was to deprecate Power Editor and build a new version of Ads Manager. We undertook both projects simultaneously, with the plan being to keep Power Editor going until AM v2 was ready. Unfortunately that took too long, and since Power Editor was the only product that could ship all the new ads features, by the time it came that Ads Manager V2 was ready, Power Editor had 50% of revenue share and Ads Manager about 20%. So, in effect we rebadged PE as Ads Manager, pulled over some of the new features from AM V2 and called it done.
  - shaneos 6y ago
    [author] This is very accurate. As the manager of the team I had to spend a huge amount of time trying to hire internally for it. No one came to FB to work on ads, they wanted to work on stuff that their friends used. I had to develop a smooth song-and-dance to get the right type of people interested. In the end, this served me very well in my career, being well practiced at getting engineers excited to join your team is a valuable skill.
Cthulhu_ 6y ago
I hope those 13 engineers got paid millions. So much weight and responsibility on their shoulders! They were responsible for nearly $2 billion in 2013 (25% of $7.87bn), and nearly $14bn in 2016 when the work was finished (50% of $27.63bn)
- simonh 6y ago
  I see this sort of comment occasionally. If those engineers hadn't been assigned to work on the project, somebody else would have. It doesn't look like they did that fantastic a job of it either. Also this was a front-end to managing the ad system, not the ad system itself, or the infrastructure it ran on, or the features of facebook that drew in the audience in the first place, etc, etc. Yes some engineers genuinely do make contributions that have unique value, but frankly most of us are guns for hire and if we're lucky enough to get on a fun, interesting and valuable project that's great, but frankly one out of three is the best we usually get.
  - komali2 6y ago
    Smacks of maybe survivorship bias. If OP hadn't noticed this chrome dependency, how are we to speculate on the likelihood of someone else noticing it? They even said nobody else wanted to work on it.
    - lupire 6y ago
      Well OP is tooting their own horn
- shaneos 6y ago
  [author] Many did and it was well earned. The team was awesome and had great camaraderie, but the work was horrible. Everything that broke led to a SEV for our team, even though 80% of the time it was not our code (somewhere deep in the API that had insufficient logging and observability). All our customers hated the product since it was so buggy, and our internal customers (sales ppl at FB) hated it for the same reason. They didn't see the 16 hour days, nights and weekend that we were all putting in to fight fires, do a fundamental rewrite from two JS frameworks to an untested new one (React), and add hundreds of new product features to keep up with FB's exploding Ads org. It was three years of hell to be frank, but very rewarding if you're into that kind of thing. It took a special type of person to join that team, knowing how long it would be before any payoff was really visible in the product, all while being vilified inside and outside the company even though you didn't build the thing, you just agreed to jump in and help fix it.
obviouslynotme 6y ago
You see this all the time. As companies get larger, they become more and more difficult to understand. This is even more so the case with software companies since they are essentially complexity factories.
ska 6y ago
> If the product was responsible for 25% Being a (however necessary) part of the toolchain is importantly different than being responsible for the revenue. Such tools often fall under "it ain't broke, don't mess with it". Until they are broke, of course.
- shaneos 6y ago
  [author] There were a number of critical parts of the chain responsible for FB Ads revenue. It all started at the UI or API (which were 75% and ~25% of FB revenue when I left in 2017). Then you had the DB layer, managed by another team, the AI/Targeting layer and one or two others. If any had failed fundamentally, FB was in big trouble. However, other than Power Editor and Chrome, the rest all ran in our data centers and were under our control.
shaneos 6y ago
[author here] Getting 5 engineers was actually a huge ask. Just a year earlier the entire Ads Interfaces org had 5 engineers in it (I was #6). That was 5 engineers for all API and front end code for a $4B/year revenue company. Facebook has a history of using tiny teams to lift impossible mountains - e.g. FB Photos was originally written by two engineers and overtook Flickr, the current leader, with just that investment.

by Osiris 6y ago

simonh 6y ago

I've seen this sort of thing happen many times. What usually happens is the original team that built it leave the company or move on to other teams, and the one engineer left actually has a different job as well now but is stuck supporting it part time as a favour. It's not that you couldn't assign other engineers to help support it, but they wouldn't have any idea how to do so because there's probably no documentation and only that half an engineer around to show them how it works.

disgruntledphd2 6y ago
It was because Facebook practiced open allocation for engineers at the time. Essentially, you had a choice of teams and the manager would need to convince the engineer to work on the project. At the time, ads was not a high status part of FB engineering, and as such, it was very difficult to get people to work on ads products.
- ForHackernews 6y ago
  It's funny that people want to work at Facebook or Google but don't want to work on ads. In other companies, everyone wants to work near the profit center and avoid cost centers.
  - rvba 6y ago
    Because those projects are hard. It is much easier to make a greenfield project that will get 70% done and cancelled before anyone even asks to fix bugs.
    - username90 6y ago
      Can't have been that hard if half an engineer was enough to do the job.
      - simonh 6y ago
        The half an engineer was just the last team member around, only doing essential maintenance to keep it working.
        username90 6y ago
        So why allocate more engineers to the project when it was doing fine with half a head?
        kelnos 6y ago
        The article states that they needed to rebuild most of it because the entire thing depended on a deprecated Chrome feature that could be removed at any time.
        rconti 6y ago
        Because the collections agent for that technical debt hadn't called _yet_
        simonh 6y ago
        To borrow a phrase from the investment industry, the fact that a ticking time bomb has never previously gone off is not always a reliable indication of future performance.
  - deleted 6y ago
    [deleted]
- throwawayboise 6y ago
  Hard to imagine that anything responsible for 25% of revenue (billions of dollars, at FB scale) would not be seen as "high status" if not by the engineers, then by management.
  - shaneos 6y ago
    [author] It wasn't. The plan at the time was to deprecate Power Editor and build a new version of Ads Manager. We undertook both projects simultaneously, with the plan being to keep Power Editor going until AM v2 was ready. Unfortunately that took too long, and since Power Editor was the only product that could ship all the new ads features, by the time it came that Ads Manager V2 was ready, Power Editor had 50% of revenue share and Ads Manager about 20%. So, in effect we rebadged PE as Ads Manager, pulled over some of the new features from AM V2 and called it done.
- shaneos 6y ago
  [author] This is very accurate. As the manager of the team I had to spend a huge amount of time trying to hire internally for it. No one came to FB to work on ads, they wanted to work on stuff that their friends used. I had to develop a smooth song-and-dance to get the right type of people interested. In the end, this served me very well in my career, being well practiced at getting engineers excited to join your team is a valuable skill.

Cthulhu_ 6y ago

I hope those 13 engineers got paid millions. So much weight and responsibility on their shoulders! They were responsible for nearly $2 billion in 2013 (25% of $7.87bn), and nearly $14bn in 2016 when the work was finished (50% of $27.63bn)

simonh 6y ago
I see this sort of comment occasionally. If those engineers hadn't been assigned to work on the project, somebody else would have. It doesn't look like they did that fantastic a job of it either. Also this was a front-end to managing the ad system, not the ad system itself, or the infrastructure it ran on, or the features of facebook that drew in the audience in the first place, etc, etc. Yes some engineers genuinely do make contributions that have unique value, but frankly most of us are guns for hire and if we're lucky enough to get on a fun, interesting and valuable project that's great, but frankly one out of three is the best we usually get.
- komali2 6y ago
  Smacks of maybe survivorship bias. If OP hadn't noticed this chrome dependency, how are we to speculate on the likelihood of someone else noticing it? They even said nobody else wanted to work on it.
  - lupire 6y ago
    Well OP is tooting their own horn
shaneos 6y ago
[author] Many did and it was well earned. The team was awesome and had great camaraderie, but the work was horrible. Everything that broke led to a SEV for our team, even though 80% of the time it was not our code (somewhere deep in the API that had insufficient logging and observability). All our customers hated the product since it was so buggy, and our internal customers (sales ppl at FB) hated it for the same reason. They didn't see the 16 hour days, nights and weekend that we were all putting in to fight fires, do a fundamental rewrite from two JS frameworks to an untested new one (React), and add hundreds of new product features to keep up with FB's exploding Ads org. It was three years of hell to be frank, but very rewarding if you're into that kind of thing. It took a special type of person to join that team, knowing how long it would be before any payoff was really visible in the product, all while being vilified inside and outside the company even though you didn't build the thing, you just agreed to jump in and help fix it.

obviouslynotme 6y ago

You see this all the time. As companies get larger, they become more and more difficult to understand. This is even more so the case with software companies since they are essentially complexity factories.

ska 6y ago

> If the product was responsible for 25% Being a (however necessary) part of the toolchain is importantly different than being responsible for the revenue. Such tools often fall under "it ain't broke, don't mess with it". Until they are broke, of course.

shaneos 6y ago
[author] There were a number of critical parts of the chain responsible for FB Ads revenue. It all started at the UI or API (which were 75% and ~25% of FB revenue when I left in 2017). Then you had the DB layer, managed by another team, the AI/Targeting layer and one or two others. If any had failed fundamentally, FB was in big trouble. However, other than Power Editor and Chrome, the rest all ran in our data centers and were under our control.

shaneos 6y ago

[author here] Getting 5 engineers was actually a huge ask. Just a year earlier the entire Ads Interfaces org had 5 engineers in it (I was #6). That was 5 engineers for all API and front end code for a $4B/year revenue company. Facebook has a history of using tiny teams to lift impossible mountains - e.g. FB Photos was originally written by two engineers and overtook Flickr, the current leader, with just that investment.