3 ms·

This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool qual

This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever. So it should be no wonder that even tests are measured. Flaky tests get disabled. That's nice and keeps trust high in the test framework. The downside is that you are likely to forget to fix the disabled test, so now some functionality that was important enough to test in the first place becomes untested and leaves room for others to introduce regressions. This has happened in my team, but the pace at FB is so quick that there is no way to tie up all these loose ends without giving up your personal time. Or maybe it requires someone more diligent than me. Either way, I just wanted to share that I personally found the idea of measuring test runs and acting on insights from thise measurements really powerful. It's a theme that can be found at all levels of FB infrastructure.

akhilcacharya 6y ago
> FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever. This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.
- 10x-dev 6y ago
  Not sure what you mean. There is no hard and fast top down enforced rules. Aside from natural Hack bias at FB, you can use any language you want. People develop tools in Rust, D, C++, bash, etc. As far as libraries go, there is a mandate to use in-house patched versions of various sdks and libraries, for security/customization reasons. Otherwise, go nuts. People write a whole lot of in-house software at FB, some of which is later open sourced. > That just doesn't happen at Amazon Oh please. We had a mandate to use Apollo. PHP is banned at Amazon (whatever your view of PHP is, that is a top down directive). The only freedom we actually had at Amazon was to use whatever open source software we could without contributing back to it (I'm sure there are examples where we did, ut not at the scale of FB). And lastly, Amazon was all about telling me what to do. One day I got a phone call and I got moved from front-end app development to backend c++. Nobody asked me if I wanted to do it or if I would be any good at it. FB spends months in bootcamp with teams trying to convince you to join them.
- Supermancho 6y ago
  > This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use That's not a natural consequence of the structure. If the top-down directives are issued by competent and financially accountable leadership, you get a nice efficiency pipeline.
AnonC 6y ago
> FB approaches internal code development with the same data driven rigor as their business decisions. How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature. I have a few anecdotes where, as a Facebook group administrator of some large (tens of thousands of members) and small groups (few thousand members), there were so many bugs in moderating the group with an added mega bug where the “Report issue to Facebook” (or something to that effect) would also throw an error. Those bugs lasted years (going into the current year) before I gave up and quit Facebook. Based on my experience over several years, I doubt there’s much rigor on many things. It may just be one of those marketing messages that have a little bit of truth and a lot of subterfuge.
- anonymoushn 6y ago
  In Facebook Ad Manager it is similarly hard to perform any task without having your day ruined.
- 10x-dev 6y ago
  My point was the tools are there and are pretty advanced, compared to what I see available outside of the company. Whether an internal team uses those tools effectively depends entirely on them and their expertise.
- vvwvvwvvw 6y ago
  They sure have a lot of rigor in sabotaging ad blocking efforts. If anything their anti anti ablock works all the time.
chii 6y ago
> tie up all these loose ends without giving up your personal time. so this just means they've crammed more into your backlog than you can do. I would not give up any personal, unpaid time to do it, unless there's some promotion that you need which this is a demonstration/investment for.
- 10x-dev 6y ago
  Seeing as I made the backlog, I have nobody but myself to blame :) I know what you mean though. Our deadlines are self-imposed, but the planning process is there to make sure that the effort remains high. I believe the managers are responsible for squeezing out as much juice as they can in a polite way (as opposed to other places where they were pretty direct about it). I suppose the combination of opportunity + compensation justifies the stress level and excitement at this point in my life. I've worked at other places where people primarily coasted with low pay but received good retirement benefits. It seemed to work for them.
hinkley 6y ago
I kinda miss working on teams where disabling the tests was plan B instead of plan G. The disabled tests should hopefully reduce code coverage. Also someone should be graphing out the volume and flux in disabled tests from week to week, to tap the brakes if it looks like we're rushing forward blindly instead of doing our due diligence.

akhilcacharya 6y ago

> FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever. This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.

10x-dev 6y ago
Not sure what you mean. There is no hard and fast top down enforced rules. Aside from natural Hack bias at FB, you can use any language you want. People develop tools in Rust, D, C++, bash, etc. As far as libraries go, there is a mandate to use in-house patched versions of various sdks and libraries, for security/customization reasons. Otherwise, go nuts. People write a whole lot of in-house software at FB, some of which is later open sourced. > That just doesn't happen at Amazon Oh please. We had a mandate to use Apollo. PHP is banned at Amazon (whatever your view of PHP is, that is a top down directive). The only freedom we actually had at Amazon was to use whatever open source software we could without contributing back to it (I'm sure there are examples where we did, ut not at the scale of FB). And lastly, Amazon was all about telling me what to do. One day I got a phone call and I got moved from front-end app development to backend c++. Nobody asked me if I wanted to do it or if I would be any good at it. FB spends months in bootcamp with teams trying to convince you to join them.
Supermancho 6y ago
> This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use That's not a natural consequence of the structure. If the top-down directives are issued by competent and financially accountable leadership, you get a nice efficiency pipeline.

AnonC 6y ago

> FB approaches internal code development with the same data driven rigor as their business decisions. How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature. I have a few anecdotes where, as a Facebook group administrator of some large (tens of thousands of members) and small groups (few thousand members), there were so many bugs in moderating the group with an added mega bug where the “Report issue to Facebook” (or something to that effect) would also throw an error. Those bugs lasted years (going into the current year) before I gave up and quit Facebook. Based on my experience over several years, I doubt there’s much rigor on many things. It may just be one of those marketing messages that have a little bit of truth and a lot of subterfuge.

anonymoushn 6y ago
In Facebook Ad Manager it is similarly hard to perform any task without having your day ruined.
10x-dev 6y ago
My point was the tools are there and are pretty advanced, compared to what I see available outside of the company. Whether an internal team uses those tools effectively depends entirely on them and their expertise.
vvwvvwvvw 6y ago
They sure have a lot of rigor in sabotaging ad blocking efforts. If anything their anti anti ablock works all the time.

chii 6y ago

> tie up all these loose ends without giving up your personal time. so this just means they've crammed more into your backlog than you can do. I would not give up any personal, unpaid time to do it, unless there's some promotion that you need which this is a demonstration/investment for.

10x-dev 6y ago
Seeing as I made the backlog, I have nobody but myself to blame :) I know what you mean though. Our deadlines are self-imposed, but the planning process is there to make sure that the effort remains high. I believe the managers are responsible for squeezing out as much juice as they can in a polite way (as opposed to other places where they were pretty direct about it). I suppose the combination of opportunity + compensation justifies the stress level and excitement at this point in my life. I've worked at other places where people primarily coasted with low pay but received good retirement benefits. It seemed to work for them.

hinkley 6y ago

I kinda miss working on teams where disabling the tests was plan B instead of plan G. The disabled tests should hopefully reduce code coverage. Also someone should be graphing out the volume and flux in disabled tests from week to week, to tap the brakes if it looks like we're rushing forward blindly instead of doing our due diligence.