8 ms·
Rapid release at massive scale
- nimchimpsky 9y ago"engineer productivity remained constant for both Android and iOS, whether measured by lines of code pushed or the number of pushes." Both of those metrics are incredibly shit ways to measure productivity. I guess thats one explanation as to why the facebook app is 200mb+. They've been superproductive with all those lines of code. A better metric would rely on actual features or bugs, imo.
- Osmose 9y agoI think it's obvious that they're bad at a small scale (individually or for smallish teams, maybe < 30 engineers?), but I don't think they're _obviously_ bad for teams of 50+ engineers. Also, there is a difference between using them as metrics that you want to raise vs metrics you just don't want to drop or fluctuate wildly over time.
- nimchimpsky 9y agoI just think they are bad for anyone. Its not hard to measure actual useful things like features or bug. Why even bother measuring lines of code/number of commits.
- dikaiosune 9y ago> Its not hard to measure actual useful things like features or bug. Presumably your consulting business has really taken off now that you have such unique insight into a historically unempirical area of our field?
- richardwhiuk 9y agoMeasuring bugs and features leads to other equally horrendous problematic behaviors.
- ktRolster 9y agoIf you use "number of lines of code" as a metric for productivity, soon people will double-space their code. We went through that in the 90s.
- khazhou 9y agoWere there any other signs you had shitty programmers on your team?
- ktRolster 9y agoIt's not about shitty or good, it's about management. Whatever you reward, that's what people will give you. See also: https://users.cs.duke.edu/~ola/courses/cps108/code/java/harpoon/images/dilbert/bug.gif https://users.cs.duke.edu/~ola/courses/cps108/code/java/harp...
- Osmose 9y agoYou're arguing against an extreme, though. Sure, if LOC added (or even just changed, which is the saner metric) was linked directly to bonuses, it'd be ripe for abuse. But that completely ignores the more reasonable use in the article, as one (of several) metrics used to measure the impact of their release process.
- testvox 9y ago
- 0xbear 9y agoSome people at Google are actually quite proud of six digit _negative_ line counts they've contributed. And I think they have every right to be proud of that.
- Swizec 9y agoThat’s why you track changes made. Throughout my career imve found that number of changes comitted correlates pretty well with features delivered and bugs fixed. Also with feature and bug size. Yes there are edge cases when it takes a day of debugging to make a 1-line fix, but those are rare. Just like it’s very rare to deliver a useful new feature by changing a single line. Yes there are also features that are tracked as a real ticket and require a 1-line copy change and nothing else. Nobody thinks doing those makes you hella productive, it just needs to be done. As for padding lines and changes. That’s what code review is for.
- 0xbear 9y agoThat would be a bad metric for those "negative" folks as well. Their job is called "code cultivation", and they modify everything in Google's monorepo to make it suck less. Often this is done using automated tools, with a subsequent full run of affected tests and human code review. I've never seen anything like this anywhere else, and I probably won't for the rest of my career.
- 0xbear 9y agoAs to what one should track, one should track _results_ if you've managed to accomplish something useful with very little code, that is decidedly better than accomplishing the same with much more code and hundreds of check-ins. Simplicity is the ultimate sophistication.
- Swizec 9y ago> one should track _results_ One should, but companies often care more about effort than results. They can manage based on effort, they can't manage based on results. If you spend 2 days getting the same results as somebody else does in 5 days, guess what, they don't want you milling around those extra 3 days and bringing morale down. Gotta give you more work!
- khazhou 9y agoSome people, perhaps you as well, take offense at LOC metrics because they're (in theory) easy to game, via redundant code, whitespace, verbose vs concise languages, etc. And because good developers know that small code and systems mean fewer opportunities for lurking bugs and points of failure. BUT, if a team of developers is not padding their code, nor has shitty programmers who write page after page of nonsense, then in the end the proportion of functionality to LOC should be fairly stable, and thus LOC is a useful and meaningful proxy metric for the amount of new or changed functionality in a codebase.
- nimchimpsky 9y agoI'm not offended I just think its shit. Its not difficult to track "velocity" via story points,features or bugs and is at least a step in the right direction. LOC going down is often adding the most value.
- khazhou 9y agoSure, deleting unused or bad code is a good thing, or refactoring or rewriting something more tightly. But you can't implement a new feature just by deleting code. I've never been in a codebase that did more and more by getting smaller. People also take offense because a management weenie can misread the metric and penalize people who delete code, or write concise code, or solved a super tricky critical bug after a month of debugging with a one-line change. But I still claim, these all normalize in the end. If a developer on my team has committed only 30 lines in three months, sorry but I'm gonna be skeptical of their actual productivity.
- lomnakkus 9y ago> But you can't implement a new feature just by deleting code. No, but you can implement a "story", i.e. the removal of code is presumably a ticket somewhere in the system. (Or could become one.) > I've never been in a codebase that did more and more by getting smaller. Run on a smaller system? Run faster? (I'm not sure whether or not I agree with the parent's point, nor yours, just offering suggestions.)
- lloydde 9y agoI'd guess LoC changed is implicit in "lines of code pushed", which is a blunt measurement, but isn't the same as the traditional LOC metric and doesn't directly correlate to (rewarding) a large download.
- ahupp 9y ago> Both of those metrics are incredibly shit ways to measure productivity. LOC or commits are certainly bad ways of comparing two different people, teams, etc. But they are fine ways of measuring whether a specific event at a specific time changed productivity, assuming a decently large team etc.
- nebabyte 9y agoNot when they know that's what they're being measured on...
- Cthulhu_ 9y agoThat's not really relevant; if they knew they were being measured before and after the change, it would still show if the new system had an impact on productivity - even faked.
- ahupp 9y agoSure, nobody should setup incentives based on LOC. But in this case it's not about incentives, it just about seeing if something changed. Nobody writing code is going to be rewarded or penalized if this number changes.
- mahyarm 9y agoI think what they are trying to measure is how much is this build/release system effecting productivity of the engineers and changes their behavior. Not how productive the engineers are per say. If the build system is super slow and just breaks all the time, your going to see a drop in the number of commits landed per day. If git is completely down and commits per day drop per 0, then you have an outage ;)
- elgenie 9y agoLiterally the next sentence: """Similarly, the number of critical issues arising from mobile releases is almost constant regardless of the number of deployments, indicating that our code quality does not suffer as we continue to scale.""" This is 'productivity' in the sense of the release eng team attempting to measure the volume of code changes they're pushing out and whether changes they made to the push/release process impacted other teams' ability to iterate. Unless you have some insight as to why adopting a more continuous deployment process would change the ratio between LOC pushed or code-changing-diffs pushed to features-or-bugs specifically at Facebook such that they would not function as acceptable proxies, questioning the metric chosen seems hollow. It's certainly not 'productivity' in the sense of whether people on product teams are considered to have done a good job or are deserving of bonuses / promotions.
- coldtea 9y ago>Both of those metrics are incredibly shit ways to measure productivity. In theory yes. In practice, when you know your team and their average output and quality, no. >I guess thats one explanation as to why the facebook app is 200mb+. They've been superproductive with all those lines of code Or you know, most of it is assets, debugging symbols, stray frameworks and libs not stripped, etc. As is the case with almost any app beyond a few MB.
- hacker_9 9y agoBetter measure would be number of tests passing that prove business logic.
- rodorgas 9y agoObligatory Bill Gates quote: Measuring programming progress by lines of code is like measuring aircraft building progress by weight.
- dogruck 9y agoI like this quote because it's so accurate. LOC does provide some information. You'd be unhappy if either LOC or weight were close to zero. That said, one of my favorite metrics is number of deleted LOC.
- tehlike 9y agoBetter metric for a business must include revenue.
- latchkey 9y agoI've had a deployment cycle like this since I started using Google App Engine... 6 years ago.
- alangpierce 9y agoThe interesting thing about the article isn't that they're able to release continuously; there's nothing technically hard about deploying quickly. The interesting thing is that they're able to make the continuous release system work with thousands of engineers actively working in the same codebase without destroying quality. The three-tiered release system, monitoring alerts, feature flags, and good testing infrastructure seem to be what makes all of that possible.
- breeny592 9y agoExactly - releasing is the easiest part of the process. A lot of orgs don't have continuous deployment because of reasons such as: - they don't have a good enough automated testing suite (or at least don't trust it fully), and thus rely on "sign offs" to have people commit to saying it's quality - they don't measure in production properly (no real error alerts, no way to measure release success), and often deal with things in a "go or no-go" type way - they don't canary test. To me this one is critical - the only way to get real production use is to have real production users actually using the site/platform/app, just a sample of them, to see what could go wrong, especially with new features A lot of managers I've worked with are shocked whenever I pull out the "continuous deployment is easy. doing it well is hard" line.
- latchkey 9y agoYup, we do all that, except the 10k developers scale. Ha. I had the advantage of starting with fresh codebases and a small team. Obviously, adding this to an existing organization is much more difficult. When you setup your system correctly from the start, it also becomes a great hiring tool. Once you show developers the environment you work in, mouths drop and they almost beg to work for you. Doing it is hard, but not impossible. At this point there really is no excuse to not start things off like this. It is about 1-2 weeks of effort to setup a new project with all the right tooling on top of GCP thanks to the features they give you as part of their platform.
- ianamartin 9y agoI must be really out of the loop because I just don't see enough of what Facebook is doing to justify all of this garbage. There's a lot of lip service in the article about user experience, and I guess there are some changes here and there, but wtf is happening here? I definitely want and use CI/CD tools for my team's software, but what the fuck are you really doing when you are making this many changes per day? Call me an old fart, but if you are in a situation where you need to make that many changes per day, you are utterly fucked from almost every angle. Every aspect of this article sounds to me like people have no idea what they are trying to do, so they write code and push it, and it goes live. And everyone is very happy about this, for some insane reason. No offense to anyone, but this is not a reality I want to live in. And the article doesn't do much to defend the notion.
- aytekin 9y ago- They are running a lot of tests to see what works. - They are doing a lot of fixes. The number of bugs found is proportional to the number of users you have and number of changes you are making.
- flukus 9y ago"Just because you can doesn't mean you should", seems like they forgot about this principal. I think it's important that you can commit and push to prod quickly, shit happens and you need to be able to fix it quickly. But I agree with you, I'm a user not an alpha (or dev) tester.
- coldtea 9y agoThe parent makes a point. Across their whole product line (client apps, Hip Hop VMs, Flow, whatever), there might be millions of lines of code. But what exactly seems to actually change year over year on FB itself (server/client of the actual social website) that warrants so many commits?
- cromulent 9y agoI would suspect that the engagement level of the site changes. They have the audience to be able to run massive amounts of experimentation to see how people use the site and respond to advertising. To the average user, not much changes except for a feed of news.
- trjordan 9y agoOne of the undersold (imho) parts of this post is the system that allows tiering of releases with gates for their release. When most companies talk about CI/CD, they mean that master gets deployed to production, full stop. Rollbacks mean changing the code. In reality, when code hits master, there is ALWAYS a lag while it gets deployed, and it's worth having a system that holds that source of truth. Where release engineering gets interesting is how you handle the happy path vs. a breaking release. I like that Facebook separated out deploy from release. It means that you can roll the release out relatively slowly, checking metrics as you go. Bad metrics mean blocking the release, which means turning off the feature via feature flag. I think for the rest of the world, that would mean halting the release and notifying the developer. Disclosure: I work with smart people who spend lots of time thinking about this and writing blog posts like "Deploy != Release": https://blog.turbinelabs.io/deploy-not-equal-release-part-one-4724bc1e726b https://blog.turbinelabs.io/deploy-not-equal-release-part-on...
- zatkin 9y agoHow are they able to achieve slicing their userbase into thousands of pieces to do this gradual release?
- keypusher 9y agoOne way to accomplish this would be reverse proxy layer with sticky sessions that routs some percentage of traffic to X and some to Y.
- sweetdreamerit 9y agoA sort of A/B test, where what is tested is the implementation.
- mccv 9y agoThis is in fact exactly what we (https://www.turbinelabs.io https://www.turbinelabs.io) do. With enough proxy work you can tee some traffic to version Y and observe success rate/latency, then discard the response (or compare it with version X's response), at least for idempotent requests.
- deleted 9y ago[deleted]
- Gravityloss 9y agoIs their distributed database code also in this same repo and goes through the same process?
- oliverzheng 9y ago(Former FB eng.) No. Storage and backend services are on a different tier and release schedule entirely.
- xupybd 9y agoI can't be the only one immature enough to read that title as a double entendre.
- imaginenore 9y agoIt's not clear from the text how they deal with the severe production bugs. By the time the bug is found, the master branch is full of new code. So your bugfix has to deploy with all that new code? And with so many people checking in to the master branch, how is it not permanently broken? With 1000 devs pushing code, you're bound to have severe bugs daily.
- Too 9y agoIt's not like people push straight to master. There is most likely a pull request system or other form of review tool in between, with both code review and static analysis.
- fmavituna 9y agoIt's interesting that static or dynamic automated security testing don't exist in their process.
- czbond 9y agoI have to believe it is part of their pipeline. Even smaller companies do at least static analysis, if not much more.
- rdsubhas 9y agowhen both the delivery (pipelines) and the units going in them (container images with deployment descriptors) are automated, its really easy and straight-forward to plug-in a variety of automated checks (e.g. https://github.com/coreos/clair https://github.com/coreos/clair, organizational policies, governance, etc)
- ernsheong 9y agoHey Facebook, rapid release is great, but you broke my Messages, Notifications, Quick Help, and caret buttons on my https://www.facebook.com https://www.facebook.com navigation bar. It's been broken for a few hours now, clicking on it does absolutely nothing. Chrome 60 browser, macOS. Maybe you need to slow down.
- thomasjudge 9y agoI see this happen not infrequently. Clearly their values still weight "move fast & [don't worry too much when we] break things" over reliability. Which for a site of this nature is a somewhat defensible choice, but for those of us who come from enterprise software for example, or expect a reasonably stable user experience, it's occasionally between disconcerting & annoying
- tehlike 9y agoNope, just means that they didnt measure that particular metric. It gets better over time.
- ernsheong 9y agoWow, lots of Facebookers downvoting me :)
- kasperset 9y agoThe massive scale is getting massive at a massive rate.
- jonstewart 9y agoIt makes sense upon reflection, but something I thought interesting was how they run linters and static analysis tools in parallel with building. I've been used to build-and-test pipelines where these are done serially, because what's the point in building if the linter fails, and what's the point in static analysis if the build fails? But the point is, they can be done in parallel and that reduces latency of feedback to the developer.
- newscracker 9y agoNote: Everything in this comment is based on personal experience and observations as a Facebook user. They're all my opinions too. In my experience of using Facebook (primarily groups), it is a highly buggy platform and it's very hard to say that it behaves consistently or even that features are really ready before release - this, IMO, implies that both development and testing as well as rolling out the releases are messed up. This post talks about building a better "conveyor belt", so to speak, to release changes, but if the basic product is buggy and didn't get good attention in design/dev/test, no improvements in the "conveyor belt" can help make it awesome. Standard features that have existed for long may or may not work (how good are the regression tests then?). Posts and comments in groups may sometimes just disappear (thank goodness an admin activity log was added sometime in the last several months so we can stop wondering if an admin deleted anything). There's a feature in groups to mandate people wanting to join a group to answer some questions setup by the admins. Most people submit the answers but that never gets saved, and it's unknown what the trick is to get the answers to stay (this has been around for several months now?). New features aren't always announced. I see Facebook as a platform that's used to share ephemeral things. So this level of quality is probably just ok (though I don't believe it justifies the company's revenues and valuation). Since I do not conform to Facebook's ridiculous policy on using "real/authentic names", I don't even venture into contacting support if I see any issue, lest my presence be obliterated (yes, I try to keep away from Facebook, but do need it for some important awareness building because there's a large audience there). As a platform used by billions, Facebook still has a very long way to go in being reliable.