21 ms·
> It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. Eventually people lose interest. It's only human. But The Sy
by jacques_chester 14y ago
> It's an awesome concept; no-one can justifiably be bored with their projects if they chose them.
Eventually people lose interest. It's only human. But The System has been written, it is in production, and it has accrued a healthy stock of user data.
One day, The System breaks. "Only" tens of millions of users -- less than a percent of all Facebook users -- rely on the The System. But they rely on it utterly.
Who will fix The System?
Of the five engineers who wrote it:
* Jack and Mandeep left to launch myfornaxisnatrr.com
* Wei Li has moved to a different group
* John doesn't want to touch it, he only came on board to help Jack
* Michael was an intern but has since taken a job with Google.
Uh oh! There's nobody around to voluntarily fix an existing system. That's boring, and there are no incentives for fixing bugs in obscure features because only launching successful new features gains visibility from higher-ups.
Guess we'll need some mean old managers to round up a posse.
If they care enough.
Meanwhile, The System has acquired millions of users, cost millions of dollars to develop and operate and will now abruptly cause tens of millions of customers to become incredibly frustrated. And at no point has anybody stopped and asked:
"Was this the Right System to build?"
- jrockway 14y agoThe world doesn't work like this. In addition to doing-whatever-they-want-24-7, people are also trying to get promoted. That means there is always an incentive to do some task that nobody else will do. This kind of causes talent to even out between high-profile new projects and not-as-fun legacy projects.
- jacques_chester 14y agoIt doesn't work like this in places where shit being broken is something management cares about. In places where the goal is to make engineers happy and to impress other engineers, there are incentives for behaviour that are perverse from the customer's point of view.
- angstrom 14y ago> That's boring, and there are no incentives for fixing bugs in obscure features because only launching successful new features gains visibility from higher-ups. Companies really should find a way to incentivize this. Rewrites are about as silly. "Write the next generation xyz." Translates to "No one here has any idea how xyz was written, so it's being rewritten instead of modified. We look forward to rediscovering the same problems we did the first time."
- jacques_chester 14y ago> Companies really should find a way to incentivize this. It can't be done from the top. You need to start with engineers who care about quality over the long term. It has to come from within each engineer. When Zuckerberg talked about younger programmers being "better", he probably meant "more like me". But old farts are just young farts with expensively-acquired scar tissue. Most of our most treasured software development lore comes from the lesson that "move fast, break things" just devolves into "fuck, everything is broken, fix it fast".
- josteink 14y ago"Write the next generation xyz." Translates to "No one here has any idea how xyz was written, so it's being rewritten instead of modified. Sometimes, when software has been around for 10+ years, stuff does get too old. It happens. For real. I know people seem to forget that here with startups having been around for less than a year being "old players" and all that, but common. Software may have been written with a bunch of presumptions which was valid when the project was initially started, but no longer are that. Maybe it was written on top of a platform which is no longer as productive as it was when the project was initiated. Nothing kills enthusiasm and productivity like working on a platform no longer deemed modern. And then you have technical limitations and pre-conditions. Once, servers was expensive and you wanted your solution to distribute it's load to clients so that you could save on servers. Once servers was the only machines powerful enough to complete time-critical tasks within reasonable time-frames, so you architected the software accordingly. Now, we suddenly have HTML5 and webworkers and shit, so you probably want your solution to be a web-solution with a distributed computation model. I.e. full-circle. Full rewrite. Twice. What's next, I don't know. Sometimes a rewrite is just right. Saying that every rewrite is based in incompetence is simplistic.
- michaelmartin 14y agoI get what you're saying about the likelihood of developers having moved on to different projects/companies, but I don't think that issue is specific to this approach in any way. Good software should be written on the assumption that someone else is going to be editing your code at a later date anyway. If it's highly readable, then being unable to grab the original developer is less of an issue. This is one of the reasons for the peer code reviews during the original development. I doubt it's really that hard to find developers to work on issues like this. I've always thought the idea that their work impacts millions of people, even though that is a tiny percentage of the total userbase like you said, was one of the big attractions of working at Facebook. If I knew of an issue impacting that many people, I'd be delighted to have solved it.
- mkjones 14y agoI actually haven't seen this much as an issue at Facebook (I've been an engineer there for a few years). There are some (usually very smart) people who care a lot about systems working reliably, and almost always seem to be a few who are willing to jump on issues like this. One of the advantages of the "bootcamp" approach outlined in the OP is that people feel empowered to jump from product to product, so if something you care about breaks for a small set of users and you wish to fix it, you can do that. As a few others have pointed out, it doesn't hurt to mention this at review time, and people are often publicly recognized and thanked for these "thankless" efforts.