3 ms·

I don't agree. At least, I don't see the problem the same way. Companies become more complex as they get bigger, and morally ambiguous as they get larger, and

I don't agree. At least, I don't see the problem the same way. Companies become more complex as they get bigger, and morally ambiguous as they get larger, and once a company reaches the point that it's making hundreds of decisions per day that affect other peoples' lives, it's impossible for it not to fuck a few up. Look at what happened to Google. It hasn't actually become "a villain", but they've made some very public mistakes (Real Names policy, dropping the ball on "Real Games") and some catastrophic private ones (implementing Welch-style stack-ranking in "calibration scores"). Google was designed by good people with the best intentions, and this stuff still happened. It seems hard not to have cataclysmic fuckups on a daily basis at a certain size. For a semi-related side note, I'm starting to agree with the Yegge hypothesis on codebase size: it's not that certain programming techniques (IDE vs. no IDE) or language properties (static vs. dynamically typed) fail "at scale" and others don't. All do. Largeness in a codebase is uniformly undesirable; it makes pretty much any programming environment, language choice, and development methodology go to hell. I've yet to see a programming environment where programmers (a) have to interact with a genuinely large codebase (and aren't working in a well-carved-out walled garden) and (b) enjoy their jobs. Once you have too much code, maintenance becomes the majority of the job, and per-programmer productivity falls to 10% speed (~15 LoC per day is commonly cited, although LoC is actually a horrible measure seeing as codebase size is a problem, not accomplishment) and morale goes into the toilet; programmers are unusual in the working world in that their happiness is positively correlated with how hard they are working (or, I should say, able to work). The only long-term solution is the Unix philosophy and small-program methodology, but MBA types seem to prefer huge, over-ambitious, all-or-nothing monolithic projects. So small-program Unix philosophy has been losing for a long time in favor of huge, object-oriented mudballs, even though the small-program way (each program does one thing and does it well; and if you need complexity, you build a system and give it the respect-- modularity, thought given to robustness and fault-tolerance-- that a system deserves) is an objectively much better way of doing things. Companies may have the same illness. Bigness may become terminally inefficient in the near future. To make large companies work well, you have to carve out "honor's colleges" (to get the best people) and walled gardens and sandboxes and research labs, but that effort is guaranteed to meet political resistance simply on account of the complexity of the company. Imagine if a large, boring technology company decided to reinvent itself and really get behind R&D and put 10% of its engineers into blue-sky R&D work. It would be great for the company and society in the long run, but the political fighting would be immense. You'd have a sudden airdrop of desirable work/jobs, you'd have powerful people pledging to support the change only if their proteges got jobs in this sexy new research division (so the positions of power would be allocated politically rather than on merit and leadership) and the conflicting demands and requirements coming out of big-company complexity (any time an initiative needs approval, the gate-keeper will ask for some payoff, and usually an inappropriate one that compromises the initiative) might end up miring it in a bog from the start. I'm not saying it couldn't work. It easily could. But the political problem would be harder than anything else. It wasn't always this way with companies-- a lot of things were achieved by large companies because only huge corporations could even attempt them-- but it might be becoming that way. With the rapidly increasing technical complexity of modern work, we might be reaching a point where for a company to hold $10 billion worth of value and employ 15,000 people is no longer effective or desirable. Old-style huge companies were a lot simpler, because the bulk of their people were doing the same grunt work. In a modern "knowledge economy" where people are doing different work, and in which the work of poorly-motivated or unskilled people is of negative value (rather than merely low positive value coming from the weakest performers in typical "commodity" labor) huge companies may just be unmanageable. It's not about becoming "the villain" or having to make "hard choices". It's just about complexity. It's about the fact large, powerful things pretty much always underperform relative to our expectations of them because complexity imposes so much drag, and it's hard to see this until one has a good sense of what unmanaged, undesirable complexity (that's 99% of all complexity) looks like and where it comes from. Huge organizations don't become "villainous" by intent. They become complex and inefficient and reach a point where the only thing anyone can agree on is growth-for-growth's-sake, which macroscopically makes it look like the firm is driven by sociopathy and hubris. But it's not that simple. There are a lot of well-intended, talented people in these megalithic companies with great ideas they'd love to implement. The problem is that the only thing that seems to come out of the company macroscopically (instead of being cancelled out by internal forces and drag) is the one thing everything in the company can agree on: more money and power and headcount and just flat-out size for the company, under the assumption that increasing the firm's "bigness" will improve the position (and compensation) of each person within it.

akkartik 14y ago
The biggest problem facing humanity today isn't the singularity or global warming. It is that we don't know how to grow successful things without making them suck. For want of a better name, I tend to call this gravity-like tendency 'the tragedy of the commons'. But I think I'm interpreting it more broadly than is conventional[1]. Some examples that aren't generally associated with the idea: a) The patent system. It has decayed because it's an externality to be dumped to, in the interest of individuals to overload and to game[2]. b) The cat signal[3]. We have a tendency to flatten nuanced subjects (apathy) into flat memes (here lolcats) and fight them with a single weapon (here blackouts[4]) that grows ineffective with use. Blackouts are an externality; people will eventually grow sick of them. Everyone knows this. So the game-theoretic optimum strategy is to exploit them as quickly as possible before they're milked dry. c) Code becomes an externality as the collaborators grow. Think of the last time you saw someone bolt an argument onto a function. The number of arguments isn't 'important'. It doesn't affect anyone's performance metrics. Everyone has a vision for the codebase, needs to mould it somehow to further his ambitions. The codebase becomes a mute vehicle for everyone's ambitions, and as such an externality to be dumped to. Bolting arguments onto functions is merely the smallest sin that can be committed[5]. There's elements of regulatory capture here[6]. I think the two together serve to explain the fall of Rome, the rise of bureaucracies and the decay of the reddit frontpage. But the problem is solvable. I think the solution will come from software. But not the kind of software we tend to write today. Something is deeply broken about how we write software, and I've been trying to tease out what it is. My current hypothesis is that the enemy is abstraction. We tend to prematurely freeze interfaces. Everytime you freeze an interface in your code you give up ownership over its future evolution. You can't take things out, so it's going to get polluted over time, get things bolted on. It immediately turns into an ugly step child of your otherwise beautiful code. And so we neglect interfaces, treat them as externalities, bolt features on, make simple interfaces ugly and complex. Interfaces are like walls separating jurisdictions. Neither jurisdiction cares about the wall. Every now and again someone says, "this interface is too complex; I'm going to simplify it". And creates a new simple interface. But they then commit the Great Error: they freeze their interface in its turn, and the cycle repeats with it. All we've done is add another layer of crap atop all our layers[7]. As we add layers on top the bloat compounds. I think if you truly looked at the tower of abstractions in your codebase today, and took out everything that didn't serve your use today, you'd end up with a codebase two or three orders of magnitude smaller. I've been exploring replacing backwards compatibility with unit tests. Imagine a world where code sharing took place but with no guarantees for the future. This version here makes certain guarantees, but if you upgrade or do a git pull, all bets are off. It might delete a function you rely heavily on. Or it may make it work differently one time in a thousand[8] and cause subtle, insidious bugs. The only way for you to guard against this is by having a comprehensive suite of unit tests. You have to be certain that if the tests pass the software is safe for public consumption. It's a very different discipline; the wall can now move, and both sides watch its evolution closely using the barbed wire of tests. But it could lead to far smaller codebases, and our lives as software developers would be less circumscribed. Every interface we have to support is a line drawn through the state space constraining our movement, the kinds of software we could write. It's a wall limiting our jurisdiction. Fewer interfaces imply fewer walls, greater freedom of movement across mental terrain, better software. And if our software is better, less hamstrung, more responsive, we may be able to automate more. We may be able to think about all the dysfunctions in the real world, constantly seek new externalities and create sensors to monitor their health.[9] --- Related (if you squint a little): http://www2.macleans.ca/2012/06/11/artisan-chocolate-and-social-revolution http://www2.macleans.ca/2012/06/11/artisan-chocolate-and-soc... [1] The traditional motifs (http://en.wikipedia.org/wiki/Tragedy_of_the_commons http://en.wikipedia.org/wiki/Tragedy_of_the_commons) are both more concrete and more libertarian. I don't think the tragedy can be solved by eliminating all commons. [2] http://akkartik.name/blog/2010-12-19-18-19-59-soc http://akkartik.name/blog/2010-12-19-18-19-59-soc [3] http://internetdefenseleague.org http://internetdefenseleague.org [4] Another example is our overuse of anti-biotics. [5] No judgement; I'm as guilty of this as anyone can be. [6] http://en.wikipedia.org/wiki/Regulatory_capture http://en.wikipedia.org/wiki/Regulatory_capture [7] Think of the xkcd on standards: http://xkcd.com/927 http://xkcd.com/927 [8] http://news.ycombinator.com/item?id=4295681 http://news.ycombinator.com/item?id=4295681 [9] But we have to guard against tying everything to a single metric. The last thing I want is a manager reading this and deciding to flatly limit the number of arguments to a function by diktat. The moment you create a metric you encourage people to stop exercising judgement and simply game the metric. Metrics to fight externalities can themselves become externalities.

akkartik 14y ago

The biggest problem facing humanity today isn't the singularity or global warming. It is that we don't know how to grow successful things without making them suck. For want of a better name, I tend to call this gravity-like tendency 'the tragedy of the commons'. But I think I'm interpreting it more broadly than is conventional[1]. Some examples that aren't generally associated with the idea: a) The patent system. It has decayed because it's an externality to be dumped to, in the interest of individuals to overload and to game[2]. b) The cat signal[3]. We have a tendency to flatten nuanced subjects (apathy) into flat memes (here lolcats) and fight them with a single weapon (here blackouts[4]) that grows ineffective with use. Blackouts are an externality; people will eventually grow sick of them. Everyone knows this. So the game-theoretic optimum strategy is to exploit them as quickly as possible before they're milked dry. c) Code becomes an externality as the collaborators grow. Think of the last time you saw someone bolt an argument onto a function. The number of arguments isn't 'important'. It doesn't affect anyone's performance metrics. Everyone has a vision for the codebase, needs to mould it somehow to further his ambitions. The codebase becomes a mute vehicle for everyone's ambitions, and as such an externality to be dumped to. Bolting arguments onto functions is merely the smallest sin that can be committed[5]. There's elements of regulatory capture here[6]. I think the two together serve to explain the fall of Rome, the rise of bureaucracies and the decay of the reddit frontpage. But the problem is solvable. I think the solution will come from software. But not the kind of software we tend to write today. Something is deeply broken about how we write software, and I've been trying to tease out what it is. My current hypothesis is that the enemy is abstraction. We tend to prematurely freeze interfaces. Everytime you freeze an interface in your code you give up ownership over its future evolution. You can't take things out, so it's going to get polluted over time, get things bolted on. It immediately turns into an ugly step child of your otherwise beautiful code. And so we neglect interfaces, treat them as externalities, bolt features on, make simple interfaces ugly and complex. Interfaces are like walls separating jurisdictions. Neither jurisdiction cares about the wall. Every now and again someone says, "this interface is too complex; I'm going to simplify it". And creates a new simple interface. But they then commit the Great Error: they freeze their interface in its turn, and the cycle repeats with it. All we've done is add another layer of crap atop all our layers[7]. As we add layers on top the bloat compounds. I think if you truly looked at the tower of abstractions in your codebase today, and took out everything that didn't serve your use today, you'd end up with a codebase two or three orders of magnitude smaller. I've been exploring replacing backwards compatibility with unit tests. Imagine a world where code sharing took place but with no guarantees for the future. This version here makes certain guarantees, but if you upgrade or do a git pull, all bets are off. It might delete a function you rely heavily on. Or it may make it work differently one time in a thousand[8] and cause subtle, insidious bugs. The only way for you to guard against this is by having a comprehensive suite of unit tests. You have to be certain that if the tests pass the software is safe for public consumption. It's a very different discipline; the wall can now move, and both sides watch its evolution closely using the barbed wire of tests. But it could lead to far smaller codebases, and our lives as software developers would be less circumscribed. Every interface we have to support is a line drawn through the state space constraining our movement, the kinds of software we could write. It's a wall limiting our jurisdiction. Fewer interfaces imply fewer walls, greater freedom of movement across mental terrain, better software. And if our software is better, less hamstrung, more responsive, we may be able to automate more. We may be able to think about all the dysfunctions in the real world, constantly seek new externalities and create sensors to monitor their health.[9] --- Related (if you squint a little): http://www2.macleans.ca/2012/06/11/artisan-chocolate-and-social-revolution http://www2.macleans.ca/2012/06/11/artisan-chocolate-and-soc... [1] The traditional motifs (http://en.wikipedia.org/wiki/Tragedy_of_the_commons http://en.wikipedia.org/wiki/Tragedy_of_the_commons) are both more concrete and more libertarian. I don't think the tragedy can be solved by eliminating all commons. [2] http://akkartik.name/blog/2010-12-19-18-19-59-soc http://akkartik.name/blog/2010-12-19-18-19-59-soc [3] http://internetdefenseleague.org http://internetdefenseleague.org [4] Another example is our overuse of anti-biotics. [5] No judgement; I'm as guilty of this as anyone can be. [6] http://en.wikipedia.org/wiki/Regulatory_capture http://en.wikipedia.org/wiki/Regulatory_capture [7] Think of the xkcd on standards: http://xkcd.com/927 http://xkcd.com/927 [8] http://news.ycombinator.com/item?id=4295681 http://news.ycombinator.com/item?id=4295681 [9] But we have to guard against tying everything to a single metric. The last thing I want is a manager reading this and deciding to flatly limit the number of arguments to a function by diktat. The moment you create a metric you encourage people to stop exercising judgement and simply game the metric. Metrics to fight externalities can themselves become externalities.