5 ms·

Moving Past the Scaling Myth

100 points by tmorton 10y ago

ktRolster 10y ago
Fred Brooks pointed out that with a small team of competent programmers, any organizational methodology will work. If you pay attention, the best Agile systems focus on improving the skills of developers instead of forcing people to follow the 'steps' or a formula.
- bsdpython 10y ago
  People over process every time. It's really hard to build a team of good people though so instead companies try to tweak their process. Process can matter e.g. a good team can be wrecked by a bad process but a bad team is bad no matter the process.
  - muratk 10y ago
    Depends on what you define by “bad” team. I'm in the Philippines, and there's lots of BPO companies here (call centers, dev shops, …). They have to deal with a high rate of ppl leaving, and train their employees from scratch. Still, those companies are successful and ppl want those relatively well paid jobs, even if they'll jump ship for any reason at all. In this kind of environment, process is everything.
nickpsecurity 10y ago
Hmmm. Maybe. What I've found is that IT often re-invents past designs and methods to solve modern problems. Matter of fact, the way the cloud providers scaled was to re-implement mainframes on x86 boxes. The next level is applying lessons from older research in modern ways to improve efficiency of various parts of HW and SW. If anything, quite a few people came up with timeless lessons that teach you how to do efficient, reliable computation. They keep getting reused. So, I don't think we can take it as far as author is suggesting where it's like classical vs quantum physics. Maybe at ASIC vs software level. Even those were partly joined with synthesis & coprocessing tools. I just don't see it as every high-level description I read about things is based on similar principles at each layer of the stack. Given similar constraints or goals, you would use similar strategy. There's certainly divergence or outliers but more repetition of patterns than anything. The only myths are that technology/fads X, Y, and Z should've been widespread adopted over the ones (or enhancements of them) that consistently worked. The author is seeing results of people building on stuff that came with assumptions that don't match new problem. Or people straight-up ignoring root cause in their solutions or methods. Common problems.
- deleted 10y ago
  [deleted]
wpietri 10y ago
This makes a lot of sense to me. At the very least, I think that most American organizations undergo a substantial cultural transition as they scale, so it makes sense we'd also need a process transition. The problem I see is that people aren't really willing to be honest about the cultural transition that happens, so we also can't be honest about the process transition. I think Agile-ish approaches work very well in startups, because the structure is pretty flat, and the goals are shared. But as companies grow, they tend to become what I think of business feudalism: hierarchical, control-oriented, territorial. For that, it makes sense you need different processes. And I think large company Agile is in effect Waterfall with a faster cadence, so you get that different process. But nobody will admit it. "We're doing Agile," they say, with too-bright eyes and gritted teeth. What I wonder is: what if instead of killing the peer culture and the human-centered process as we scaled, we kept them?
- michaelfeathers 10y ago
  At scale many organizations create substructure that contain strong peer culture and human-centered process, so it isn't lost, it's localized. I think that the big challenge is to recognize that hierarchical structure isn't good or bad, it's something that most natural systems do in response to scaling. If we understand it as a design parameter we can make it humane.
  - wpietri 10y ago
    The first part makes sense to me, although I don't think I've ever personally seen it happen. What percentage of companies would you say operate in the fashion you describe? When you talk about hierarchy as natural, I'm not sure we're talking about the same thing. I mean hierarchy as a system of power/control relationships, with each unit having an actor in charge and totally containing sub-units. E.g., the classic army chain of command. I can't think of anything natural that has that. Flocks and troops have leaders, but I think humanity is unique in its extension of social dominance to many-layered nested structures controlling millions of individuals. So I'd call it essentially artificial. I agree that we can apply a hierarchical analytical model to many natural systems, but I think there are plenty of other models we could apply, just ones that aren't as easy for current human brains. E.g., we could talk about fractal, self-similar, or recursive processes; those have no connotation of primate dominance. Or note the shift away from "great chain of being" analyses of nature toward ecological and non-teleological analytical models. When all you have is social primates, everything looks like a hierarchy. I would certainly like to believe that we can make hierarchies humane. But given that a fundamental characteristic is individual power over others that increases as you go up the chain, and given human fallibility, cognitive limits, and human nature, it seems to me they have to become essentially inhumane at some size. And in practice I certainly see a correlation between hierarchy size and inhumanity. Correlation isn't proof, of possibility, of course. Just natural tendency. Are there large hierarchies you think violate that correlation, where they have gotten more humane with scale?
exabrial 10y ago
I really wish the scaling myth would die. Everyone pretends they have a scalability problem because it's sexy and as such, travesties have been committed (you know, in case we scale). How about instead designing software where the architecture and code is clear, consistent, and readable by the next guy?
- gaius 10y ago
  My philosophy: plan to scale 10x. If you ever approach it - which in most cases is unlikely - you can justify completely rearchitecting, but only to scale another 10x. If not, well your design is probably not too top-heavy anyway. It's people who want a single design to take them from 1x to 1000000x who get into trouble.
  - hobofan 10y ago
    I disagree with that statement. Of course a design that works at 1x doesn't always work a 10000x, but why shouldn't a design that works a 10000x not also work at 1x? At least for technical scalability the answer almost always lies in overhead. Yes, while the system scales perfectly linearly at O(n), the hidden constant is often neglected, which means at small values of n the solution becomes unfeasible. So while Cassandra might for one of my use cases be the perfect candidate due to its feature set and linear scalability it becomes unfeasible at a small scale because I have to run at least 3 nodes with high overhead (because Java, sigh) + an additional 3 nodes with Zookeeper with high overhead (because Exhibitor + Curator, all in Java). /rant, I don't like bashing on Java but overhead is definitely one of its weak spots, and here is where it comes into play.
    - gaius 10y ago
      Scaling down is always easier than scaling up, because you are going back over known territory. Scaling up is hard because it takes you into unknowns. Rumsfeld's Theory applies.
      - hobofan 10y ago
cateye 10y ago
One of the underlying problems is that there isn't a formula (= process) for innovation. Most software development tries to create a competitive advantage by creating something unique that isn't easy to replicate. So, every process around being able to deliver that by just applying that process creates a false hope. Or it is not meant for this purpose but the "buyers" aren't aware of it and have other expectations.
DanielBMarkham 10y ago
As somebody who loves startups and small teams, and who has a job working with both small and big organizations, I have been living this scaling thing for a long time. I'd like to give you a simplistic answer, like "All you need, kid, is a small team! For anything!" Slogans like that are true, but yet they are terribly misleading, because 1) many organizations are already terribly overstaffed, and 2) it doesn't really help to tell teams "What do I do right now?" So here's as simple as I can make it: Good organizations will do whatever is necessary to make things that people want, even if that means instead of programming, the programmers sit on the phone and do some manual chore for people as they call. Before you code anything, you have to be hip-to-hip with some real person that you're providing value to. But as soon as you have those five folks sitting on the phones doing something useful? You gotta immediately automate. Everything. This means you're going to have all freaking kinds of problems as you move from helping ten people a day to helping a million. You have to automate solutions, access, coordination, resource allocation, failovers, and so on -- the list is almost endless (but not quite) As they grow, poor organizations take a scaling problem and assign it to a person. Somebody does it once, then they're stuck with it for life. Good organizations continue to do it manually, then immediately automate. Somebody does it once, then the team immediately starts "plugging all the holes" and fixing the edge cases so that nobody ever has to manually be responsible for that again. Growing "sloppy" means you end up helping those million people a day -- but you have hundreds of people on staff. Meetings take time. Coordination takes time. There are a ton of ways to screw up. People tend to blame one another. Growing good means you can be like WhatsApp, servicing a billion users with a tiny team of folks. If you're already an existing BigCorp and have been around for a while -- odds are that you are living with this sloppy growth pattern. That means you need to start, right now, with identifying all the feedback loops, like release management, and automating them, such as putting in a CI/CD pipeline. Not only that, but there are scores of things just like that. You have a lot of work to do. It might be easier just to start over. In fact, in most cases the wisest thing to do is start over. Now picture this: you're an Agile team at BigCorp and you've got the fever. Woohoo! Let's release often, make things people want, and help make the world a better place. But looking around, all you see is ten thousand other developers in a huge, creaky machine that's leaking like a sieve. You go to a conference with a thousand other BigCorps, just like yours. Are you going to want to hear about how it's better just to trash things and start over, about the 40 things you need to have automated right now but don't, or how to make your section of 150 programmers work together; how to "scale agile"? Scaling Agile is an issue because the market says it is an issue.
tango12 10y ago
Reg. the architecture question: People generally start carving out microservices once they hit some scale. What about starting off with microservices (because tools like k8s make the otherwise insane management overhead more tractable)? Could that be a possible solution? The problem of scaling state, databases need to handle anyway. Ideally the same database just keeps on working as you go up. Or atleast the protocol is the same, and you switch out the implementation from a single instance to a clustered version.

100 points by tmorton 10y ago

Fred Brooks pointed out that with a small team of competent programmers, any organizational methodology will work. If you pay attention, the best Agile systems focus on improving the skills of developers instead of forcing people to follow the 'steps' or a formula.

bsdpython 10y ago
People over process every time. It's really hard to build a team of good people though so instead companies try to tweak their process. Process can matter e.g. a good team can be wrecked by a bad process but a bad team is bad no matter the process.
- muratk 10y ago
  Depends on what you define by “bad” team. I'm in the Philippines, and there's lots of BPO companies here (call centers, dev shops, …). They have to deal with a high rate of ppl leaving, and train their employees from scratch. Still, those companies are successful and ppl want those relatively well paid jobs, even if they'll jump ship for any reason at all. In this kind of environment, process is everything.

nickpsecurity 10y ago

Hmmm. Maybe. What I've found is that IT often re-invents past designs and methods to solve modern problems. Matter of fact, the way the cloud providers scaled was to re-implement mainframes on x86 boxes. The next level is applying lessons from older research in modern ways to improve efficiency of various parts of HW and SW. If anything, quite a few people came up with timeless lessons that teach you how to do efficient, reliable computation. They keep getting reused. So, I don't think we can take it as far as author is suggesting where it's like classical vs quantum physics. Maybe at ASIC vs software level. Even those were partly joined with synthesis & coprocessing tools. I just don't see it as every high-level description I read about things is based on similar principles at each layer of the stack. Given similar constraints or goals, you would use similar strategy. There's certainly divergence or outliers but more repetition of patterns than anything. The only myths are that technology/fads X, Y, and Z should've been widespread adopted over the ones (or enhancements of them) that consistently worked. The author is seeing results of people building on stuff that came with assumptions that don't match new problem. Or people straight-up ignoring root cause in their solutions or methods. Common problems.

deleted 10y ago
[deleted]

wpietri 10y ago

This makes a lot of sense to me. At the very least, I think that most American organizations undergo a substantial cultural transition as they scale, so it makes sense we'd also need a process transition. The problem I see is that people aren't really willing to be honest about the cultural transition that happens, so we also can't be honest about the process transition. I think Agile-ish approaches work very well in startups, because the structure is pretty flat, and the goals are shared. But as companies grow, they tend to become what I think of business feudalism: hierarchical, control-oriented, territorial. For that, it makes sense you need different processes. And I think large company Agile is in effect Waterfall with a faster cadence, so you get that different process. But nobody will admit it. "We're doing Agile," they say, with too-bright eyes and gritted teeth. What I wonder is: what if instead of killing the peer culture and the human-centered process as we scaled, we kept them?

michaelfeathers 10y ago
At scale many organizations create substructure that contain strong peer culture and human-centered process, so it isn't lost, it's localized. I think that the big challenge is to recognize that hierarchical structure isn't good or bad, it's something that most natural systems do in response to scaling. If we understand it as a design parameter we can make it humane.
- wpietri 10y ago
  The first part makes sense to me, although I don't think I've ever personally seen it happen. What percentage of companies would you say operate in the fashion you describe? When you talk about hierarchy as natural, I'm not sure we're talking about the same thing. I mean hierarchy as a system of power/control relationships, with each unit having an actor in charge and totally containing sub-units. E.g., the classic army chain of command. I can't think of anything natural that has that. Flocks and troops have leaders, but I think humanity is unique in its extension of social dominance to many-layered nested structures controlling millions of individuals. So I'd call it essentially artificial. I agree that we can apply a hierarchical analytical model to many natural systems, but I think there are plenty of other models we could apply, just ones that aren't as easy for current human brains. E.g., we could talk about fractal, self-similar, or recursive processes; those have no connotation of primate dominance. Or note the shift away from "great chain of being" analyses of nature toward ecological and non-teleological analytical models. When all you have is social primates, everything looks like a hierarchy. I would certainly like to believe that we can make hierarchies humane. But given that a fundamental characteristic is individual power over others that increases as you go up the chain, and given human fallibility, cognitive limits, and human nature, it seems to me they have to become essentially inhumane at some size. And in practice I certainly see a correlation between hierarchy size and inhumanity. Correlation isn't proof, of possibility, of course. Just natural tendency. Are there large hierarchies you think violate that correlation, where they have gotten more humane with scale?

exabrial 10y ago

I really wish the scaling myth would die. Everyone pretends they have a scalability problem because it's sexy and as such, travesties have been committed (you know, in case we scale). How about instead designing software where the architecture and code is clear, consistent, and readable by the next guy?

gaius 10y ago
My philosophy: plan to scale 10x. If you ever approach it - which in most cases is unlikely - you can justify completely rearchitecting, but only to scale another 10x. If not, well your design is probably not too top-heavy anyway. It's people who want a single design to take them from 1x to 1000000x who get into trouble.
- hobofan 10y ago
  I disagree with that statement. Of course a design that works at 1x doesn't always work a 10000x, but why shouldn't a design that works a 10000x not also work at 1x? At least for technical scalability the answer almost always lies in overhead. Yes, while the system scales perfectly linearly at O(n), the hidden constant is often neglected, which means at small values of n the solution becomes unfeasible. So while Cassandra might for one of my use cases be the perfect candidate due to its feature set and linear scalability it becomes unfeasible at a small scale because I have to run at least 3 nodes with high overhead (because Java, sigh) + an additional 3 nodes with Zookeeper with high overhead (because Exhibitor + Curator, all in Java). /rant, I don't like bashing on Java but overhead is definitely one of its weak spots, and here is where it comes into play.
  - gaius 10y ago
    Scaling down is always easier than scaling up, because you are going back over known territory. Scaling up is hard because it takes you into unknowns. Rumsfeld's Theory applies.
    - hobofan 10y ago

cateye 10y ago

One of the underlying problems is that there isn't a formula (= process) for innovation. Most software development tries to create a competitive advantage by creating something unique that isn't easy to replicate. So, every process around being able to deliver that by just applying that process creates a false hope. Or it is not meant for this purpose but the "buyers" aren't aware of it and have other expectations.

DanielBMarkham 10y ago

As somebody who loves startups and small teams, and who has a job working with both small and big organizations, I have been living this scaling thing for a long time. I'd like to give you a simplistic answer, like "All you need, kid, is a small team! For anything!" Slogans like that are true, but yet they are terribly misleading, because 1) many organizations are already terribly overstaffed, and 2) it doesn't really help to tell teams "What do I do right now?" So here's as simple as I can make it: Good organizations will do whatever is necessary to make things that people want, even if that means instead of programming, the programmers sit on the phone and do some manual chore for people as they call. Before you code anything, you have to be hip-to-hip with some real person that you're providing value to. But as soon as you have those five folks sitting on the phones doing something useful? You gotta immediately automate. Everything. This means you're going to have all freaking kinds of problems as you move from helping ten people a day to helping a million. You have to automate solutions, access, coordination, resource allocation, failovers, and so on -- the list is almost endless (but not quite) As they grow, poor organizations take a scaling problem and assign it to a person. Somebody does it once, then they're stuck with it for life. Good organizations continue to do it manually, then immediately automate. Somebody does it once, then the team immediately starts "plugging all the holes" and fixing the edge cases so that nobody ever has to manually be responsible for that again. Growing "sloppy" means you end up helping those million people a day -- but you have hundreds of people on staff. Meetings take time. Coordination takes time. There are a ton of ways to screw up. People tend to blame one another. Growing good means you can be like WhatsApp, servicing a billion users with a tiny team of folks. If you're already an existing BigCorp and have been around for a while -- odds are that you are living with this sloppy growth pattern. That means you need to start, right now, with identifying all the feedback loops, like release management, and automating them, such as putting in a CI/CD pipeline. Not only that, but there are scores of things just like that. You have a lot of work to do. It might be easier just to start over. In fact, in most cases the wisest thing to do is start over. Now picture this: you're an Agile team at BigCorp and you've got the fever. Woohoo! Let's release often, make things people want, and help make the world a better place. But looking around, all you see is ten thousand other developers in a huge, creaky machine that's leaking like a sieve. You go to a conference with a thousand other BigCorps, just like yours. Are you going to want to hear about how it's better just to trash things and start over, about the 40 things you need to have automated right now but don't, or how to make your section of 150 programmers work together; how to "scale agile"? Scaling Agile is an issue because the market says it is an issue.

tango12 10y ago

Reg. the architecture question: People generally start carving out microservices once they hit some scale. What about starting off with microservices (because tools like k8s make the otherwise insane management overhead more tractable)? Could that be a possible solution? The problem of scaling state, databases need to handle anyway. Ideally the same database just keeps on working as you go up. Or atleast the protocol is the same, and you switch out the implementation from a single instance to a clustered version.