15 ms·

Why Are Enterprises So Slow?

290 points by okket 8y ago

MindfulMonk 8y ago
"Resistance is futile. Dev and Ops must be separate because that’s what we wrote down years ago." Really well written piece that clearly explains the reality of "Enterprise DevOps".
- exabrial 8y ago
  Yep, literally written down: https://en.m.wikipedia.org/wiki/Sarbanes–Oxley_Act https://en.m.wikipedia.org/wiki/Sarbanes–Oxley_Act If a corporate lawyer ever ambushes you with this, the best thing to do is ask instead how you can make your current process auditable, rather than tearing the org apart. If that doesn't work, start looking for a more innovative place to work!
kenhwang 8y ago
I'd say it's mostly regulation, risk-aversion, and company size. When you're small, it's easy to avoid regulatory attention and whatever slap-on-the-wrist fine you may get isn't a lot of money. The change and risk aversion really just comes from keeping the business running. When things work, try not to break them. Old large enterprises tend to have a lot of business critical things that "just work" and have been for a long time. Younger/smaller companies tend to have fewer battle scarred systems. But once they get scarred, they typically add a process or policy to prevent it from happening again. After enough of them, congrats, you're enterprise speed.
- rossdavidh 8y ago
  All true, but I would add: there's a lot less downside to change, when nothing much is working yet. Much of enterprise slowness is irrational, but much is not. Big systems are harder to predict consequences for, and big old systems have a lot that already works.
  - kenhwang 8y ago
    Add in the tendency to consolidate and standardize technologies as the company grows for maintainability just means even little changes quickly ripple through the system. It's like microservices, but with systems of people and businesses.
- LoSboccacc 8y ago
  and SLAs. once you get big enough your client contracts usually have penalties for failures of any kind. that puts the pressure on getting it right on the first try or at least within contract terms, instead of working toward the goal incrementally. partners that allow for some agile, iterative back and forth becomes rarer as they increase in size, especially since often their own system lack the flexibility for continuous integration or safely rolling back changes/data. that's why you get requirements, specifications, architects and solution managers in the enterprises. people love to mock them for being dysfunctional and some truly are, but once contracts get into the millions and lawyer get involved mistakes are no longer "learning opportunities".
programminggeek 8y ago
They are slow because they are complex as proven by the fact that it's complex to even explain why they are slow.
romed 8y ago
Eh. Article is a bit short on facts. Starting with the first bullet, the only place I've ever worked that rolled out OS updates continuously with a maximum 15-day latency was also an enormous global enterprise with tens of thousands of employees and plenty of regulatory red tape. Now I work at a much smaller company that thinks of itself as "scrappy" but is utterly paralyzed by OS updates in their production fleet because they don't have the tooling and processes around it. It is the tooling and the process that gives you the ability to move quickly!
- ocdtrekkie 8y ago
  A huge upside of Windows 10's six month rolling releases is that they initially broke a lot of badly written things, and software and processes have had to improve to handle them better. The end result two years later is OS upgrades rarely break much, and regular updates never do. Enterprises can be unstuck but it can be painful. I think enterprises are still shifting from the old approach where you make it work and then it's fine, to realizing that in an internet-connected world, you have to keep up at all times. A few big liability cases would do wonders to push this shift along.
  - bigger_cheese 8y ago
    I hope what you are saying about Win 10 is true. My organisation is in process of transitioning from Windows 7 to Windows 10 and things are breaking all over the place. Almost everytime we change versions of software things break when we went moved to office 365 there was breakage (Stuff I would have thought for sure would have been well tested - PST files in outlook for example broke). Same thing when we went from Internet Explorer 8 to 11 our internal intranet didn't work.
    - ocdtrekkie 8y ago
      Nearly everything that works in Windows 7 works in Windows 10, if you poke enough holes in it. The biggest exception to that is really just drivers, because hardware manufacturers have no incentive to update them past a certain point. There was definitely software that had to be updated to the latest versions to work, but that's where we needed to get to anyways. I work in an environment that includes custom-written apps from the Windows XP era, and usually the absolute worst we hit was "hey, this device doesn't work on Windows 10, we have to buy a newer one". Not a big deal when it's a $6 USB-to-serial adapter, but people will look at you like you're crazy when you say the network is incompatible with an HVAC system. (Solution in that case, just take it off the network, you're better off.) But believe it or not, you can still sync Windows Mobile 2003 devices with Windows 10 PCs, Microsoft's backwards compatibility is still insanely solid in most cases.
jeffmcmahan 8y ago
"If it doesn’t go well he’s out of a job and his boss might go to prison" Yeah, I don't think that happens much.
- bboreham 8y ago
  That’s the point. It doesn’t happen very often because people took care not to sign the document that would put them in prison. Your point may be that people should go to prison for presiding over bad things that weren’t actually against the law. That path sounds tricky.
Forge36 8y ago
I maintain code written by my company's founder from 1998. It hadn't seen many updates to maintain it. It saw a LOT of enhancements. I'm currently unwinding those changes into a full library of support features. It's also not my primary focus. Many new features to fulfill customer needs is a common reason. Many customers, with new requests, issues reported. It's slow because if I'm not methodical I cause problems for people who don't want this next enhancements. It's slow, because the current emergency takes precedence, and taking to long on fixes means a meeting on why something is behind. Business priority is not "deliver fast" it's "keep running, don't attract attention".
jac_no_k 8y ago
Spot on. [apologies for low effort post, but I have nothing more to add.]
- sephware 8y ago
  That's what the upvote button is for.
tzury 8y ago
As Chris Dixon put is: "A friend of mine got a job at a big company and was shocked to see his colleagues worked just a few productive hours a day. They didn’t seem to care about their work or have relevant expertise. My friend said: “Wow, this company is going under.” Then the company released its quarterly reports and profits rose to an all-time high. The momentum of the company’s brand and relationships was sufficient to propel it forward." http://cdixon.org/2012/05/19/the-default-state-of-a-startup-is-failure/ http://cdixon.org/2012/05/19/the-default-state-of-a-startup-...
- coffeemug 8y ago
  It isn't just relationships and momentum. XKCD had an interesting what if -- what would happen if a bowling ball was blown up to the size of Earth?[1] Turns out at that scale metal behaves like a liquid. Work at big companies is kind of like that. Once a company gets big enough, any given person doesn't really know anything. Seemingly no one is doing anything productive -- it's meetings all the way down. But the software still gets shipped. It doesn't happen the same way it happens at a startup where someone sits down and codes a feature. It happens in a totally different way, where one hundred engineers go to meetings all day and talk to QA and analytics and bizdev, and maybe commit one line of broken code a week. It takes way more people, and the production process is totally different, and it takes way longer, but stuff still happens (often quite successfully). [1] https://what-if.xkcd.com/46/ https://what-if.xkcd.com/46/
  - cpeterso 8y ago
    I read (circa 2000) that IBM's product development processes would need nine months to ship a product that was just an empty box.
    - blihp 8y ago
      That was probably referencing the story of the origin of the IBM PC. This anecdote was given as the reason they got the green light to develop the PC in Boca Raton independent of the rest of IBM.
nikanj 8y ago
Big companies have something to lose. A brand, clients, revenue and profits. People don’t want to risk those. A starup has nothing to lose. The site is janky for two hours? The only users are friends, and the outage might actually bring more visibilty if you make a good write-up. Incidentally, this ties together with “why do young people found more startups than senior executives”
debacle 8y ago
It's people. Enterprises are made out of people.
daniel_iversen 8y ago
Great article. There’s also other good reasons why enterprises move slowly ( cultural and other organisational structures and priorities that conflicts with making the same kinds of decisions a startup might make). A good book is The Innovators Dilemma: https://en.m.wikipedia.org/wiki/The_Innovator%27s_Dilemma https://en.m.wikipedia.org/wiki/The_Innovator%27s_Dilemma
MK_Dev 8y ago
The article is pretty spot on. One thing the author has left out is the fact that all of the problems are magnified 100x when the people involved are incompetent, which, unfortunately, happens a lot in enterprises (one can also argue that slow speed contributes to job security under normal circumstances).
kochikame 8y ago
No one knows anything in big companies Lots of smart, talented and energetic people are expending most of their power just figuring out who to talk to about what or what the hell that out-of-the-blue email from corporate finance means for them. Just figuring out how to get a thing done can be exhausting.
- carlivar 8y ago
  So, Dunbar's Number
- Scene_Cast2 8y ago
  Yep, can attest to this. One of the reasons why senior people are extremely productive & valuable is that they seem to know the landscape more than others, and therefore know who to talk to & where to look. Half-jokingly, as soon as an "individual contributor" knows enough to be productive, they're promoted to management.
  - csydas 8y ago
    WRT the half-joke, this is one of those truisms that probably has a lot of basis in reality, but there needs to be a corollary about the choice of the individual as well. I was heavily invested in our public facing technical side, working a lot with our on-boarding, teaching newbies to be really self-sufficient, working through the technical issues with our clients directly, and then I was asked to take on a Management position. I liked the pay raise, but I made it clear exactly what I wanted from the position, and though we butt heads on it still, my boss respects the fact that I still am heavily invested with our technical team while doing the Managerial work. We had to revisit the job definition a few times to find a description that matched what I was (am) doing and also made HR happy, but we did it. The point is that in some cases at least, a promotion to management doesn't have to be a death knell for "individual contributors". It depends on the company's willingness to accommodate the wants of the individual, and the individual's own desire to continue contributing. I know too many people who did get the Management golden ticket and relished it for just being an excuse to boss people around and make more dashboards in our CRM.
gtycomb 8y ago
Scanning over responses so far I think a relevant aspect is not underlined strongly enough -- the lack of expertise in some of the key positions in companies "to move things". The status quo prevails. Play it safe, don't shake the boat, bring in Gartner or some such consultant on strategy, and hum along. In the end, outsource a project with internal fanfare (bringing relief to people concerned). This happens all too often. This is how life goes, and as some here pointed out already, even if large inefficiencies are unaddressed a company can be profitable for significant time because of an established market position, brand, extensive regulations in place to protect their interests, some little known high profit products, etc. However give it time, and we see them fade away when upstarts take the risks, have the vision, shake up their market one piece at a time. And the cycle repeats.
berti 8y ago
Risk-aversion, cost of relatively small changes can look large, and the tendency to "design by committee". You can send an email out to some colleagues throughout other units of the business and get inundated with reasons why it won't work, and edge-cases you need to handle, whereas if you'd just implemented it, they'd likely work-around or improve their processes and everyone would have been better off.
tokyodude 8y ago
For me the #1 reason why they are slow is because everything requires consensus or maybe another way of putting it it's hard for people to make a decision on their own. Let's say I want to change some part of the UI of some app. If I'm by myself I just change it. If I'm on a team first I have to convince whoever has the power to decide to change it to allow it to be changed, that might even be a whole committee. Then several people will be involved, likely a UX designer, possible a separate UI designer, all of which will have to discuss the feature and all of which also have other responsibilities. The larger the company the worse this usually gets. Maybe the change has to be run by the security team to make sure the new UI is not a security issue. Maybe it has to run by the privacy team to make sure it's not going to leak data. Maybe it has to be run by an inclusivity team to make sure it's not offensive in some culture I'm not familiar with. Maybe it has to be run by a localization team to make sure it will work in every language. All of those things add time. Maybe they are all required and good, especially at a large company with a global marketplace but they certainly add to the time taken to do something that a tiny team wouldn't do.
- killjoywashere 8y ago
  A lot of answers to these questions, and the questions themselves, come from further down the food chain. The flip side of the coin is that no start-up is moving a billion tons of rice, or can park an army off a foreign shore in a week, or lay 10,000 tons of steel rail across a continent every day, or delivery a million packages a day, or make 10,000 processors a day, each with the complexity of the entire European system of roads. So speed is relative to the size of the task at hand. Large enterprise can move big things so fast it literally takes the air out of the room.
  - monkeydreams 8y ago
    > So speed is relative to the size of the task at hand. I think of organisational decision making like organic decision making - that the speed of the organisation's reactions are dependent upon its frame of reference. The frame of reference here is the time between sensing and reacting. The eyes and ears of small businesses are very close to the brains, and so decisions can be made rapidly by the small number of people who have capability to marshal resources into a response. Enterprises have layers of processing between the eyes and brains, along with many more sensors than a small business, meaning that questions take longer to be settled and decisions made. However no small business can wield the same power as a large organisation when the army of deciders all focus on a single outcome.
eksemplar 8y ago
I work in a muniplacity of 10.000+ employees, and network with other managers from 97 other muniplacities as well as the government, and I think most of those points are wrong. At least in enterprise that aren’t IT companies. There aren’t really regulators the way it’s described, and you wouldn’t worry about them the way it’s described either. Unless you made changes without planning. What you would consider when upgrading your OS, for instance, is how you teach the new OS to the 5000 employees who can’t tell the difference between and iPhone and an android tablet. Ownership is easy in enterprise, we have organized everything into departments and IT operations is owned by IT. It’s audited that way, and things are changed according to the guidelines. You’d literally never have an audit come in and point any of the things suggested out, unless your enterprise isn’t run like an enterprise. The first step in the manual for upgrading, is to make sure you stay within the law and within company policy, and you’d never run a project all the way through without involving people who know regulations. I don’t think enterprise is risk averse either, but I do think you need to be able to prove the benefits of a change, to be allowed the cost of it. You don’t in a startup, because startups aren’t managed very well because they can adopt, but I don’t think that’s necessarily an advantage. A lot of changes in IT are unnecessary, costly and done without any benefit realization. I don’t think enterprise is slow either. We’re on the forefront on how to use stuff like ADFS and identity management, to the point where we’ve had to sent the countries leading consultants home because they where total newbies compared to our guys. I just think we’re only fast where it matters. We still use stuff like LinqToSql in production for instance, which is a hopelessly old tech to a lot of people, and it’s also slow and inefficient. Except, when you look at it from an organizational point of view, it allows our developers to do their SQL integrations around 90% faster, and we rarely have more than 60.000 continuous users, so the efficiency never matters. The cost benefits of using LinqToSql is crazy high because developer time is expensive. I don’t think a startup would use it, but I also don’t know of a single startup that doesn’t need to be explained how our entire public service does identity management because they are completely incompetent on matters they can’t look up on stackexchange. Maybe it’s different in America, but I kind of doubt it, because the focus of enterprise will always be on value generation and benefit realization, where as the focus in startups can be almost anything in the world.
euske 8y ago
I have learned that these "slow" traits are all some form of safety/security system. They're in place to avoid corruption, single point of failure, bias, etc. and try to modularize/compartmentize the risk. Most of them are terribly designed and often working against its original purpose, but nonetheless they're there. Building a big enterprise that functions properly is just as hard as building a large software (if not harder), and upgrading a small organization to a big one without a problem is nearly impossible, again just like software.
- taneq 8y ago
  That's a good comparison because corporations effectively are software, running on a cluster of humans. There's a reason we call subroutines "procedures".
CaptainJustin 8y ago
It's easy to say no and no one get's measured on opportunity-loss
kleiba 8y ago
I thought they were all capable of warp speed??
marcus_holmes 8y ago
Nah, it's because of the conjunction of two different factors: First, changing anything large and complex with a lot of moving parts is difficult. You can see the exact same effect in a code base as an organisation. If your code base is small, making a change has limited consequences and you can usually predict them all up front. If your code base is large, you lose the ability to predict what the consequences of a change are. So you need to do it to find out. If you don't have good test suites, with excellent regression tests, then the chances of missing a breaking change are good, and "doing it to find out" will result in a broken system. Organisations don't have test suites, or regression tests. So there's a lot of effort done up front to try to minimise the impact of changes. A lot more work involved in even the smallest changes. Secondly, small organisations are optimised for flexibility, large organisations are optimised for efficiency. A two-person company needs to be flexible and adaptable. In startup terms, they probable haven't hit product-market fit, and are still developing their product, changing it to meet the market as they discover what the market needs. Change is constant, and so the organisation needs to be able to change fast. Doing things efficiently is premature, because the efficient process may change tomorrow and become unefficient. Large organisations have found a business model and grown to fill it. They know exactly what they're making, for whom, for how much, and have a well-defined product and market. All the bottom-line growth will come from reducing costs rather than increasing sales. So they focus on reducing costs. But there's a trade-off between flexibility and efficiency. As the organisation becomes more efficient and more cost-effective, it also becomes less flexible. At the apex of this process, the organisation is so streamlined that any change (by definition) becomes less efficient and therefore less profitable, and therefore any change is difficult and unwanted.
- bboreham 8y ago
  I think you’re missing the context that Ian worked for a large bank, which is a lot more like a software company than you might think.
rawoke083600 8y ago
I remember some "law" about communication in a "network" think its a power-law-distributions as to how bigger the network (organization) the longer it takes to communicate/relay a msg. And that's just in the best case scenarios. Then we start talking about ego and ppl and all odds are off. Unless you working on the production line... fewer ppl are "usually" better!
LaserToy 8y ago
Why distributed systems are hard? Because of many actors and communications overhead. Big companies are slow for the same reason.
pwnguin 8y ago
> In order to ensure that changes to systems can be attributed to responsible individuals, there is usually some kind of system that tracks and audits changes. One person will raise a ‘change record’, which will usually involve filling out an enormous form, and then this change must be ‘signed off’ by one or more other person to ensure that changes don’t happen without due oversight. And in any other planet, we call that a 'Pull Request.' My first real tech job was a community college that really bought into the whole ITIL framework. Important changes typically had to go through a Change Management Board, which met weekly. Meanwhile, key authentication systems involved passing passwords from PHP to perl to bash to vbscript in cleartext, in such a way that dollar signs and other string interpolation sigils would be processed, and therefore were banned. The person who wrote this kludge is now in charge of IT security for the college. And there was no version control to speak of anywhere, definitely no puppet or chef or ansible. It worked, but there were pretty much monthly fuckups along the lines of 'and then the utility truck backed into our power distribution cabinet' or 'the SAN vendor's technician mentioned this is the third time this week he's been on site with a client to deploy this emergency stability patch,' or 'the new guy upgraded the antivirus running on our databases, and we can't roll back because nobody has the old installer anymore' or my personal favorite: 'this position requires oncall duties 24/7/365.'
- greedo 8y ago
  My company uses ITIL. Definitely a four letter word. I feel like Peter Gibbons much of the time. For me to deploy a server, I need to create a Change Request, have it reviewed by my local Change Review Board. They know nothing, and are a rubber stamp. Next it goes the the Main Change Review Board. They are higher up, and know even less. I have to have my CR into the LCRB a week before they meet, and then it takes an additional week for the MCRB to approve it. So at least two weeks to deploy a new VM. Retirement can't come soon enough.
jedberg 8y ago
None of these things are insurmountable, your organization just has to be willing to try. For example, Dev and Ops don't have to be separate, even in a regulated world. You just have to make the effort to show your auditors that there are sufficient controls in place to make changes safe. You also have to work to make sure as much change is out of scope as possible. Ok, maybe the payments system requires that one person writes code and one person deploys it, but if you make sure that almost all of the payment system is out of scope except for the actual money processing, it will be rare that you need to change that one piece of code, so effectively you've avoided the problem. Same with change control -- you don't have to have a big onerous form. You can build a system that automatically logs all the relevant information as to who made a change, so that you have a log that you can show your auditors or whomever needs to check. Again though, your organization has to be willing to spend time on the tooling, but the tooling scales better than a new process. And then for security, again, you don't necessarily need sign off on every change. If you build automated auditing systems that audit every change for security implications, and have sufficient logging, you can be reasonably sure that the changes are secure, and then only dig deeper on the ones that are suspect, or get people to agree to seek approval for things that are big changes or might cause a security problem. If your organization is willing to spend the time building the software and working with the auditors to explain how everything works, you can avoid most of these problems, and then your enterprise will scale much better.
da_murvel 8y ago
You haven't experienced slow until you've worked for a administrative authority, trust me.
sorokod 8y ago
I would argue that an "enterprise" is slow because, as an entity that needs to protected it's structure, it "wants" to be slow. In other words it is not a bug in the system, it is how the system is designed. It is true that there are multiple inefficiencies: kingdom building, incompetence and laziness, risk aversion, broken communication lines etc... but in a very fundamental way an "enterprise" doesn't "want" to be fixed
aappleby 8y ago
Because we have a billion users.
qznc 8y ago
I think the article missed the point because it compares unregulated startups with regulated enterprises. There certainly are more enterprises in regulated industries because working there requires deeper pockets than most startups have. I believe enterprises are slow because they optimize for efficiency in the sense to use the least amount of energy to reach a goal. They do this by learning institutionally which means by tuning their processes, standards, norms, guidelines, etc. This works great if you need to mass-produce physical things. Agile is very inefficient. All this continuous integration machinery, all those team meetings, all these frequent reprioritizations. Agile can be very effective though because you notice earlier when you are wrong. The hard thing to do is that you have to stop when you went wrong. Otherwise you are just doing a more wasteful waterfall. For enterprises this is a cultural change which is hard to do. The problem is that these enterprises have to change because software is eating the world. Fortunately for them, regulation provides a good moat against disruption. For now. I already blogged something like this before: http://beza1e1.tuxen.de/cost_of_agile.html http://beza1e1.tuxen.de/cost_of_agile.html
INTPenis 8y ago
I've been in the position that I've transitioned from being a small agile company to being assimilated by a giant enterprise and being forced to follow their processes. So from my perspective enterprises are slow because of a few reasons but I'll list what I think are the most relevant and valid reasons first. 1) Slow and steady is easier to manage when you have a lot of people. Simply summarized. For example when we merged with other companies to become a giant company we were little silos who all did things our own way. Not only that but some of these silos were habitual doers. Devops people who went from idea to implementation within days. They were used to walking into the CEOs office and asking for 5 rack servers, and getting it. :) I loved that climate but what happens when those passionate people disappear and they haven't documented any of what they have done? You end up wasting a lot of time figuring it out with new staff. So a natural progression was to use project and service managers and then we're already slowing things down a lot. 2) Service packaging. When given a situation like the one described above, what exactly are we selling? No mountain is high enough for us, no request impossible to implement. So clients get whatever they want within moments during this stage. And then we're merged and become a giant company. There might be a merger of 2 or more smaller companies that worked in this way. You now have a giant mess of custom made services and offers to clients. I don't envy the people who need to sort that mess out. So this causes you to be a lot more careful and have a lot more meetings before anything can be implemented. 3) More classic reason for slowness in big enterprise is that it is in fact very big and it takes special skill to be able to work with people you never met. The talent you need might be across the country or in Bangalore. That's a situation you must adapt to. But some people send one e-mail and expect things to get done. No, use the phone! Be proactive but without being annoying. Setup a case where you track your progress, even if it's just trying to contact a storage specialist in Bangalore. The case will remind you every week to call them until you get them on the phone. Or it will remind you that you've done this 3 times now and you can escalate. Case management also slows down processes. 4) And of course regulations, also classic. We have government contracts that don't even allow us to get help from people outside the country. And sometimes have such strict rules that it takes weeks to approve a new employee access to their systems. But a combination of 1 and 2 force you to implement processes and frameworks that tend to slow all things down. And to easier follow frameworks like ITIL for example you get big heavy tools that also slow things down and cause frustration among employees. These are all my personal observations, I have no training in ITIL or in business management.
picsao 8y ago
It depends on the Organisation structure. Usually it's a byproduct of a external view of efficiency and then the creation of tribal solos,where chieftains use work done as bargaining chip. TL, dr;within a company socialism may reign unchecked.
starbugs 8y ago
Static positional hierarchy. It's guaranteed to slow you down and may even turn your work environment toxic. It's sometimes set up to promote ego and protectionism (of your position) instead of aiming to achieve for the company. If your company's environment changes, it's so static that moving quickly becomes impossible. The only option you have then is buying up start-ups and decoupling them from the corporate structure. It's utterly absurd if you think about it thoroughly.
netcan 8y ago
In "Sapiens," the writer argues that "flexible cooperation in large groups" to be the primary limiting as the defining characteristic of Sapiens and the limiting factor on the development of human history. Early on it was proto-language enabling better cooperation within our traditional troops. Later it was abstract symbolism. Tribal identity allowed us to have tribal identity relationships with people we don't know. Concepts like "king" or "priest" allowed all sorts of things to exist which couldn't earlier. Religions, empires, armies, bureaocracies, joint stock companies... Anyway, flexible cooperation in large groups is still a limiting factor. That's why a 4 person startup can solve problems in a week that an enterprise can't solve with 40 in a month. In a nutshell, the specific mechanisms of enterprise slowness are interesting, but inevitable. There will always be something. There's flexibility, cooperation & size function. If you want to increase one, the others needs to decrease. Bureocracies limit flexibility, but allow copperation to happen over a larger group. Markets are flexible and scalable, but uncooperative. Startups are cooperative and flexible, but small. This last one is where we came from.
- zwischenzug 8y ago
  Author here - that's funny, I also wrote about that book here: https://zwischenzugs.com/2017/10/15/my-20-year-experience-of-software-development-methodologies/ https://zwischenzugs.com/2017/10/15/my-20-year-experience-of...
  - netcan 8y ago
    That's a really great post. You write well. I think this sums it up nicely: we were small enough not to need a collective fiction we had to name. I think we're of a mind on a lot of these things. Another broad concept that I think ties into this topic is "legibility."^ The "collective fictions" you guys had when you thought you had none were illegible, informal ones. Dealing with unaknowledged vagueness is something people do really well, in small groups. "Everyone should be polite and respectful" is a statement we understand, even if it's vague. If a disciplinary committee to decide if someone has been polite, then that statement is no longer enough. You'll need legibility, something like a law code. Going from hacking to RAD to agile is an attempt at legibility, a theoretically understandable and justifiable process for making decisions and writing code. ^Stolen from Venkatesh @ ribbonfarm.
amelius 8y ago
Because they lack a soul.
nwhatt 8y ago
Great piece, there are things for developers to think about but sales and marketing teams can learn even more.
vpwp 8y ago
The internal arguments I've seen in companies with 100, 1000, and 10,000 employees have one thing in common: nobody differentiates good decisions made because we were lucky and good decisions made because we understood the problem. The outcome of those conversations end up with being shut down by "let's be practical here". Companies with less than 100 avoid both of these problems simply by employees being more exposed.
segmondy 8y ago
Enterprises are slow because they are usually very profitable and are making tons of money. A well fed animal is slow.
solatic 8y ago
> So however great you are as a team, you’re delivery cadence is constrained to a local maxima based on your external dependencies, which are (effectively) non-negotiable. Precisely this. Which is why strong enterprise leadership needs to map out the value streams of their teams, understand where their bottlenecks are, and work at improving the bottlenecks. When the bottleneck is no longer affecting work throughput, find the next bottleneck, and improve it. Rinse and repeat. Doing so requires senior leadership's direct intervention, not just because only senior leadership has the full picture, but also because the main way of motivating enterprise employees is by directing the attention of senior leadership at them. > it helps if people get fired for not constructively working with the changes It doesn't just help, it's a non-negotiable necessity. Enterprise bottlenecks are not literal/physical/PVC-pipe bottlenecks, they're people. Organizations which cannot fire (literally or effectively, by moving them to non-bottlenecked projects) their non-compliant bottlenecks are organizations which cannot fix their bottlenecks.
- lostapathy 8y ago
  It is also important to realize this very problem isn’t just a matter of headcount, and can happen in a very small “enterprise.” At my last job (~40 employees total) the dev team was absolutely crippled by other teams blocking us simply because they could, and the “leadership” would still take it out on the dev team because our work was ultimately the deliverable to the customer. I got out, they sufferred a major data breach, and from all accounts it is the same old story to this day.
- jimmy1 8y ago
  I disagree that senior leadership is required, maybe indirectly, but not as direct as you suppose. The hard part is firing the right people. If you are going to fire someone, you better be absolutely right it's the cause of the bottleneck, and it really wasn't something else (If I had a dollar for every time some new or existing senior leader thought they could just come in and fire a few people to shake up a place and turn the ship around, boy I would be rich) My experience, everyone is always pointing the finger to something else -- sometimes they are right, sometimes it is a case of simple finger pointing. It becomes difficult to distinguish the finger pointers from the ones that are genuinely trying to overcome the bottleneck. And once senior leadership gets involved, the finger pointing gets exacerbated because now people feel like they might be threatened. You are right that the problem is people, but it is also cultural -- what is the culture like in your organization? Do you have a culture of problem solvers and people willing to set aside their own selfish goals to achieve the goal of removing the bottleneck? Hire people that will instill this culture into your organization -- grassroots style, and watch it spread like wildfire. To that -- I look to my HR department. How are we hiring people? Are we assessing not just on talent but also the type of personality traits we want to see exemplified in our organization? This becomes difficult but it must be done, because top down intervention can only last so long, but if you want lasting change, it has to start at the local level and spread up. That also means senior leadership empowering those at the bottom. To me its not a fire / senior leadership issue, it's culture, nurturing that culture. If you are a director or senior leader in a tech team, you need to really take a hard look at the culture you want, and P.S. this means that you yourself must exhibit these qualities as well because people always look up for role models.
altitudinous 8y ago
constant compromise democracy employees are interested in self preservation, not the enterprise itself solutions are not the best available, instead they are those that cause the least friction there is nothing different about a software enterprise from any other enterprise
DrNuke 8y ago
Good reason is strict quality assurance procedures at any level, which seems bureaucratic but it’s mandatory for b2b and big b2c. Bad reason is bureaucracy as a mean used internally to stand still against any change by slowing quality assurance to death.
throwaway456321 8y ago
Why are enterprises so slow? Simple. People in Enterprises with any decision making authority at all have comfortable lifestyles and earn a lot of money. Job #1 for these people is keep the cash and perks flowing. What's the benefit to them in changing anything, compared to the risk ? So do nothing ! Add to that in many cases, the only people with authority to take decisions are close to retirement. Something they don't want to jeopardize !
- beat 8y ago
  Have you ever actually worked in the enterprise?
  - throwaway456321 8y ago
    yes, have you ?
beat 8y ago
As someone who has spent a couple of decades in mostly enterprise environments, I assure you this article is largely correct - but it's not as bad as it seems. I'm reminded of the time I went to a meetup in Silicon Valley, talking about selling to the enterprise, and hearing a bunch of people shocked at how slow everything is, the scope of failures, and how the enterprise people must be really stupid and lazy. Nah. Enterprise people are just as smart and hardworking as startups. But they have much bigger problems. Just as it's harder to turn a battleship than a speedboat, the scale matters. One thing I pointed out to people at the meetup... if you heard of a startup doing $50m annually, you'd think they were big and successful, right? I've worked on several different projects that size in the enterprise, and they tend to be just a sideline in some behemoth doing tens of billions of dollars (or more) of business a year. One thing I often say about the enterprise, and what makes it different from startups - in the enterprise, it's more important to not be wrong than to be right. This encourages cautious behavior. Missed opportunities are better than disasters. Anyone with real experience in the enterprise has been on a proper death march. They've seen just how wrong things can get. Put a couple of years of labor in, handcuffed to a shambling corpse of a zombie project, and your attitude toward failure changes. The flip side, though, and this is awesome, is that you can work on really BIG things. Much bigger than any startup is tackling. That's fun.
- da02 8y ago
  What are the range of company profit margins on the 10s of billions of dollars enterprise companies? I thought high costs would lead to razor-thin margins.
  - beat 8y ago
    Not necessarily. They often have huge moats. And they often have highly profitable businesses just because competing is dreadfully hard and complex, so they can afford for their software that makes the business work to be sucky and slow. The real danger for the big corporation business model is technological/social shifts wiping their business model out, not competition. I saw this when I worked at the Fed, years ago. I worked in a division that had two major financial products. One was 80% of their revenue and very powerful, a nearly billion dollar business. The other (what I worked on) was 20% of the division revenue. In the span of seven or eight years, those numbers flipped. The big line of business basically collapsed due to changes in the financial industry, and the smaller product basically ate all of its revenue and then some. This meant hundreds of layoffs and some very painful realignment at the executive level, including months wasted on a power grab.
shadowtree 8y ago
Apple is an enterprise. Facebook is an enterprise. Tesla is an enterprise. Amazon is an enterprise. Goldman Sachs is an enterprise. Nissan is an enterprise. Successful enterprise cos move fast where it matters. Rolling out some desktop OS thingy when your core business is retail? No need to be excellent there. Excellence has to be in the areas where you compete, rest is a distraction.
BrendanD 8y ago
Usually it is something to do with the di-lithium crystals.
gtycomb 8y ago
This article mentions a Jeff Bezos' big mandate with six points. For an insight on the "slowness", note that we are looking at only a technical strategy, a part of something bigger related to a particular enterprise. I have seen very similar technical mandates in many large enterprises and this is not a full picture to move on. The ability to move rapidly enough comes not from having this mandate over another. I don't know the inner workings at Amazon, but if this is a technical mandate for the enterprise, the glory of Jeff Bezos is that he has a personal vision and ability to execute and orchestrate it across business units and acquisitions, have in place the right team who understand it to move forward with it. Not many organizations have the right leadership to actually implement a technical strategy like the one, that's the truth. This is an issue that I have been seeing too often unfortunately.
ABCLAW 8y ago
1) I love the domain name. So relevant to the topic. 2) I fundamentally disagree with the premise and content of this article. It is ALL window dressing. Organizational Design is a field that deals with this issue in depth. The OODA loop is one of the key ideas for understanding decision latency. Basically, it takes time to take in data, figure out what to do with it, decide on a solution, and then enact it. Basically the quicker you run through this loop, the quicker your organization's clock speed is. But why would that speed change depending on the size of the organization? Well, because the first three steps in the loop take more time with more stakeholders even if the final one wouldn't change. Humans do not think infinitely fast. As more stakeholders become involved, more thinking is required. But we systematically ignore this cost, despite it being very present and observable, because it involves thinking about thinking. See that article on the front page about how K8s are cost efficient for personal projects? It fully discounts the cost of the thinking required to learn and understand the system. The head of your multi-billion dollar corporation wouldn't need board slide deck summaries of financials broken out by division, or updates regarding process change initiatives at the [X] level if they had infinite clock speed. They'd just parse the inhuman levels of data and come to a sensible conclusion. But they can't. This means there is a ton of work repackaging and communicating information across levels of an organization to fit the available thinking time available at each level. From the perspective of someone dealing with the organization, however, things appear monolithic: when a customer service agent gives us an absurd reply to a query, we assume the entire organization is stupid.
jiveturkey 8y ago
At first blush, it is a good and maybe even great article. But there's a serious problem that the author is writing about his very small sample of experience and claiming it applies broadly. First, he sets up a false dichotomy between the (good) lean startup and the (bad) regulated enterprise. Because his enterprise experience was in a regulated industry, BTW regulated for Very Good Reasons. But actually most enterprises are not regulated, and don't just take a cautionary approach because the government bogeyman told them to and it's out of their control -- with very strong implication they would not choose that life if they could make up their own mind. The real truth is that when you, as cowboy devops, make a mistake that causes downtime or data loss, you bring huge losses to the company in downtime, loss of productivity, actual cash out, maybe even public perception if the fault is externally visible. The priorities are different, plain and simple, and rightly so. Enterprises have to protect themselves. Startups have to take risks. One way is not good and the other bad. They are just different. The idea that one needs to fight the valiant fight from within is absurd. Switch jobs to a smaller company! The author claims that it "feels like failure" which just means he is in the wrong job at the wrong company. That's all. It's pretty bad conclusion to hold up Bezos' mandate as some shining example. In reality it has caused lots of internal strife and Bezos is known to have pet projects and the rest of you losers can suck it. I do love, and will steal, the decagon of despair.
dandare 8y ago
I have been only recently hired as a product manager in a large company. I have a technical background, mostly in startups and I am struggling to understand my bosses motivation. Everything is super slow, but speed is priority only on paper. In practice the only important measure is to look good and be popular. Incompetence is not an issue as long as it is calculated in your roadmap and in your estimates. To the contrary, being proactive and pushing for improvements can only land you in trouble, because your authority is limited and as a manager you depend on the good will of many incompetent incumbents. So far I have learned that management, aka navigating politics, is harder than navigating technology.
madmulita 8y ago
Plain and simple, we are full of incompetence.
jussy 8y ago
Perhaps I missed it but where is the point that a large percentage of all enterprise systems are designed for the internal purposes of running that enterprise and not directly involved in delivering the product/s that the company is paid for?
itst 8y ago
As a colleague put it: The majority decides what's right and wrong, regardless of facts or expertise.

290 points by okket 8y ago

"Resistance is futile. Dev and Ops must be separate because that’s what we wrote down years ago." Really well written piece that clearly explains the reality of "Enterprise DevOps".

exabrial 8y ago
Yep, literally written down: https://en.m.wikipedia.org/wiki/Sarbanes–Oxley_Act https://en.m.wikipedia.org/wiki/Sarbanes–Oxley_Act If a corporate lawyer ever ambushes you with this, the best thing to do is ask instead how you can make your current process auditable, rather than tearing the org apart. If that doesn't work, start looking for a more innovative place to work!

kenhwang 8y ago

I'd say it's mostly regulation, risk-aversion, and company size. When you're small, it's easy to avoid regulatory attention and whatever slap-on-the-wrist fine you may get isn't a lot of money. The change and risk aversion really just comes from keeping the business running. When things work, try not to break them. Old large enterprises tend to have a lot of business critical things that "just work" and have been for a long time. Younger/smaller companies tend to have fewer battle scarred systems. But once they get scarred, they typically add a process or policy to prevent it from happening again. After enough of them, congrats, you're enterprise speed.

rossdavidh 8y ago
All true, but I would add: there's a lot less downside to change, when nothing much is working yet. Much of enterprise slowness is irrational, but much is not. Big systems are harder to predict consequences for, and big old systems have a lot that already works.
- kenhwang 8y ago
  Add in the tendency to consolidate and standardize technologies as the company grows for maintainability just means even little changes quickly ripple through the system. It's like microservices, but with systems of people and businesses.
LoSboccacc 8y ago
and SLAs. once you get big enough your client contracts usually have penalties for failures of any kind. that puts the pressure on getting it right on the first try or at least within contract terms, instead of working toward the goal incrementally. partners that allow for some agile, iterative back and forth becomes rarer as they increase in size, especially since often their own system lack the flexibility for continuous integration or safely rolling back changes/data. that's why you get requirements, specifications, architects and solution managers in the enterprises. people love to mock them for being dysfunctional and some truly are, but once contracts get into the millions and lawyer get involved mistakes are no longer "learning opportunities".

programminggeek 8y ago

They are slow because they are complex as proven by the fact that it's complex to even explain why they are slow.

romed 8y ago

Eh. Article is a bit short on facts. Starting with the first bullet, the only place I've ever worked that rolled out OS updates continuously with a maximum 15-day latency was also an enormous global enterprise with tens of thousands of employees and plenty of regulatory red tape. Now I work at a much smaller company that thinks of itself as "scrappy" but is utterly paralyzed by OS updates in their production fleet because they don't have the tooling and processes around it. It is the tooling and the process that gives you the ability to move quickly!

ocdtrekkie 8y ago
A huge upside of Windows 10's six month rolling releases is that they initially broke a lot of badly written things, and software and processes have had to improve to handle them better. The end result two years later is OS upgrades rarely break much, and regular updates never do. Enterprises can be unstuck but it can be painful. I think enterprises are still shifting from the old approach where you make it work and then it's fine, to realizing that in an internet-connected world, you have to keep up at all times. A few big liability cases would do wonders to push this shift along.
- bigger_cheese 8y ago
  I hope what you are saying about Win 10 is true. My organisation is in process of transitioning from Windows 7 to Windows 10 and things are breaking all over the place. Almost everytime we change versions of software things break when we went moved to office 365 there was breakage (Stuff I would have thought for sure would have been well tested - PST files in outlook for example broke). Same thing when we went from Internet Explorer 8 to 11 our internal intranet didn't work.
  - ocdtrekkie 8y ago
    Nearly everything that works in Windows 7 works in Windows 10, if you poke enough holes in it. The biggest exception to that is really just drivers, because hardware manufacturers have no incentive to update them past a certain point. There was definitely software that had to be updated to the latest versions to work, but that's where we needed to get to anyways. I work in an environment that includes custom-written apps from the Windows XP era, and usually the absolute worst we hit was "hey, this device doesn't work on Windows 10, we have to buy a newer one". Not a big deal when it's a $6 USB-to-serial adapter, but people will look at you like you're crazy when you say the network is incompatible with an HVAC system. (Solution in that case, just take it off the network, you're better off.) But believe it or not, you can still sync Windows Mobile 2003 devices with Windows 10 PCs, Microsoft's backwards compatibility is still insanely solid in most cases.

jeffmcmahan 8y ago

"If it doesn’t go well he’s out of a job and his boss might go to prison" Yeah, I don't think that happens much.

bboreham 8y ago
That’s the point. It doesn’t happen very often because people took care not to sign the document that would put them in prison. Your point may be that people should go to prison for presiding over bad things that weren’t actually against the law. That path sounds tricky.

Forge36 8y ago

I maintain code written by my company's founder from 1998. It hadn't seen many updates to maintain it. It saw a LOT of enhancements. I'm currently unwinding those changes into a full library of support features. It's also not my primary focus. Many new features to fulfill customer needs is a common reason. Many customers, with new requests, issues reported. It's slow because if I'm not methodical I cause problems for people who don't want this next enhancements. It's slow, because the current emergency takes precedence, and taking to long on fixes means a meeting on why something is behind. Business priority is not "deliver fast" it's "keep running, don't attract attention".

jac_no_k 8y ago

Spot on. [apologies for low effort post, but I have nothing more to add.]

sephware 8y ago
That's what the upvote button is for.

tzury 8y ago

As Chris Dixon put is: "A friend of mine got a job at a big company and was shocked to see his colleagues worked just a few productive hours a day. They didn’t seem to care about their work or have relevant expertise. My friend said: “Wow, this company is going under.” Then the company released its quarterly reports and profits rose to an all-time high. The momentum of the company’s brand and relationships was sufficient to propel it forward." http://cdixon.org/2012/05/19/the-default-state-of-a-startup-is-failure/ http://cdixon.org/2012/05/19/the-default-state-of-a-startup-...

coffeemug 8y ago
It isn't just relationships and momentum. XKCD had an interesting what if -- what would happen if a bowling ball was blown up to the size of Earth?[1] Turns out at that scale metal behaves like a liquid. Work at big companies is kind of like that. Once a company gets big enough, any given person doesn't really know anything. Seemingly no one is doing anything productive -- it's meetings all the way down. But the software still gets shipped. It doesn't happen the same way it happens at a startup where someone sits down and codes a feature. It happens in a totally different way, where one hundred engineers go to meetings all day and talk to QA and analytics and bizdev, and maybe commit one line of broken code a week. It takes way more people, and the production process is totally different, and it takes way longer, but stuff still happens (often quite successfully). [1] https://what-if.xkcd.com/46/ https://what-if.xkcd.com/46/
- cpeterso 8y ago
  I read (circa 2000) that IBM's product development processes would need nine months to ship a product that was just an empty box.
  - blihp 8y ago
    That was probably referencing the story of the origin of the IBM PC. This anecdote was given as the reason they got the green light to develop the PC in Boca Raton independent of the rest of IBM.

nikanj 8y ago

Big companies have something to lose. A brand, clients, revenue and profits. People don’t want to risk those. A starup has nothing to lose. The site is janky for two hours? The only users are friends, and the outage might actually bring more visibilty if you make a good write-up. Incidentally, this ties together with “why do young people found more startups than senior executives”

debacle 8y ago

It's people. Enterprises are made out of people.

daniel_iversen 8y ago

Great article. There’s also other good reasons why enterprises move slowly ( cultural and other organisational structures and priorities that conflicts with making the same kinds of decisions a startup might make). A good book is The Innovators Dilemma: https://en.m.wikipedia.org/wiki/The_Innovator%27s_Dilemma https://en.m.wikipedia.org/wiki/The_Innovator%27s_Dilemma

MK_Dev 8y ago

The article is pretty spot on. One thing the author has left out is the fact that all of the problems are magnified 100x when the people involved are incompetent, which, unfortunately, happens a lot in enterprises (one can also argue that slow speed contributes to job security under normal circumstances).

kochikame 8y ago

No one knows anything in big companies Lots of smart, talented and energetic people are expending most of their power just figuring out who to talk to about what or what the hell that out-of-the-blue email from corporate finance means for them. Just figuring out how to get a thing done can be exhausting.

carlivar 8y ago
So, Dunbar's Number
Scene_Cast2 8y ago
Yep, can attest to this. One of the reasons why senior people are extremely productive & valuable is that they seem to know the landscape more than others, and therefore know who to talk to & where to look. Half-jokingly, as soon as an "individual contributor" knows enough to be productive, they're promoted to management.
- csydas 8y ago
  WRT the half-joke, this is one of those truisms that probably has a lot of basis in reality, but there needs to be a corollary about the choice of the individual as well. I was heavily invested in our public facing technical side, working a lot with our on-boarding, teaching newbies to be really self-sufficient, working through the technical issues with our clients directly, and then I was asked to take on a Management position. I liked the pay raise, but I made it clear exactly what I wanted from the position, and though we butt heads on it still, my boss respects the fact that I still am heavily invested with our technical team while doing the Managerial work. We had to revisit the job definition a few times to find a description that matched what I was (am) doing and also made HR happy, but we did it. The point is that in some cases at least, a promotion to management doesn't have to be a death knell for "individual contributors". It depends on the company's willingness to accommodate the wants of the individual, and the individual's own desire to continue contributing. I know too many people who did get the Management golden ticket and relished it for just being an excuse to boss people around and make more dashboards in our CRM.

gtycomb 8y ago

Scanning over responses so far I think a relevant aspect is not underlined strongly enough -- the lack of expertise in some of the key positions in companies "to move things". The status quo prevails. Play it safe, don't shake the boat, bring in Gartner or some such consultant on strategy, and hum along. In the end, outsource a project with internal fanfare (bringing relief to people concerned). This happens all too often. This is how life goes, and as some here pointed out already, even if large inefficiencies are unaddressed a company can be profitable for significant time because of an established market position, brand, extensive regulations in place to protect their interests, some little known high profit products, etc. However give it time, and we see them fade away when upstarts take the risks, have the vision, shake up their market one piece at a time. And the cycle repeats.

berti 8y ago

Risk-aversion, cost of relatively small changes can look large, and the tendency to "design by committee". You can send an email out to some colleagues throughout other units of the business and get inundated with reasons why it won't work, and edge-cases you need to handle, whereas if you'd just implemented it, they'd likely work-around or improve their processes and everyone would have been better off.

tokyodude 8y ago

For me the #1 reason why they are slow is because everything requires consensus or maybe another way of putting it it's hard for people to make a decision on their own. Let's say I want to change some part of the UI of some app. If I'm by myself I just change it. If I'm on a team first I have to convince whoever has the power to decide to change it to allow it to be changed, that might even be a whole committee. Then several people will be involved, likely a UX designer, possible a separate UI designer, all of which will have to discuss the feature and all of which also have other responsibilities. The larger the company the worse this usually gets. Maybe the change has to be run by the security team to make sure the new UI is not a security issue. Maybe it has to run by the privacy team to make sure it's not going to leak data. Maybe it has to be run by an inclusivity team to make sure it's not offensive in some culture I'm not familiar with. Maybe it has to be run by a localization team to make sure it will work in every language. All of those things add time. Maybe they are all required and good, especially at a large company with a global marketplace but they certainly add to the time taken to do something that a tiny team wouldn't do.

killjoywashere 8y ago
A lot of answers to these questions, and the questions themselves, come from further down the food chain. The flip side of the coin is that no start-up is moving a billion tons of rice, or can park an army off a foreign shore in a week, or lay 10,000 tons of steel rail across a continent every day, or delivery a million packages a day, or make 10,000 processors a day, each with the complexity of the entire European system of roads. So speed is relative to the size of the task at hand. Large enterprise can move big things so fast it literally takes the air out of the room.
- monkeydreams 8y ago
  > So speed is relative to the size of the task at hand. I think of organisational decision making like organic decision making - that the speed of the organisation's reactions are dependent upon its frame of reference. The frame of reference here is the time between sensing and reacting. The eyes and ears of small businesses are very close to the brains, and so decisions can be made rapidly by the small number of people who have capability to marshal resources into a response. Enterprises have layers of processing between the eyes and brains, along with many more sensors than a small business, meaning that questions take longer to be settled and decisions made. However no small business can wield the same power as a large organisation when the army of deciders all focus on a single outcome.

eksemplar 8y ago

I work in a muniplacity of 10.000+ employees, and network with other managers from 97 other muniplacities as well as the government, and I think most of those points are wrong. At least in enterprise that aren’t IT companies. There aren’t really regulators the way it’s described, and you wouldn’t worry about them the way it’s described either. Unless you made changes without planning. What you would consider when upgrading your OS, for instance, is how you teach the new OS to the 5000 employees who can’t tell the difference between and iPhone and an android tablet. Ownership is easy in enterprise, we have organized everything into departments and IT operations is owned by IT. It’s audited that way, and things are changed according to the guidelines. You’d literally never have an audit come in and point any of the things suggested out, unless your enterprise isn’t run like an enterprise. The first step in the manual for upgrading, is to make sure you stay within the law and within company policy, and you’d never run a project all the way through without involving people who know regulations. I don’t think enterprise is risk averse either, but I do think you need to be able to prove the benefits of a change, to be allowed the cost of it. You don’t in a startup, because startups aren’t managed very well because they can adopt, but I don’t think that’s necessarily an advantage. A lot of changes in IT are unnecessary, costly and done without any benefit realization. I don’t think enterprise is slow either. We’re on the forefront on how to use stuff like ADFS and identity management, to the point where we’ve had to sent the countries leading consultants home because they where total newbies compared to our guys. I just think we’re only fast where it matters. We still use stuff like LinqToSql in production for instance, which is a hopelessly old tech to a lot of people, and it’s also slow and inefficient. Except, when you look at it from an organizational point of view, it allows our developers to do their SQL integrations around 90% faster, and we rarely have more than 60.000 continuous users, so the efficiency never matters. The cost benefits of using LinqToSql is crazy high because developer time is expensive. I don’t think a startup would use it, but I also don’t know of a single startup that doesn’t need to be explained how our entire public service does identity management because they are completely incompetent on matters they can’t look up on stackexchange. Maybe it’s different in America, but I kind of doubt it, because the focus of enterprise will always be on value generation and benefit realization, where as the focus in startups can be almost anything in the world.

euske 8y ago

I have learned that these "slow" traits are all some form of safety/security system. They're in place to avoid corruption, single point of failure, bias, etc. and try to modularize/compartmentize the risk. Most of them are terribly designed and often working against its original purpose, but nonetheless they're there. Building a big enterprise that functions properly is just as hard as building a large software (if not harder), and upgrading a small organization to a big one without a problem is nearly impossible, again just like software.

taneq 8y ago
That's a good comparison because corporations effectively are software, running on a cluster of humans. There's a reason we call subroutines "procedures".

CaptainJustin 8y ago

It's easy to say no and no one get's measured on opportunity-loss

kleiba 8y ago

I thought they were all capable of warp speed??

marcus_holmes 8y ago

Nah, it's because of the conjunction of two different factors: First, changing anything large and complex with a lot of moving parts is difficult. You can see the exact same effect in a code base as an organisation. If your code base is small, making a change has limited consequences and you can usually predict them all up front. If your code base is large, you lose the ability to predict what the consequences of a change are. So you need to do it to find out. If you don't have good test suites, with excellent regression tests, then the chances of missing a breaking change are good, and "doing it to find out" will result in a broken system. Organisations don't have test suites, or regression tests. So there's a lot of effort done up front to try to minimise the impact of changes. A lot more work involved in even the smallest changes. Secondly, small organisations are optimised for flexibility, large organisations are optimised for efficiency. A two-person company needs to be flexible and adaptable. In startup terms, they probable haven't hit product-market fit, and are still developing their product, changing it to meet the market as they discover what the market needs. Change is constant, and so the organisation needs to be able to change fast. Doing things efficiently is premature, because the efficient process may change tomorrow and become unefficient. Large organisations have found a business model and grown to fill it. They know exactly what they're making, for whom, for how much, and have a well-defined product and market. All the bottom-line growth will come from reducing costs rather than increasing sales. So they focus on reducing costs. But there's a trade-off between flexibility and efficiency. As the organisation becomes more efficient and more cost-effective, it also becomes less flexible. At the apex of this process, the organisation is so streamlined that any change (by definition) becomes less efficient and therefore less profitable, and therefore any change is difficult and unwanted.

bboreham 8y ago
I think you’re missing the context that Ian worked for a large bank, which is a lot more like a software company than you might think.

rawoke083600 8y ago

I remember some "law" about communication in a "network" think its a power-law-distributions as to how bigger the network (organization) the longer it takes to communicate/relay a msg. And that's just in the best case scenarios. Then we start talking about ego and ppl and all odds are off. Unless you working on the production line... fewer ppl are "usually" better!

LaserToy 8y ago

Why distributed systems are hard? Because of many actors and communications overhead. Big companies are slow for the same reason.

pwnguin 8y ago

> In order to ensure that changes to systems can be attributed to responsible individuals, there is usually some kind of system that tracks and audits changes. One person will raise a ‘change record’, which will usually involve filling out an enormous form, and then this change must be ‘signed off’ by one or more other person to ensure that changes don’t happen without due oversight. And in any other planet, we call that a 'Pull Request.' My first real tech job was a community college that really bought into the whole ITIL framework. Important changes typically had to go through a Change Management Board, which met weekly. Meanwhile, key authentication systems involved passing passwords from PHP to perl to bash to vbscript in cleartext, in such a way that dollar signs and other string interpolation sigils would be processed, and therefore were banned. The person who wrote this kludge is now in charge of IT security for the college. And there was no version control to speak of anywhere, definitely no puppet or chef or ansible. It worked, but there were pretty much monthly fuckups along the lines of 'and then the utility truck backed into our power distribution cabinet' or 'the SAN vendor's technician mentioned this is the third time this week he's been on site with a client to deploy this emergency stability patch,' or 'the new guy upgraded the antivirus running on our databases, and we can't roll back because nobody has the old installer anymore' or my personal favorite: 'this position requires oncall duties 24/7/365.'

greedo 8y ago
My company uses ITIL. Definitely a four letter word. I feel like Peter Gibbons much of the time. For me to deploy a server, I need to create a Change Request, have it reviewed by my local Change Review Board. They know nothing, and are a rubber stamp. Next it goes the the Main Change Review Board. They are higher up, and know even less. I have to have my CR into the LCRB a week before they meet, and then it takes an additional week for the MCRB to approve it. So at least two weeks to deploy a new VM. Retirement can't come soon enough.

jedberg 8y ago

None of these things are insurmountable, your organization just has to be willing to try. For example, Dev and Ops don't have to be separate, even in a regulated world. You just have to make the effort to show your auditors that there are sufficient controls in place to make changes safe. You also have to work to make sure as much change is out of scope as possible. Ok, maybe the payments system requires that one person writes code and one person deploys it, but if you make sure that almost all of the payment system is out of scope except for the actual money processing, it will be rare that you need to change that one piece of code, so effectively you've avoided the problem. Same with change control -- you don't have to have a big onerous form. You can build a system that automatically logs all the relevant information as to who made a change, so that you have a log that you can show your auditors or whomever needs to check. Again though, your organization has to be willing to spend time on the tooling, but the tooling scales better than a new process. And then for security, again, you don't necessarily need sign off on every change. If you build automated auditing systems that audit every change for security implications, and have sufficient logging, you can be reasonably sure that the changes are secure, and then only dig deeper on the ones that are suspect, or get people to agree to seek approval for things that are big changes or might cause a security problem. If your organization is willing to spend the time building the software and working with the auditors to explain how everything works, you can avoid most of these problems, and then your enterprise will scale much better.

da_murvel 8y ago

You haven't experienced slow until you've worked for a administrative authority, trust me.

sorokod 8y ago

I would argue that an "enterprise" is slow because, as an entity that needs to protected it's structure, it "wants" to be slow. In other words it is not a bug in the system, it is how the system is designed. It is true that there are multiple inefficiencies: kingdom building, incompetence and laziness, risk aversion, broken communication lines etc... but in a very fundamental way an "enterprise" doesn't "want" to be fixed

aappleby 8y ago

Because we have a billion users.

qznc 8y ago

I think the article missed the point because it compares unregulated startups with regulated enterprises. There certainly are more enterprises in regulated industries because working there requires deeper pockets than most startups have. I believe enterprises are slow because they optimize for efficiency in the sense to use the least amount of energy to reach a goal. They do this by learning institutionally which means by tuning their processes, standards, norms, guidelines, etc. This works great if you need to mass-produce physical things. Agile is very inefficient. All this continuous integration machinery, all those team meetings, all these frequent reprioritizations. Agile can be very effective though because you notice earlier when you are wrong. The hard thing to do is that you have to stop when you went wrong. Otherwise you are just doing a more wasteful waterfall. For enterprises this is a cultural change which is hard to do. The problem is that these enterprises have to change because software is eating the world. Fortunately for them, regulation provides a good moat against disruption. For now. I already blogged something like this before: http://beza1e1.tuxen.de/cost_of_agile.html http://beza1e1.tuxen.de/cost_of_agile.html

INTPenis 8y ago

I've been in the position that I've transitioned from being a small agile company to being assimilated by a giant enterprise and being forced to follow their processes. So from my perspective enterprises are slow because of a few reasons but I'll list what I think are the most relevant and valid reasons first. 1) Slow and steady is easier to manage when you have a lot of people. Simply summarized. For example when we merged with other companies to become a giant company we were little silos who all did things our own way. Not only that but some of these silos were habitual doers. Devops people who went from idea to implementation within days. They were used to walking into the CEOs office and asking for 5 rack servers, and getting it. :) I loved that climate but what happens when those passionate people disappear and they haven't documented any of what they have done? You end up wasting a lot of time figuring it out with new staff. So a natural progression was to use project and service managers and then we're already slowing things down a lot. 2) Service packaging. When given a situation like the one described above, what exactly are we selling? No mountain is high enough for us, no request impossible to implement. So clients get whatever they want within moments during this stage. And then we're merged and become a giant company. There might be a merger of 2 or more smaller companies that worked in this way. You now have a giant mess of custom made services and offers to clients. I don't envy the people who need to sort that mess out. So this causes you to be a lot more careful and have a lot more meetings before anything can be implemented. 3) More classic reason for slowness in big enterprise is that it is in fact very big and it takes special skill to be able to work with people you never met. The talent you need might be across the country or in Bangalore. That's a situation you must adapt to. But some people send one e-mail and expect things to get done. No, use the phone! Be proactive but without being annoying. Setup a case where you track your progress, even if it's just trying to contact a storage specialist in Bangalore. The case will remind you every week to call them until you get them on the phone. Or it will remind you that you've done this 3 times now and you can escalate. Case management also slows down processes. 4) And of course regulations, also classic. We have government contracts that don't even allow us to get help from people outside the country. And sometimes have such strict rules that it takes weeks to approve a new employee access to their systems. But a combination of 1 and 2 force you to implement processes and frameworks that tend to slow all things down. And to easier follow frameworks like ITIL for example you get big heavy tools that also slow things down and cause frustration among employees. These are all my personal observations, I have no training in ITIL or in business management.

picsao 8y ago

It depends on the Organisation structure. Usually it's a byproduct of a external view of efficiency and then the creation of tribal solos,where chieftains use work done as bargaining chip. TL, dr;within a company socialism may reign unchecked.

starbugs 8y ago

Static positional hierarchy. It's guaranteed to slow you down and may even turn your work environment toxic. It's sometimes set up to promote ego and protectionism (of your position) instead of aiming to achieve for the company. If your company's environment changes, it's so static that moving quickly becomes impossible. The only option you have then is buying up start-ups and decoupling them from the corporate structure. It's utterly absurd if you think about it thoroughly.

netcan 8y ago

In "Sapiens," the writer argues that "flexible cooperation in large groups" to be the primary limiting as the defining characteristic of Sapiens and the limiting factor on the development of human history. Early on it was proto-language enabling better cooperation within our traditional troops. Later it was abstract symbolism. Tribal identity allowed us to have tribal identity relationships with people we don't know. Concepts like "king" or "priest" allowed all sorts of things to exist which couldn't earlier. Religions, empires, armies, bureaocracies, joint stock companies... Anyway, flexible cooperation in large groups is still a limiting factor. That's why a 4 person startup can solve problems in a week that an enterprise can't solve with 40 in a month. In a nutshell, the specific mechanisms of enterprise slowness are interesting, but inevitable. There will always be something. There's flexibility, cooperation & size function. If you want to increase one, the others needs to decrease. Bureocracies limit flexibility, but allow copperation to happen over a larger group. Markets are flexible and scalable, but uncooperative. Startups are cooperative and flexible, but small. This last one is where we came from.

zwischenzug 8y ago
Author here - that's funny, I also wrote about that book here: https://zwischenzugs.com/2017/10/15/my-20-year-experience-of-software-development-methodologies/ https://zwischenzugs.com/2017/10/15/my-20-year-experience-of...
- netcan 8y ago
  That's a really great post. You write well. I think this sums it up nicely: we were small enough not to need a collective fiction we had to name. I think we're of a mind on a lot of these things. Another broad concept that I think ties into this topic is "legibility."^ The "collective fictions" you guys had when you thought you had none were illegible, informal ones. Dealing with unaknowledged vagueness is something people do really well, in small groups. "Everyone should be polite and respectful" is a statement we understand, even if it's vague. If a disciplinary committee to decide if someone has been polite, then that statement is no longer enough. You'll need legibility, something like a law code. Going from hacking to RAD to agile is an attempt at legibility, a theoretically understandable and justifiable process for making decisions and writing code. ^Stolen from Venkatesh @ ribbonfarm.

amelius 8y ago

Because they lack a soul.

nwhatt 8y ago

Great piece, there are things for developers to think about but sales and marketing teams can learn even more.

vpwp 8y ago

The internal arguments I've seen in companies with 100, 1000, and 10,000 employees have one thing in common: nobody differentiates good decisions made because we were lucky and good decisions made because we understood the problem. The outcome of those conversations end up with being shut down by "let's be practical here". Companies with less than 100 avoid both of these problems simply by employees being more exposed.

segmondy 8y ago

Enterprises are slow because they are usually very profitable and are making tons of money. A well fed animal is slow.

solatic 8y ago

> So however great you are as a team, you’re delivery cadence is constrained to a local maxima based on your external dependencies, which are (effectively) non-negotiable. Precisely this. Which is why strong enterprise leadership needs to map out the value streams of their teams, understand where their bottlenecks are, and work at improving the bottlenecks. When the bottleneck is no longer affecting work throughput, find the next bottleneck, and improve it. Rinse and repeat. Doing so requires senior leadership's direct intervention, not just because only senior leadership has the full picture, but also because the main way of motivating enterprise employees is by directing the attention of senior leadership at them. > it helps if people get fired for not constructively working with the changes It doesn't just help, it's a non-negotiable necessity. Enterprise bottlenecks are not literal/physical/PVC-pipe bottlenecks, they're people. Organizations which cannot fire (literally or effectively, by moving them to non-bottlenecked projects) their non-compliant bottlenecks are organizations which cannot fix their bottlenecks.

lostapathy 8y ago
It is also important to realize this very problem isn’t just a matter of headcount, and can happen in a very small “enterprise.” At my last job (~40 employees total) the dev team was absolutely crippled by other teams blocking us simply because they could, and the “leadership” would still take it out on the dev team because our work was ultimately the deliverable to the customer. I got out, they sufferred a major data breach, and from all accounts it is the same old story to this day.
jimmy1 8y ago
I disagree that senior leadership is required, maybe indirectly, but not as direct as you suppose. The hard part is firing the right people. If you are going to fire someone, you better be absolutely right it's the cause of the bottleneck, and it really wasn't something else (If I had a dollar for every time some new or existing senior leader thought they could just come in and fire a few people to shake up a place and turn the ship around, boy I would be rich) My experience, everyone is always pointing the finger to something else -- sometimes they are right, sometimes it is a case of simple finger pointing. It becomes difficult to distinguish the finger pointers from the ones that are genuinely trying to overcome the bottleneck. And once senior leadership gets involved, the finger pointing gets exacerbated because now people feel like they might be threatened. You are right that the problem is people, but it is also cultural -- what is the culture like in your organization? Do you have a culture of problem solvers and people willing to set aside their own selfish goals to achieve the goal of removing the bottleneck? Hire people that will instill this culture into your organization -- grassroots style, and watch it spread like wildfire. To that -- I look to my HR department. How are we hiring people? Are we assessing not just on talent but also the type of personality traits we want to see exemplified in our organization? This becomes difficult but it must be done, because top down intervention can only last so long, but if you want lasting change, it has to start at the local level and spread up. That also means senior leadership empowering those at the bottom. To me its not a fire / senior leadership issue, it's culture, nurturing that culture. If you are a director or senior leader in a tech team, you need to really take a hard look at the culture you want, and P.S. this means that you yourself must exhibit these qualities as well because people always look up for role models.

altitudinous 8y ago

constant compromise democracy employees are interested in self preservation, not the enterprise itself solutions are not the best available, instead they are those that cause the least friction there is nothing different about a software enterprise from any other enterprise

DrNuke 8y ago

Good reason is strict quality assurance procedures at any level, which seems bureaucratic but it’s mandatory for b2b and big b2c. Bad reason is bureaucracy as a mean used internally to stand still against any change by slowing quality assurance to death.

throwaway456321 8y ago

Why are enterprises so slow? Simple. People in Enterprises with any decision making authority at all have comfortable lifestyles and earn a lot of money. Job #1 for these people is keep the cash and perks flowing. What's the benefit to them in changing anything, compared to the risk ? So do nothing ! Add to that in many cases, the only people with authority to take decisions are close to retirement. Something they don't want to jeopardize !

beat 8y ago
Have you ever actually worked in the enterprise?
- throwaway456321 8y ago
  yes, have you ?

beat 8y ago

As someone who has spent a couple of decades in mostly enterprise environments, I assure you this article is largely correct - but it's not as bad as it seems. I'm reminded of the time I went to a meetup in Silicon Valley, talking about selling to the enterprise, and hearing a bunch of people shocked at how slow everything is, the scope of failures, and how the enterprise people must be really stupid and lazy. Nah. Enterprise people are just as smart and hardworking as startups. But they have much bigger problems. Just as it's harder to turn a battleship than a speedboat, the scale matters. One thing I pointed out to people at the meetup... if you heard of a startup doing $50m annually, you'd think they were big and successful, right? I've worked on several different projects that size in the enterprise, and they tend to be just a sideline in some behemoth doing tens of billions of dollars (or more) of business a year. One thing I often say about the enterprise, and what makes it different from startups - in the enterprise, it's more important to not be wrong than to be right. This encourages cautious behavior. Missed opportunities are better than disasters. Anyone with real experience in the enterprise has been on a proper death march. They've seen just how wrong things can get. Put a couple of years of labor in, handcuffed to a shambling corpse of a zombie project, and your attitude toward failure changes. The flip side, though, and this is awesome, is that you can work on really BIG things. Much bigger than any startup is tackling. That's fun.

da02 8y ago
What are the range of company profit margins on the 10s of billions of dollars enterprise companies? I thought high costs would lead to razor-thin margins.
- beat 8y ago
  Not necessarily. They often have huge moats. And they often have highly profitable businesses just because competing is dreadfully hard and complex, so they can afford for their software that makes the business work to be sucky and slow. The real danger for the big corporation business model is technological/social shifts wiping their business model out, not competition. I saw this when I worked at the Fed, years ago. I worked in a division that had two major financial products. One was 80% of their revenue and very powerful, a nearly billion dollar business. The other (what I worked on) was 20% of the division revenue. In the span of seven or eight years, those numbers flipped. The big line of business basically collapsed due to changes in the financial industry, and the smaller product basically ate all of its revenue and then some. This meant hundreds of layoffs and some very painful realignment at the executive level, including months wasted on a power grab.

shadowtree 8y ago

Apple is an enterprise. Facebook is an enterprise. Tesla is an enterprise. Amazon is an enterprise. Goldman Sachs is an enterprise. Nissan is an enterprise. Successful enterprise cos move fast where it matters. Rolling out some desktop OS thingy when your core business is retail? No need to be excellent there. Excellence has to be in the areas where you compete, rest is a distraction.

BrendanD 8y ago

Usually it is something to do with the di-lithium crystals.

gtycomb 8y ago

This article mentions a Jeff Bezos' big mandate with six points. For an insight on the "slowness", note that we are looking at only a technical strategy, a part of something bigger related to a particular enterprise. I have seen very similar technical mandates in many large enterprises and this is not a full picture to move on. The ability to move rapidly enough comes not from having this mandate over another. I don't know the inner workings at Amazon, but if this is a technical mandate for the enterprise, the glory of Jeff Bezos is that he has a personal vision and ability to execute and orchestrate it across business units and acquisitions, have in place the right team who understand it to move forward with it. Not many organizations have the right leadership to actually implement a technical strategy like the one, that's the truth. This is an issue that I have been seeing too often unfortunately.

ABCLAW 8y ago

1) I love the domain name. So relevant to the topic. 2) I fundamentally disagree with the premise and content of this article. It is ALL window dressing. Organizational Design is a field that deals with this issue in depth. The OODA loop is one of the key ideas for understanding decision latency. Basically, it takes time to take in data, figure out what to do with it, decide on a solution, and then enact it. Basically the quicker you run through this loop, the quicker your organization's clock speed is. But why would that speed change depending on the size of the organization? Well, because the first three steps in the loop take more time with more stakeholders even if the final one wouldn't change. Humans do not think infinitely fast. As more stakeholders become involved, more thinking is required. But we systematically ignore this cost, despite it being very present and observable, because it involves thinking about thinking. See that article on the front page about how K8s are cost efficient for personal projects? It fully discounts the cost of the thinking required to learn and understand the system. The head of your multi-billion dollar corporation wouldn't need board slide deck summaries of financials broken out by division, or updates regarding process change initiatives at the [X] level if they had infinite clock speed. They'd just parse the inhuman levels of data and come to a sensible conclusion. But they can't. This means there is a ton of work repackaging and communicating information across levels of an organization to fit the available thinking time available at each level. From the perspective of someone dealing with the organization, however, things appear monolithic: when a customer service agent gives us an absurd reply to a query, we assume the entire organization is stupid.

jiveturkey 8y ago

At first blush, it is a good and maybe even great article. But there's a serious problem that the author is writing about his very small sample of experience and claiming it applies broadly. First, he sets up a false dichotomy between the (good) lean startup and the (bad) regulated enterprise. Because his enterprise experience was in a regulated industry, BTW regulated for Very Good Reasons. But actually most enterprises are not regulated, and don't just take a cautionary approach because the government bogeyman told them to and it's out of their control -- with very strong implication they would not choose that life if they could make up their own mind. The real truth is that when you, as cowboy devops, make a mistake that causes downtime or data loss, you bring huge losses to the company in downtime, loss of productivity, actual cash out, maybe even public perception if the fault is externally visible. The priorities are different, plain and simple, and rightly so. Enterprises have to protect themselves. Startups have to take risks. One way is not good and the other bad. They are just different. The idea that one needs to fight the valiant fight from within is absurd. Switch jobs to a smaller company! The author claims that it "feels like failure" which just means he is in the wrong job at the wrong company. That's all. It's pretty bad conclusion to hold up Bezos' mandate as some shining example. In reality it has caused lots of internal strife and Bezos is known to have pet projects and the rest of you losers can suck it. I do love, and will steal, the decagon of despair.

dandare 8y ago

I have been only recently hired as a product manager in a large company. I have a technical background, mostly in startups and I am struggling to understand my bosses motivation. Everything is super slow, but speed is priority only on paper. In practice the only important measure is to look good and be popular. Incompetence is not an issue as long as it is calculated in your roadmap and in your estimates. To the contrary, being proactive and pushing for improvements can only land you in trouble, because your authority is limited and as a manager you depend on the good will of many incompetent incumbents. So far I have learned that management, aka navigating politics, is harder than navigating technology.

madmulita 8y ago

Plain and simple, we are full of incompetence.

jussy 8y ago

Perhaps I missed it but where is the point that a large percentage of all enterprise systems are designed for the internal purposes of running that enterprise and not directly involved in delivering the product/s that the company is paid for?

itst 8y ago

As a colleague put it: The majority decides what's right and wrong, regardless of facts or expertise.