15 ms·

Why are large companies so difficult to rescue (regarding internal technology)

208 points by lkrubner 7y ago

aj7 7y ago
MBAs and other “professional managers.” Persons in a position of power that have not done any company business or tasks, either never, or for more than 5 years.
rossdavidh 7y ago
Having worked at and for both large and small companies, in several industries, every word of this rang true.
CharlesW 7y ago
This is an enlightening article, but I think it's missing an implicit conclusion to accompany the explicit conclusion that "one is constantly fighting against history". History is absolutely one part of it, but the other part is that there must be a benevolent dictator (person or small group/council) with the authority and courage to make technology decisions. Without that role, bikeshedding can often drain valuable inertia — sometimes fatally — from the decision-making process.
- otterley 7y ago
  My experience is that the actual power wielded by a high level executive is usually inversely proportional to the size of the organization they manage. Such executives are typically more effective at guiding strategic direction, not prescribing tactics. Large, mature organizations have a lot of inertia, and that inertia is difficult to overcome except by perhaps the most charismatic individuals. I’ve rarely seen a leader dictate or bully their way to effective, positive change. Resistance is extremely likely for various reasons, and it’s not likely to be in the form of vocal mutiny. Instead, it typically appears in the form of excuses or lethargy. A waiting game is played, usually with success, until the musical chairs shift the management once again.
  - zaphirplane 7y ago
    My experience is in troubled companies executives move in then bring their people across
    - bjelkeman-again 7y ago
      And that starts another round of changes, which may be going in a different direction, again, as described as “history” in the OP.
  - jeffreygoesto 7y ago
    Sorry, I don't know if there's an english version, but this book was unusually non-bs and fun to read: https://www.amazon.de/Die-sieben-Irrt%C3%BCmer-Change-Managements/dp/359339135X https://www.amazon.de/Die-sieben-Irrt%C3%BCmer-Change-Manage... It is about how Schleuter turned around Audi in the 90s.
- cpeterso 7y ago
  The scale of the story was intriguing, but it feels like a murder mystery novel missing the last chapter. What did the company finally do to solve the problem? :) Tef's "How do you cut a monolith in half?" is relevant: https://programmingisterrible.com/post/162346490883/how-do-you-cut-a-monolith-in-half https://programmingisterrible.com/post/162346490883/how-do-y...
JamesBarney 7y ago
I have spent more time than I'm comfortable with over the last couple of years railing against microservices. But this seems like a classic case where microservices are needed. Each team can handle their own writing/validation and expose them to themselves and other services(including the global service). Of course you run into issues with transaction occurring across multiple databases but these problems are hard but solvable. > All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log. This sounds crazy. I don't know any large companies that have successfully implemented it. This is basically arguing for a giant central database across the entire company. Good luck getting the 300 people necessary into a room and agreeing on a schema. P.S. I don't know if he's using database as a shorthand for a service. If he is then you can ignore everything I've written.
- inimino 7y ago
  I'm not sure I'd call it microservices, since they could be monolithic inside each organization, but I had the same thought. Just spend the time to agree on a standardized API that everyone can support, and then the central API just proxies to the appropriate subsidiary. I wonder why they haven't done it that way.
  - ThePadawan 7y ago
    Personal experience from a past job: Coming from a clunky, overused but underdesigned monolithic structure, the work to standardize inter-team/inter-department communication is really hard. In general, the easiest thing is to overspecify those APIs and fight anyone who wants to simplify them. Complicated work looks good on a CV, after all. So now you have hard-to-implement, full-of-cruft API designs. At that point, teams realize that the easiest thing is to work around them. And off you go into splintered components which ignore the standard APIs as much as possible. Turns out that's much easier, and you deliver results with higher velocity! From far enough away, you can just see "more standard APIs => higher velocity", so obviously you keep doing that, right?
willart4food 7y ago
It's evolutionary in a Darwinian sense: they need to die so that their parts (bankruptcy/liquidation/sale) can be used by new and upcoming companies. There are exceptions like... IBM!
- hnzix 7y ago
  > There are exceptions like... IBM! Good grief, IBM are the Mongols. It all makes sense now.
  - pjc50 7y ago
    An extremely slow horde.
    - hnzix 7y ago
      Does this make Oracle the Ottoman Empire? I'm suddenly overwhelmed by the realpolitik similarities. Agile is Che Guevara?
      - osullivj 7y ago
        I used to work with an old IBMer from the MQ team. He'd also worked at DEC and the BBC before joining my team at Chase in the late 90s. He used to joke that it was no accident that IBM almost collapsed at the same time as the Soviet Union as they were very similar organisations!
      - keiru 7y ago
        >Agile is Che Guevara? Are you refering to the idealism, the guerilla dynamics, or how cool it looks on a t-shirt?
deleted 7y ago
[deleted]
doblecouponday 7y ago
What about an API gateway which acts as the public facing proxy for 20 other apis? What about a single API with 20 object relational mappers and a switch map?
adamzochowski 7y ago
The made up story sound similar to Hertz losing 32 million while hiring Accenture to do internal API rewrite in 2 years. https://www.theregister.co.uk/2019/04/23/hertz_accenture_lawsuit/ https://www.theregister.co.uk/2019/04/23/hertz_accenture_law...
- huffmsa 7y ago
  Hertz is what I've landed on. -- Female CEO -- 100+ years old -- rentals -- Has quietly bought up it's competition Only thing which doesn't line up is employee numbers. And they had Accenture do Angular app development.
  - msoad 7y ago
    How does gender of the CEO matter here? [edit] now I understood. sorry
    - huffmsa 7y ago
      Because we're trying to figure out which company the author is alluding to. And the author stipulated female CEO. Sometimes a cigar is just a cigar and not "zomg tech sexism!!1!", Sigmund.
    - rerx 7y ago
      I guess because of this quote FTA: > And what happens when the company gets a new CEO? What if she decides to break up the company again?
      - wizzairflyer 7y ago
        Isn't the "she" here refering to a potential new CEO? I assumed it was akin to the style of academic writing where you use she instead of he to refer to a person whose gender is unknown in order to fight gender stereotype or something similar (example: "the developer, when she sees fit, can[...]")
codeflo 7y ago
In some sense, departments in a large company develop an institutional resistance to IT centralization efforts as a defense against the inevitable next reorg. Or rather, departments that don’t have this resistance didn’t survive the last one. Though I wonder, if large companies are routinely this inefficient (which matches my limited experience as well), how do they survive? Naively put, why isn’t every large company killed by a startup next week?
- cosmie 7y ago
  > if large companies are routinely this inefficient (which matches my limited experience as well), how do they survive? Because every other large company is as equally inefficient. When it's par for the course, it's not seen as an issue (or even seen at all). Efficiency is also usually not the metric that gets optimized towards - other dimensions such as predictability, reliability, longevity, consistency, stability tend to take precedence. Take the loyalty program vendor mentioned in the article - sure it's limiting their flexibility for what they want to do now, but it still exists. So depending on your viewpoint, that was a pretty solid choice in vendor. Also, large companies have the scale to absorb the costs of their inefficiency with fairly minimal impact on their unit economics. And if a startup does pop up and gain traction with a much more efficient operational model, BigCo can write whatever check is necessary to gobble them up before it becomes a risk.
  - jacques_chester 7y ago
    > Because every other large company is as equally inefficient. I think there are commonalities in pathologies. After all: humans are humans and large enough groups of people will exhibit regression to the mean. But the exact details can vary so much that I am often reminded of Tolstoy's observation that: > Happy families are all alike; every unhappy family is unhappy in its own way.
- badpun 7y ago
  Large companies thrive in areas where effects of scale are necessary or at least very beneficial, which happen to be most of them.
- 7y ago
Aeolun 7y ago
I’m starting to be of the opinion that large companies are impossible to ‘rescue’. Unless you count sustained change over 20 years as a rescue. I swear, there are so many exceptions and special cases embedded in a large company that it’s impossible to replace any one product without also overhauling the rest, oh and overhauling customer expectations, because if they’re used to a static website that takes 10s for each page load, then whatever replaces it better takes at least 10s to load.
- julienreszka 7y ago
  What do you mean, why would they not want faster reload time?
  - EdwardDiego 7y ago
    Maybe they wrote an Excel spreadsheet that imports data by scraping the webpage using VBA (don't laugh, it can happen) and they relied on that 10 seconds as part of their VBA code?
  - Aeolun 7y ago
    My question exactly. Unfortunately getting the actual requirements after the 10 layers of indirection in the enterprise means you just hear the Product Owner say that the requirement is the page load should take 10 seconds. By that point any justification has been long lost.
  - rakoo 7y ago
    I've already had the case of clients complaining when the load time was greatly reduced, because they felt the only way it could have been so fast is that the system didn't do as much as before. Which was true in a way (the best way to be fast is to do as little as possible) but the results were correct so nothing was wrong.
- Shivetya 7y ago
  the fun I find is there are so many touch points that fixing or replacing gets bogged down in just identifying them. then you are up against each touch points budget and hours availability to adjust to changes you have set forth. one solution I have seem similar to the story's problem was to adapt an edi like solution, there was one group whose entire job was to transform data from group to group and each group in turn had to meet the interchange formats. that can be a political fire storm in itself because the other truth of big companies is that there are many fiefdoms within it and some have a lot of pull.
deboflo 7y ago
Excellent article. I’ve experienced a few of these issues at two vastly different companies (one had 60k employees, another only had 200). I implore developers to avoid working at companies with this class of problems. Work for a company that is strongly customer focused instead.
- Angostura 7y ago
  But you can have a company that is strongly customer-focussed that still faces this class of problem, surely? It would seem to me that solving these problems at a large company where there actually is trust and esprit de corps to get it solved could be tremendously satisfying.
kfk 7y ago
I think his paragraph on outsourcing and "core competencies" is spot on. I would take it a bit further, however, since another consequence of this is IT trying to buy every solution from an external software vendor (Oracle, SAP, MuleSoft, etc.). The word "custom" becomes a "bad word" and IT uses it whenever there is an effort to build software internally. Unfortunately very often external vendor solutions are not apt to the task in big corporations. There is also the issue that each vendor implements its own database and very quickly in IT you spend most of your time "integrating" solutions with each other, especially writes as the article author points out. The issue of trust is also real, but I noticed that if you build a strong team with a strong "internal brand" you can drive a lot of the execution and technical decision making from there (the world used for these teams in enterprise is "CoE's" but I mean something a bit more substantial by that than what you usually read around). The thing is that in reality your CEO doesn't care about your technical implementation details. This is a blessing and a curse. The blessing part is that if you offer a team with a brand and a vision that has collected some internal trust from multiple parties over a couple of years, you might be able to take your own technical decisions independently from IT or other departments around the world. As for the technical part of this, I am not an expert, but security, RBAC and friends are hard problems, I'd be surprised if you find a vendor that can do this for all your edge cases. I'd be for a fully internally built solution. But again, no expert.
- PaulAJ 7y ago
  "Core competencies" is a widely misunderstood term. Lots of people equate it to "business model", as in "we sell widgets so therefore selling widgets is our core competence". A thing is a core competence if, and only if: * It makes a difference to your customers. * It is difficult for your competitors to replicate. * It provides access to a wide range of markets. Janitorial services, as the OP says, do not tick any of these boxes (unless you are a restaurant or something, in which case it ticks the first). Black and Decker's core competence is not selling cheap hand tools, it is making small electric motors. Think about it. https://en.wikipedia.org/wiki/Core_competency https://en.wikipedia.org/wiki/Core_competency IT frequently ticks all three: if your IT goes down your customers may know about it before you do. Your IT is going to be difficult for competitors to replicate (as long as you haven't outsourced the whole thing), and you can use that IT in lots of different markets. However there are too many senior managers who have heard the words "core competence", not read the article, and assumed that since they are not running an IT company it follows that IT is not a core competence.
aryehof 7y ago
The (latest) CEO calls to "unify" (restructure - yet again) the company. That vague call results in the age-old purchase of a "technology" to solve the problem from a vendor - MuleSoft. An "API" is the answer they are told they need - easy. Millions of dollars later after much time has past, the project is of course in trouble. The author is asked if he knows anything about APIs. He says sure, they're easy. Just expose the database through an API. How hard can it be. My prediction (guarantee) is that a lot more time and money will go down the drain before this API effort is finally abandoned. The issue is the chain of naive and poor decisions being made, which are exacerbated by the advice of the author.
- Angostura 7y ago
  > which are exacerbated by the advice of the author. Really? what advice is being given here that would exacerbate anything? From what I see, the author is carefully enumerating the problems which need to be carefully addressed before this kind of change can be made. He is deliberately holding up the 'You just need an API, how hard can it be?' perspective as a naive one and looking at why it isn't that simple. I found this an interesting and useful argument, particular the element about trust and loss of control
  - C1sc0cat 7y ago
    From experience working on these sorts of sites - No its not - how you structure a large world wide care hire site is a known thing now its not rocket science. In the past I have worked on one of the Big car hire sites and in my opinion Accenture and Hertz where just incompetent. Probably Non Culpable Incompetence for Hertz, Accenture not so much.
- huffmsa 7y ago
  Cases like these are either solved by giving someone absolute power to steer the project, resulting in squished toes and hurt feelings and potentially becoming a HBS postmortem, or accepting that it's going to take a while to reconcile all of the different fiefdoms. There seems to be a lack of translation to "what should actually be built." Management wants a unified API, but my first (and probably incorrect) idea would be to develop an ORM + set of functional libraries which can be implemented independently by each of the involved parties. That has its own risks, but getting everyone speaking the same language is a good place to start.
rkagerer 7y ago
Inertia
deleted 7y ago
[deleted]
ptah 7y ago
it sounds like centralisation is the problem, why not build something generic and then have separate instances for each legal entity
moreentropy 7y ago
Welcome to enterprise IT. You'll receive your badge in two weeks eventually and your login won't work for another month.
- tonyedgecombe 7y ago
  It’s soul destroying work, I’m starting to wonder if there is something wrong with people who appear to thrive in these companies.
  - brixon 7y ago
    Global changes like this are near impossible, but most of the work is for the local internal customer which is very possible and rewarding. Large projects are normally won or lost on the strength of the upper management, it is normally not a technical issue that fails large projects in large companies.
hackerbabz 7y ago
Isn't the issue with the "primitive" customer loyalty company a classic case for abstraction? They should build an ORM like internal API to deal with the loyalty program. It can send emails or even order a person to make a phone call if necessary. Whatever "primitive" means the "Invented-here" API can abstract that functionality, so that the internal API devs can access and update the data from the external company.
deleted 7y ago
[deleted]
simongray 7y ago
Oooh, this is by the same guy that wrote my favourite anti-OOP rant: http://www.smashcompany.com/technology/object-oriented-programming-is-an-expensive-disaster-which-must-end http://www.smashcompany.com/technology/object-oriented-progr... It's worth reading, although your feelings may get hurt if you're not already sold on functional programming. He's not very diplomatic.
geertj 7y ago
A ‘tech first’ approach (create an API) will not make this company agile. The question that the CEO of SuperRentalCorp needs to answer is ‘do we want to become a tech company?’ If yes, then start a multi year transformation starting with defining tech career ladders, a strategy how to train internals and hire externals, how to separate with those intervals that are unable to follow. Be clear and offer an opportunity for those that want to stay and be generous for those that can’t. If done correctly you will have the necessary basis for beginning a tech transform in a couple of years. If the CEO does not want to be a tech company then I’d investigate options for spinning out a service company to serve the rental market, ideally in collaboration with some competitors.
jacques_chester 7y ago
I work for Pivotal and I feel I have some relevant experiences (from which the following personal opinions are derived). I started in Labs, our consulting division, known for being vociferously about XP, Lean product management and User-Centred Design. These days I work in R&D. The latter has been very educational, as we've been involved in building larger and larger systems, with lots of "oh that's what our customers meant when they talked about problem-of-scale XYZ". I know that my own temptation as a Labs Pivot was to imply that customers just weren't agile-ing hard enough; R&D has kicked a bit of that stuffing out of me. As Brooks argued in No Silver Bullet, there is essential complexity and accidental complexity. Enterprises deal with amounts of both that are hard to comprehend. You get some fascinating problems and pathologies at Enterprise scale. As the article implies, nobody ever really intends to create them, they emerge from the system's evolution. I think the core logic of outsourcing is sound, but it's like many simple ideas: simple, but not easy. It's actually a restatement of economics 101 -- comparative advantage and gains from trade. Even if you are strictly better at doing everything yourself, it still makes sense to trade with others. At Pivotal we often discuss this as "the value line", the point in the stack where, we argue, it is more effective to delegate your technology problems to us (or -- this is how I feel personally -- to a comparable vendor, like Red Hat, over pure DIY) and to turn your own technology efforts towards the things that add value for your customers. Even if you could do a better job than any of the vendors, it might not make sense for you to do so. That value line isn't static, of course, and it's differently located for each customer. And it remains necessary to retain enough inhouse expertise to sensibly assess vendor solutions. Enterprise software and software development is a complex "experience good". This isn't limited to software, it's true of many complex purchases -- I have a book in my collection called Industrial Megaprojects which contains a fascinating parade of such examples. All that said, transformation is possible in my experience. The key is, as the article describes, that it takes a long time, a lot of work and a lot of money. It most of all requires enough time for cultural norms to resettle and for trust to solidify. I've worked with companies operated on a fear basis and basically, the best we could do was to create insulated bubbles of agility. We've worked with other companies which have more or less transformed themselves once we brought a spark. Sometimes we will be the true vanguard of a revolution, sometimes we will be the fad-of-the-week that the hardened veterans will shrug and mouth the slogans for while continuing with the actual way things get done. Sometimes, because many of our customers are so vast, we will be heroes to one division and villains in another. The best that I and my peers can do is to help each client as honestly, empathetically and fully as we can to transform themselves. But, as the article points out, that is hard.
JoeAltmaier 7y ago
Ah, my sister endured all this sort of thing during 20 years as a VP in corporate America. She successfully deployed new data systems to 120 plants in 50 regions in one year. Didn't cost $25M. Her method? Ruthlessly purge the region of the old data system and install the new (web-based API to a central, new data system). Investigate every regional difference and consolidate into one model. Before deployment day, get all the regional Directors in one room and tell them it was going to happen. Tell them there was no going back, no push-back would be permitted, and have the CEO in the room to confirm this. It all went well, with her team of a dozen professionals that did a surgical strike to each region. IT equipment installers, trainers. Even one lady who's job was to ferret out the hidden PC somebody was squirrelling away in a closet (there was always that person) and rip it out, take it to her car and lock it in the trunk. All went well, except for one region. That regional Director had missed the big day, and tried (vainly) to push-back for all the familiar reasons: special circumstance, retraining was going to keep them down for too long, and on and on. 100 years of data systems gone in a year (including paper systems) and replaced with a web API.
- keiru 7y ago
  I was reading the article thinking that I would take an approach just like that. I don't think there is any other way to ensure success.
  - brazzy 7y ago
    It requires you to have complete and unwavering backing from top management even when some of the guys whose toes you're stepping on haven been golf buddies with said top management for 20 years. Sounds like an unrealistic expectation.
    - jandrese 7y ago
      It also depends on you having a crack IT team who can get it right the first time. No months of stuff crashing or having a horrible interface that impedes work or being intolerably slow. You have to be confident enough to push back against the exec who says "oh, we'll just hire $massive_consultancy_firm to do it" and takes years to roll out a half baked hackjob on top of their stupidly expensive CMS.
motohagiography 7y ago
This blog post is a better case study than any HBR or consulting firm study I've read because the author has a grasp of each of the technology, governance, and economics of the problem. Purely economic or political models of organizations assume you can just reform one to align with incentives, but tech is a physical limitation, it's the new inescapable geography. The companies and institutions I've encountered all essentially say, "We want success but without change," and then wonder why their initiatives fail. The point about "Agile" being a euphemism for "trust," is huge. Startups can be farcically naive (or cynical to the point of evil) about how trust in organizations works. C&C enterprise structures don't scale, and they become immensely vulnerable to challengers as a result. This article reminded me of the opportunity that large companies create, like stored potential energy in the form of opportunity costs. It's like a kingdom spread so widely that a small raiding army can feed itself on what these companies left undefended while planning campaigns. How much of this article is true for the rest of the Fortune 500?
whatshisface 7y ago
>To a large extent “be agile” is almost synonymous with “trust each other.” If you’re wondering why large companies have trouble being agile, it is partly because it is impossible for 11,000 people to trust each other the way 5 people can. I wish the author had explained this more. What is it about agile that requires more trust than anything else? Can't you write lies in any format equally easy?
- hibbelig 7y ago
  Well, there is agile as described in the Agile Manifesto: https://agilemanifesto.org/ https://agilemanifesto.org/ And then there is Half-Arsed Agile: https://www.halfarsedagilemanifesto.org/ https://www.halfarsedagilemanifesto.org/ The author meant the former. Writing lies seems to point to the latter. Writing lies is not "valuing individuals and interactions".
soup10 7y ago
Large old software companies have hilarious deep layers of abstraction and tooling on top of legacy code from people that have long left the company. Usually getting anything done requires gathering word of mouth arcana from multiple people.
- marcoonroad 7y ago
  Yep, some of these people leaving the company are developers who suffered from burnout crisis inside the 'Big Company'. :p I'm one of these sort of professional developers leaving 'Big Companies' due burnout. I think that 'Big Companies' need to deal with Ethics before trying to deal with Agile/Scrum. xD
softwaredoug 7y ago
The article I think misses reasons for keeping the different business units separate (assuming they are indeed BUs). They can act as the smaller “agile” companies in their own markets that the author seems to say can get a lot more done. Who cares if there’s a lot of duplication. There’s a huge human reason to keep these separate: autonomy. Nothing sucks more than being stuck in org bureaucracy for years never solving a customer problem, watching the smaller poorly funded competition do better. You’ll never retain talent doing that, which frankly is the real issue large companies struggle with. And you’ll never satisfy customers when reuse and not quickly solving customer problems is the priority. Consolidation and “reuse” is often the problem, not the solution.
RocketSyntax 7y ago
Peter Drucker: the largest drain on corporate productivity is recommunication. Short term goals. Narrow focuses of executive team. Lack of technical executives. Brain drain.
pacala 7y ago
One piece caught my eye: > In terms of the best integration architecture, what seems to me the only long-term solution is something like the unified log architecture that Jay Kreps wrote about back in 2013. All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log. Then, from the linked Jon Kreps post, 2013 [0]: > In order to allow horizontal scaling we chop up our log into partitions. Each partition is a totally ordered log, but there is no global ordering between partitions (other than perhaps some wall-clock time you might include in your messages). The assignment of the messages to a particular partition is controllable by the writer, with most users choosing to partition by some kind of key (e.g. user id). Partitioning allows log appends to occur without co-ordination between shards and allows the throughput of the system to scale linearly with the Kafka cluster size. In practice, wouldn't the end "centralized log" be essentially a collection of the logs from each original/logical database, now managed by a specialized log team? To read the log, the interested consumer team would have to create the appropriate read-only replica of the original database, using the respective database technology / binaries / schemas. As such, isn't the overall system better described as "database replica as a service", as opposed to the finer grained implication of the "unified log as a service" of the article / Jon Kreps post? I'm trying to figure out if the implication of such decisions are organizational, "who's responsible for the logs, who's responsible for the replicas", vs. architectural, "where does data bit X go, how does it maintain consistency with data bit Y, how do we read its current state?". [0] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
otakucode 7y ago
Interesting article, and it points out real problems that large organizations face. It made me think of the issues I've seen dealing with government contracts in the US. Policies can create perverse incentives and have unintended consequences regardless of their intention, and when dealing with large organizations those sorts of things have to be considered as they end up steering the ship. Near the end of the article the author mentions that 5 people can look each other in the eye and trust each other, while 11,000 can't. This suggested a question to me: Where does that stop? How many people do you have to have before a 'local culture' isn't enough and trust breaks down? I've read things in the past about Dunbar's Number which suggests this number is around 110. That seems to be about the number of people that individuals can 'keep in mind'. I have often wondered whether organizations might benefit from taking that limit into mind and structuring things so that no level or group would be permitted to grow beyond that. To grow larger, build a hierarchy where groups of about 100 choose a representative whose task is to know the concerns of everyone in the group and represent those concerns when discussing with reps from other groups, etc.

208 points by lkrubner 7y ago

aj7 7y ago

MBAs and other “professional managers.” Persons in a position of power that have not done any company business or tasks, either never, or for more than 5 years.

rossdavidh 7y ago

Having worked at and for both large and small companies, in several industries, every word of this rang true.

CharlesW 7y ago

This is an enlightening article, but I think it's missing an implicit conclusion to accompany the explicit conclusion that "one is constantly fighting against history". History is absolutely one part of it, but the other part is that there must be a benevolent dictator (person or small group/council) with the authority and courage to make technology decisions. Without that role, bikeshedding can often drain valuable inertia — sometimes fatally — from the decision-making process.

otterley 7y ago
My experience is that the actual power wielded by a high level executive is usually inversely proportional to the size of the organization they manage. Such executives are typically more effective at guiding strategic direction, not prescribing tactics. Large, mature organizations have a lot of inertia, and that inertia is difficult to overcome except by perhaps the most charismatic individuals. I’ve rarely seen a leader dictate or bully their way to effective, positive change. Resistance is extremely likely for various reasons, and it’s not likely to be in the form of vocal mutiny. Instead, it typically appears in the form of excuses or lethargy. A waiting game is played, usually with success, until the musical chairs shift the management once again.
- zaphirplane 7y ago
  My experience is in troubled companies executives move in then bring their people across
  - bjelkeman-again 7y ago
    And that starts another round of changes, which may be going in a different direction, again, as described as “history” in the OP.
- jeffreygoesto 7y ago
  Sorry, I don't know if there's an english version, but this book was unusually non-bs and fun to read: https://www.amazon.de/Die-sieben-Irrt%C3%BCmer-Change-Managements/dp/359339135X https://www.amazon.de/Die-sieben-Irrt%C3%BCmer-Change-Manage... It is about how Schleuter turned around Audi in the 90s.
cpeterso 7y ago
The scale of the story was intriguing, but it feels like a murder mystery novel missing the last chapter. What did the company finally do to solve the problem? :) Tef's "How do you cut a monolith in half?" is relevant: https://programmingisterrible.com/post/162346490883/how-do-you-cut-a-monolith-in-half https://programmingisterrible.com/post/162346490883/how-do-y...

JamesBarney 7y ago

I have spent more time than I'm comfortable with over the last couple of years railing against microservices. But this seems like a classic case where microservices are needed. Each team can handle their own writing/validation and expose them to themselves and other services(including the global service). Of course you run into issues with transaction occurring across multiple databases but these problems are hard but solvable. > All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log. This sounds crazy. I don't know any large companies that have successfully implemented it. This is basically arguing for a giant central database across the entire company. Good luck getting the 300 people necessary into a room and agreeing on a schema. P.S. I don't know if he's using database as a shorthand for a service. If he is then you can ignore everything I've written.

inimino 7y ago
I'm not sure I'd call it microservices, since they could be monolithic inside each organization, but I had the same thought. Just spend the time to agree on a standardized API that everyone can support, and then the central API just proxies to the appropriate subsidiary. I wonder why they haven't done it that way.
- ThePadawan 7y ago
  Personal experience from a past job: Coming from a clunky, overused but underdesigned monolithic structure, the work to standardize inter-team/inter-department communication is really hard. In general, the easiest thing is to overspecify those APIs and fight anyone who wants to simplify them. Complicated work looks good on a CV, after all. So now you have hard-to-implement, full-of-cruft API designs. At that point, teams realize that the easiest thing is to work around them. And off you go into splintered components which ignore the standard APIs as much as possible. Turns out that's much easier, and you deliver results with higher velocity! From far enough away, you can just see "more standard APIs => higher velocity", so obviously you keep doing that, right?

willart4food 7y ago

It's evolutionary in a Darwinian sense: they need to die so that their parts (bankruptcy/liquidation/sale) can be used by new and upcoming companies. There are exceptions like... IBM!

hnzix 7y ago
> There are exceptions like... IBM! Good grief, IBM are the Mongols. It all makes sense now.
- pjc50 7y ago
  An extremely slow horde.
  - hnzix 7y ago
    Does this make Oracle the Ottoman Empire? I'm suddenly overwhelmed by the realpolitik similarities. Agile is Che Guevara?
    - osullivj 7y ago
      I used to work with an old IBMer from the MQ team. He'd also worked at DEC and the BBC before joining my team at Chase in the late 90s. He used to joke that it was no accident that IBM almost collapsed at the same time as the Soviet Union as they were very similar organisations!
    - keiru 7y ago
      >Agile is Che Guevara? Are you refering to the idealism, the guerilla dynamics, or how cool it looks on a t-shirt?

deleted 7y ago

[deleted]

doblecouponday 7y ago

What about an API gateway which acts as the public facing proxy for 20 other apis? What about a single API with 20 object relational mappers and a switch map?

adamzochowski 7y ago

The made up story sound similar to Hertz losing 32 million while hiring Accenture to do internal API rewrite in 2 years. https://www.theregister.co.uk/2019/04/23/hertz_accenture_lawsuit/ https://www.theregister.co.uk/2019/04/23/hertz_accenture_law...

huffmsa 7y ago
Hertz is what I've landed on. -- Female CEO -- 100+ years old -- rentals -- Has quietly bought up it's competition Only thing which doesn't line up is employee numbers. And they had Accenture do Angular app development.
- msoad 7y ago
  How does gender of the CEO matter here? [edit] now I understood. sorry
  - huffmsa 7y ago
    Because we're trying to figure out which company the author is alluding to. And the author stipulated female CEO. Sometimes a cigar is just a cigar and not "zomg tech sexism!!1!", Sigmund.
  - rerx 7y ago
    I guess because of this quote FTA: > And what happens when the company gets a new CEO? What if she decides to break up the company again?
    - wizzairflyer 7y ago
      Isn't the "she" here refering to a potential new CEO? I assumed it was akin to the style of academic writing where you use she instead of he to refer to a person whose gender is unknown in order to fight gender stereotype or something similar (example: "the developer, when she sees fit, can[...]")

codeflo 7y ago

In some sense, departments in a large company develop an institutional resistance to IT centralization efforts as a defense against the inevitable next reorg. Or rather, departments that don’t have this resistance didn’t survive the last one. Though I wonder, if large companies are routinely this inefficient (which matches my limited experience as well), how do they survive? Naively put, why isn’t every large company killed by a startup next week?

cosmie 7y ago
> if large companies are routinely this inefficient (which matches my limited experience as well), how do they survive? Because every other large company is as equally inefficient. When it's par for the course, it's not seen as an issue (or even seen at all). Efficiency is also usually not the metric that gets optimized towards - other dimensions such as predictability, reliability, longevity, consistency, stability tend to take precedence. Take the loyalty program vendor mentioned in the article - sure it's limiting their flexibility for what they want to do now, but it still exists. So depending on your viewpoint, that was a pretty solid choice in vendor. Also, large companies have the scale to absorb the costs of their inefficiency with fairly minimal impact on their unit economics. And if a startup does pop up and gain traction with a much more efficient operational model, BigCo can write whatever check is necessary to gobble them up before it becomes a risk.
- jacques_chester 7y ago
  > Because every other large company is as equally inefficient. I think there are commonalities in pathologies. After all: humans are humans and large enough groups of people will exhibit regression to the mean. But the exact details can vary so much that I am often reminded of Tolstoy's observation that: > Happy families are all alike; every unhappy family is unhappy in its own way.
badpun 7y ago
Large companies thrive in areas where effects of scale are necessary or at least very beneficial, which happen to be most of them.
7y ago

Aeolun 7y ago

I’m starting to be of the opinion that large companies are impossible to ‘rescue’. Unless you count sustained change over 20 years as a rescue. I swear, there are so many exceptions and special cases embedded in a large company that it’s impossible to replace any one product without also overhauling the rest, oh and overhauling customer expectations, because if they’re used to a static website that takes 10s for each page load, then whatever replaces it better takes at least 10s to load.

julienreszka 7y ago
What do you mean, why would they not want faster reload time?
- EdwardDiego 7y ago
  Maybe they wrote an Excel spreadsheet that imports data by scraping the webpage using VBA (don't laugh, it can happen) and they relied on that 10 seconds as part of their VBA code?
- Aeolun 7y ago
  My question exactly. Unfortunately getting the actual requirements after the 10 layers of indirection in the enterprise means you just hear the Product Owner say that the requirement is the page load should take 10 seconds. By that point any justification has been long lost.
- rakoo 7y ago
  I've already had the case of clients complaining when the load time was greatly reduced, because they felt the only way it could have been so fast is that the system didn't do as much as before. Which was true in a way (the best way to be fast is to do as little as possible) but the results were correct so nothing was wrong.
Shivetya 7y ago
the fun I find is there are so many touch points that fixing or replacing gets bogged down in just identifying them. then you are up against each touch points budget and hours availability to adjust to changes you have set forth. one solution I have seem similar to the story's problem was to adapt an edi like solution, there was one group whose entire job was to transform data from group to group and each group in turn had to meet the interchange formats. that can be a political fire storm in itself because the other truth of big companies is that there are many fiefdoms within it and some have a lot of pull.

deboflo 7y ago

Excellent article. I’ve experienced a few of these issues at two vastly different companies (one had 60k employees, another only had 200). I implore developers to avoid working at companies with this class of problems. Work for a company that is strongly customer focused instead.

Angostura 7y ago
But you can have a company that is strongly customer-focussed that still faces this class of problem, surely? It would seem to me that solving these problems at a large company where there actually is trust and esprit de corps to get it solved could be tremendously satisfying.

kfk 7y ago

I think his paragraph on outsourcing and "core competencies" is spot on. I would take it a bit further, however, since another consequence of this is IT trying to buy every solution from an external software vendor (Oracle, SAP, MuleSoft, etc.). The word "custom" becomes a "bad word" and IT uses it whenever there is an effort to build software internally. Unfortunately very often external vendor solutions are not apt to the task in big corporations. There is also the issue that each vendor implements its own database and very quickly in IT you spend most of your time "integrating" solutions with each other, especially writes as the article author points out. The issue of trust is also real, but I noticed that if you build a strong team with a strong "internal brand" you can drive a lot of the execution and technical decision making from there (the world used for these teams in enterprise is "CoE's" but I mean something a bit more substantial by that than what you usually read around). The thing is that in reality your CEO doesn't care about your technical implementation details. This is a blessing and a curse. The blessing part is that if you offer a team with a brand and a vision that has collected some internal trust from multiple parties over a couple of years, you might be able to take your own technical decisions independently from IT or other departments around the world. As for the technical part of this, I am not an expert, but security, RBAC and friends are hard problems, I'd be surprised if you find a vendor that can do this for all your edge cases. I'd be for a fully internally built solution. But again, no expert.

PaulAJ 7y ago
"Core competencies" is a widely misunderstood term. Lots of people equate it to "business model", as in "we sell widgets so therefore selling widgets is our core competence". A thing is a core competence if, and only if: * It makes a difference to your customers. * It is difficult for your competitors to replicate. * It provides access to a wide range of markets. Janitorial services, as the OP says, do not tick any of these boxes (unless you are a restaurant or something, in which case it ticks the first). Black and Decker's core competence is not selling cheap hand tools, it is making small electric motors. Think about it. https://en.wikipedia.org/wiki/Core_competency https://en.wikipedia.org/wiki/Core_competency IT frequently ticks all three: if your IT goes down your customers may know about it before you do. Your IT is going to be difficult for competitors to replicate (as long as you haven't outsourced the whole thing), and you can use that IT in lots of different markets. However there are too many senior managers who have heard the words "core competence", not read the article, and assumed that since they are not running an IT company it follows that IT is not a core competence.

aryehof 7y ago

The (latest) CEO calls to "unify" (restructure - yet again) the company. That vague call results in the age-old purchase of a "technology" to solve the problem from a vendor - MuleSoft. An "API" is the answer they are told they need - easy. Millions of dollars later after much time has past, the project is of course in trouble. The author is asked if he knows anything about APIs. He says sure, they're easy. Just expose the database through an API. How hard can it be. My prediction (guarantee) is that a lot more time and money will go down the drain before this API effort is finally abandoned. The issue is the chain of naive and poor decisions being made, which are exacerbated by the advice of the author.

Angostura 7y ago
> which are exacerbated by the advice of the author. Really? what advice is being given here that would exacerbate anything? From what I see, the author is carefully enumerating the problems which need to be carefully addressed before this kind of change can be made. He is deliberately holding up the 'You just need an API, how hard can it be?' perspective as a naive one and looking at why it isn't that simple. I found this an interesting and useful argument, particular the element about trust and loss of control
- C1sc0cat 7y ago
  From experience working on these sorts of sites - No its not - how you structure a large world wide care hire site is a known thing now its not rocket science. In the past I have worked on one of the Big car hire sites and in my opinion Accenture and Hertz where just incompetent. Probably Non Culpable Incompetence for Hertz, Accenture not so much.
huffmsa 7y ago
Cases like these are either solved by giving someone absolute power to steer the project, resulting in squished toes and hurt feelings and potentially becoming a HBS postmortem, or accepting that it's going to take a while to reconcile all of the different fiefdoms. There seems to be a lack of translation to "what should actually be built." Management wants a unified API, but my first (and probably incorrect) idea would be to develop an ORM + set of functional libraries which can be implemented independently by each of the involved parties. That has its own risks, but getting everyone speaking the same language is a good place to start.

rkagerer 7y ago

Inertia

deleted 7y ago

[deleted]

ptah 7y ago

it sounds like centralisation is the problem, why not build something generic and then have separate instances for each legal entity

moreentropy 7y ago

Welcome to enterprise IT. You'll receive your badge in two weeks eventually and your login won't work for another month.

tonyedgecombe 7y ago
It’s soul destroying work, I’m starting to wonder if there is something wrong with people who appear to thrive in these companies.
- brixon 7y ago
  Global changes like this are near impossible, but most of the work is for the local internal customer which is very possible and rewarding. Large projects are normally won or lost on the strength of the upper management, it is normally not a technical issue that fails large projects in large companies.

hackerbabz 7y ago

Isn't the issue with the "primitive" customer loyalty company a classic case for abstraction? They should build an ORM like internal API to deal with the loyalty program. It can send emails or even order a person to make a phone call if necessary. Whatever "primitive" means the "Invented-here" API can abstract that functionality, so that the internal API devs can access and update the data from the external company.

deleted 7y ago

[deleted]

simongray 7y ago

Oooh, this is by the same guy that wrote my favourite anti-OOP rant: http://www.smashcompany.com/technology/object-oriented-programming-is-an-expensive-disaster-which-must-end http://www.smashcompany.com/technology/object-oriented-progr... It's worth reading, although your feelings may get hurt if you're not already sold on functional programming. He's not very diplomatic.

geertj 7y ago

A ‘tech first’ approach (create an API) will not make this company agile. The question that the CEO of SuperRentalCorp needs to answer is ‘do we want to become a tech company?’ If yes, then start a multi year transformation starting with defining tech career ladders, a strategy how to train internals and hire externals, how to separate with those intervals that are unable to follow. Be clear and offer an opportunity for those that want to stay and be generous for those that can’t. If done correctly you will have the necessary basis for beginning a tech transform in a couple of years. If the CEO does not want to be a tech company then I’d investigate options for spinning out a service company to serve the rental market, ideally in collaboration with some competitors.

jacques_chester 7y ago

I work for Pivotal and I feel I have some relevant experiences (from which the following personal opinions are derived). I started in Labs, our consulting division, known for being vociferously about XP, Lean product management and User-Centred Design. These days I work in R&D. The latter has been very educational, as we've been involved in building larger and larger systems, with lots of "oh that's what our customers meant when they talked about problem-of-scale XYZ". I know that my own temptation as a Labs Pivot was to imply that customers just weren't agile-ing hard enough; R&D has kicked a bit of that stuffing out of me. As Brooks argued in No Silver Bullet, there is essential complexity and accidental complexity. Enterprises deal with amounts of both that are hard to comprehend. You get some fascinating problems and pathologies at Enterprise scale. As the article implies, nobody ever really intends to create them, they emerge from the system's evolution. I think the core logic of outsourcing is sound, but it's like many simple ideas: simple, but not easy. It's actually a restatement of economics 101 -- comparative advantage and gains from trade. Even if you are strictly better at doing everything yourself, it still makes sense to trade with others. At Pivotal we often discuss this as "the value line", the point in the stack where, we argue, it is more effective to delegate your technology problems to us (or -- this is how I feel personally -- to a comparable vendor, like Red Hat, over pure DIY) and to turn your own technology efforts towards the things that add value for your customers. Even if you could do a better job than any of the vendors, it might not make sense for you to do so. That value line isn't static, of course, and it's differently located for each customer. And it remains necessary to retain enough inhouse expertise to sensibly assess vendor solutions. Enterprise software and software development is a complex "experience good". This isn't limited to software, it's true of many complex purchases -- I have a book in my collection called Industrial Megaprojects which contains a fascinating parade of such examples. All that said, transformation is possible in my experience. The key is, as the article describes, that it takes a long time, a lot of work and a lot of money. It most of all requires enough time for cultural norms to resettle and for trust to solidify. I've worked with companies operated on a fear basis and basically, the best we could do was to create insulated bubbles of agility. We've worked with other companies which have more or less transformed themselves once we brought a spark. Sometimes we will be the true vanguard of a revolution, sometimes we will be the fad-of-the-week that the hardened veterans will shrug and mouth the slogans for while continuing with the actual way things get done. Sometimes, because many of our customers are so vast, we will be heroes to one division and villains in another. The best that I and my peers can do is to help each client as honestly, empathetically and fully as we can to transform themselves. But, as the article points out, that is hard.

JoeAltmaier 7y ago

Ah, my sister endured all this sort of thing during 20 years as a VP in corporate America. She successfully deployed new data systems to 120 plants in 50 regions in one year. Didn't cost $25M. Her method? Ruthlessly purge the region of the old data system and install the new (web-based API to a central, new data system). Investigate every regional difference and consolidate into one model. Before deployment day, get all the regional Directors in one room and tell them it was going to happen. Tell them there was no going back, no push-back would be permitted, and have the CEO in the room to confirm this. It all went well, with her team of a dozen professionals that did a surgical strike to each region. IT equipment installers, trainers. Even one lady who's job was to ferret out the hidden PC somebody was squirrelling away in a closet (there was always that person) and rip it out, take it to her car and lock it in the trunk. All went well, except for one region. That regional Director had missed the big day, and tried (vainly) to push-back for all the familiar reasons: special circumstance, retraining was going to keep them down for too long, and on and on. 100 years of data systems gone in a year (including paper systems) and replaced with a web API.

keiru 7y ago
I was reading the article thinking that I would take an approach just like that. I don't think there is any other way to ensure success.
- brazzy 7y ago
  It requires you to have complete and unwavering backing from top management even when some of the guys whose toes you're stepping on haven been golf buddies with said top management for 20 years. Sounds like an unrealistic expectation.
  - jandrese 7y ago
    It also depends on you having a crack IT team who can get it right the first time. No months of stuff crashing or having a horrible interface that impedes work or being intolerably slow. You have to be confident enough to push back against the exec who says "oh, we'll just hire $massive_consultancy_firm to do it" and takes years to roll out a half baked hackjob on top of their stupidly expensive CMS.

motohagiography 7y ago

This blog post is a better case study than any HBR or consulting firm study I've read because the author has a grasp of each of the technology, governance, and economics of the problem. Purely economic or political models of organizations assume you can just reform one to align with incentives, but tech is a physical limitation, it's the new inescapable geography. The companies and institutions I've encountered all essentially say, "We want success but without change," and then wonder why their initiatives fail. The point about "Agile" being a euphemism for "trust," is huge. Startups can be farcically naive (or cynical to the point of evil) about how trust in organizations works. C&C enterprise structures don't scale, and they become immensely vulnerable to challengers as a result. This article reminded me of the opportunity that large companies create, like stored potential energy in the form of opportunity costs. It's like a kingdom spread so widely that a small raiding army can feed itself on what these companies left undefended while planning campaigns. How much of this article is true for the rest of the Fortune 500?

whatshisface 7y ago

>To a large extent “be agile” is almost synonymous with “trust each other.” If you’re wondering why large companies have trouble being agile, it is partly because it is impossible for 11,000 people to trust each other the way 5 people can. I wish the author had explained this more. What is it about agile that requires more trust than anything else? Can't you write lies in any format equally easy?

hibbelig 7y ago
Well, there is agile as described in the Agile Manifesto: https://agilemanifesto.org/ https://agilemanifesto.org/ And then there is Half-Arsed Agile: https://www.halfarsedagilemanifesto.org/ https://www.halfarsedagilemanifesto.org/ The author meant the former. Writing lies seems to point to the latter. Writing lies is not "valuing individuals and interactions".

soup10 7y ago

Large old software companies have hilarious deep layers of abstraction and tooling on top of legacy code from people that have long left the company. Usually getting anything done requires gathering word of mouth arcana from multiple people.

marcoonroad 7y ago
Yep, some of these people leaving the company are developers who suffered from burnout crisis inside the 'Big Company'. :p I'm one of these sort of professional developers leaving 'Big Companies' due burnout. I think that 'Big Companies' need to deal with Ethics before trying to deal with Agile/Scrum. xD

softwaredoug 7y ago

The article I think misses reasons for keeping the different business units separate (assuming they are indeed BUs). They can act as the smaller “agile” companies in their own markets that the author seems to say can get a lot more done. Who cares if there’s a lot of duplication. There’s a huge human reason to keep these separate: autonomy. Nothing sucks more than being stuck in org bureaucracy for years never solving a customer problem, watching the smaller poorly funded competition do better. You’ll never retain talent doing that, which frankly is the real issue large companies struggle with. And you’ll never satisfy customers when reuse and not quickly solving customer problems is the priority. Consolidation and “reuse” is often the problem, not the solution.

RocketSyntax 7y ago

Peter Drucker: the largest drain on corporate productivity is recommunication. Short term goals. Narrow focuses of executive team. Lack of technical executives. Brain drain.

pacala 7y ago

One piece caught my eye: > In terms of the best integration architecture, what seems to me the only long-term solution is something like the unified log architecture that Jay Kreps wrote about back in 2013. All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log. Then, from the linked Jon Kreps post, 2013 [0]: > In order to allow horizontal scaling we chop up our log into partitions. Each partition is a totally ordered log, but there is no global ordering between partitions (other than perhaps some wall-clock time you might include in your messages). The assignment of the messages to a particular partition is controllable by the writer, with most users choosing to partition by some kind of key (e.g. user id). Partitioning allows log appends to occur without co-ordination between shards and allows the throughput of the system to scale linearly with the Kafka cluster size. In practice, wouldn't the end "centralized log" be essentially a collection of the logs from each original/logical database, now managed by a specialized log team? To read the log, the interested consumer team would have to create the appropriate read-only replica of the original database, using the respective database technology / binaries / schemas. As such, isn't the overall system better described as "database replica as a service", as opposed to the finer grained implication of the "unified log as a service" of the article / Jon Kreps post? I'm trying to figure out if the implication of such decisions are organizational, "who's responsible for the logs, who's responsible for the replicas", vs. architectural, "where does data bit X go, how does it maintain consistency with data bit Y, how do we read its current state?". [0] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...

otakucode 7y ago

Interesting article, and it points out real problems that large organizations face. It made me think of the issues I've seen dealing with government contracts in the US. Policies can create perverse incentives and have unintended consequences regardless of their intention, and when dealing with large organizations those sorts of things have to be considered as they end up steering the ship. Near the end of the article the author mentions that 5 people can look each other in the eye and trust each other, while 11,000 can't. This suggested a question to me: Where does that stop? How many people do you have to have before a 'local culture' isn't enough and trust breaks down? I've read things in the past about Dunbar's Number which suggests this number is around 110. That seems to be about the number of people that individuals can 'keep in mind'. I have often wondered whether organizations might benefit from taking that limit into mind and structuring things so that no level or group would be permitted to grow beyond that. To grow larger, build a hierarchy where groups of about 100 choose a representative whose task is to know the concerns of everyone in the group and represent those concerns when discussing with reps from other groups, etc.