3 ms·

I think as systems get more complex and bogged down, it's harder to innovate. As innovation is stifled by build problems, dependency problems, political problem

I think as systems get more complex and bogged down, it's harder to innovate. As innovation is stifled by build problems, dependency problems, political problems, operational problems, the amount of code you have to read to do anything problems, etc, I think engineers start to say "could I be more productive, and therefore theoretically get more reward, elsewhere?" I think as soon as the innovators start to jump ship, growth momentum is lost and it's nearly impossible to regain that momentum. Who does interviews? Who sells candidates? What is the marketing? What kind of sense of morale will potential hires perceive? If a large refactoring project did take place, refactoring projects are generally very high risk and low reward. That is the type of project that most senior devs will avoid like the plague. It is not fun engineering. It is drudgery and toil. So a history of poor software results in required cleanup/developer pain, which results in high value devs leaving for greener pastures, which results in a perception of high inertia non-growth, which then makes it harder to sell people on the prospect of "high growth" stocks which results in the tanking of average dev quality (and morale for that matter), which then makes it even harder to hire innovative devs or leadership, which ultimately results in stagnation. Senior leadership, both managerial and technical, will play hot potato until the costs of not addressing the real, highly unpleasant, problems is too high and by then its too late. Things continue to get worse between the start of the effort to fix problems and the actual fix of the problems and a lot of people will choose not to suffer that. I didn't work at twitter, but I watched this process happen. To me, there is a clear relationship between systemically unaddressed and festering technical debt and ability to innovate with direct personnel consequences. > I bet no one has said that if only Twitters software was better it would gain more customers and advertisers. But whats the cost of setting up a new service? Of creating a new interface? Of testing a new feature? Of creating a new payment system, etc. All of these are software problems. How efficient would an organization be if every team had their own repo? If every team had their own monitoring systems? If every team implemented their own release process? If every team re-implemented the same behavior over and over again because higher up leadership couldn't allocate resources to problems that every team has, what effect does that have? That inefficiency is innovation opportunity cost. Imagine you acquire a company you think can provide an innovation to your own. What is the time/resource cost of integration and what effect do you think dev environment would have on success of the acquisition? I think technical debt clearly has compounding interest. I think companies get into a position where the debt becomes debilitating particularly through pathological short term thinking. At some point the compounding technical debt exceeds growth and it's over.

scarface74 4y ago
If Twitter has a sane architecture and all indications is that it does now, creating a new feature shouldn’t require back end refactoring. You have core shared services and you build another service on top of it. I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs. I seriously doubt Twitter is one large monolithic app And let’s not pretend that Google for instance is any better at product development. It’s failed at everything it’s done not related to advertising - ie Stadia. It’s one success hides a lot of failures.
- hayst4ck 4y ago
  https://thenewstack.io/how-airbnb-and-twitter-cut-back-on-microservice-complexities/ https://thenewstack.io/how-airbnb-and-twitter-cut-back-on-mi... > Originally, Twitter ran their public APIs through a single Ruby-on-Rails application (“Monorail”), which had grown into one of the largest Rails codebases in the world. And so it was increasingly difficult to update. By 2014, Twitter went the route of microservices, migrating the API service to a set of 14 microservices, running on an internal Java Virtual Machine (JVM)-based framework (“Maccaws”). > “While the microservices approach enabled increased development speeds at first it also resulted in a scattered and disjointed Twitter API,” Cosenza said. Independent teams designed and built endpoints for their specific use cases with little coordination. This led to fragmentation and, inevitably, a slowing of developer productivity. Microservices don't make a lot of sense to me unless you can print money or have a public infra offering. It points to an engineering org that doesn't know what they are doing. Turning O(N) code complexity (virtual complexity) into O(N) operational complexity ("physical" complexity) points to an org that doesn't understand where their complexity is coming from or how to minimize it. > creating a new feature shouldn’t require back end refactoring. The argument I was making is that it doesn't matter if it gets better. Once momentum is stopped by poor architecture, inertia is too high to get moving again. > I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs. Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so. > And let’s not pretend that Google for instance is any better at product development. Google is famous for developers "just shuffling protobufs between services." While it's generally said derogatorily, that means that developers are generally not in the weeds and minutiae of what they are trying to accomplish, but are actually working on the higher level concept they are trying to achieve. > It’s failed at everything it’s done not related to advertising gsuite, maps, search, adsense, analystics, gcp, youtube, play, android, pixel, chromebook, news, docs, project zero, etc. I don't agree that failure is the rule. Cutting lackluster offerings rather than pouring investment in them shows at the very least a sane short term strategy with the weakness of potential harm for long term consumer trust. That's a reasonable trade off to make with good arguments for both sides.

scarface74 4y ago

If Twitter has a sane architecture and all indications is that it does now, creating a new feature shouldn’t require back end refactoring. You have core shared services and you build another service on top of it. I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs. I seriously doubt Twitter is one large monolithic app And let’s not pretend that Google for instance is any better at product development. It’s failed at everything it’s done not related to advertising - ie Stadia. It’s one success hides a lot of failures.

hayst4ck 4y ago
https://thenewstack.io/how-airbnb-and-twitter-cut-back-on-microservice-complexities/ https://thenewstack.io/how-airbnb-and-twitter-cut-back-on-mi... > Originally, Twitter ran their public APIs through a single Ruby-on-Rails application (“Monorail”), which had grown into one of the largest Rails codebases in the world. And so it was increasingly difficult to update. By 2014, Twitter went the route of microservices, migrating the API service to a set of 14 microservices, running on an internal Java Virtual Machine (JVM)-based framework (“Maccaws”). > “While the microservices approach enabled increased development speeds at first it also resulted in a scattered and disjointed Twitter API,” Cosenza said. Independent teams designed and built endpoints for their specific use cases with little coordination. This led to fragmentation and, inevitably, a slowing of developer productivity. Microservices don't make a lot of sense to me unless you can print money or have a public infra offering. It points to an engineering org that doesn't know what they are doing. Turning O(N) code complexity (virtual complexity) into O(N) operational complexity ("physical" complexity) points to an org that doesn't understand where their complexity is coming from or how to minimize it. > creating a new feature shouldn’t require back end refactoring. The argument I was making is that it doesn't matter if it gets better. Once momentum is stopped by poor architecture, inertia is too high to get moving again. > I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs. Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so. > And let’s not pretend that Google for instance is any better at product development. Google is famous for developers "just shuffling protobufs between services." While it's generally said derogatorily, that means that developers are generally not in the weeds and minutiae of what they are trying to accomplish, but are actually working on the higher level concept they are trying to achieve. > It’s failed at everything it’s done not related to advertising gsuite, maps, search, adsense, analystics, gcp, youtube, play, android, pixel, chromebook, news, docs, project zero, etc. I don't agree that failure is the rule. Cutting lackluster offerings rather than pouring investment in them shows at the very least a sane short term strategy with the weakness of potential harm for long term consumer trust. That's a reasonable trade off to make with good arguments for both sides.