13 ms·

Disasters I've seen in a microservices world

256 points by gregdoesit 5y ago

kevmo314 5y ago
> Some teams were suffering from servicitis. Even worse than that, it generated a lot of friction while developing. One could not just look into a project in their IDE, but it required to have multiple projects open simultaneously to make sense of all that mess. This is real: I've worked in projects where purely transformational code was offloaded into a "service". Refactoring it into a library reduced lines of code, computational cost, and code complexity dramatically. But wouldn't it be cool if there were a framework where the developer didn't have to demarcate where services started and ended? In principle, any pure asynchronous function could be abstracted out to a service. It would be neat if the compiler did that for me and deployment of the application was more like "deploy the cluster" instead of deploying each individual service.
- slver 5y ago
  > But wouldn't it be cool if there were a framework where the developer didn't have to demarcate where services started and ended? Erlang/Elixir, for JVM Akka, for .NET Akka.NET.
  - jerf 5y ago
    No, it's a bit of a myth that those techs just magically scale everything. They don't make that problem go away at all. You still need to find homes for the services to run on. It does make it a single function call to potentially start up a new service on a remote node, but if you need to be careful about resource use you're hardly any better off than you are with anything else. In a way that can end up being too easy to just add work to systems rather willy-nilly, the ceremony in kubernetes or any other more explicit system can be a good thing.
    - slver 5y ago
      The question wasn't "how to magically scale everything". The question was why not eliminate the difference between in-process async call and a service, and that's basically what Erlang and actors do.
- brown 5y ago
  Next up, leftpad.io, leftpad as a service.
  - kevmo314 5y ago
    Haha someone actually made this: http://left-pad.io/ http://left-pad.io/
    - coding123 5y ago
      This is comedy gold: > `left-pad.io` is 100% REST-compliant as defined by some guy on Hacker News with maximal opinions and minimal evidence.
- aprdm 5y ago
  Lol, that sounds like Java in 2011... put an annotation on a function and it's a service :) SOAP and all that. Maybe they had something right! It did cause problems when users put something on a tight loop that was actually a remote call but not easy to see. Anyho... what's old is new!
  - kitd 5y ago
    The one I was thinking of was SCA. Define the interface and implement the backend, either as a service or library, and the SCA layer in between would handle how to call it.
  - rsynnott 5y ago
    Also Java in 1996; remember RMI?
    - ourmandave 5y ago
      I used RMI but put a queue in front of it. One at a time, please wait your turn...
- wrnr 5y ago
  That is what lots of actor frameworks are trying, first specify a DAG of tranformations and then instantiate that graph on a physical network of computers such that the async boundaries can be scaled independently. It's cumbersome, like writing the same program squared just to get scaling for "free". Personally I think the Persistency-as-a-library and Consistency-as-a-library architecture is a saner alternative.
- ai_ja_nai 5y ago
  Erlang?
- MichaelMoser123 5y ago
  servicitis sounds like the absence of an experienced system architect, or the absence of any kind of system architecture as such.
slver 5y ago
Disaster #1 is too small services, and Disaster #4 is huge, shared databases (between many services). Which reaffirms my overall opinion that most people writing services have no idea what's a service and what it encapsulates (it encapsulates its own state, for example, it's basically distributed OOP). BTW, remember when "your service should be 100 lines of code top" was considered a best practice? Why is it so hard for most to resist this hype nonsense wave when its oncoming and it's so hard to resist the anti-hype wave that inevitably follows it? Because hype and anti-hype are simply the oscillation of a sea of empty minds in look for a solution to a problem they don't understand. I've been writing service oriented apps for decades. In my "world" nothing has changed.
- unlimit 5y ago
  > Why is it so hard for most to resist this hype nonsense wave when its oncoming and it's so hard to resist the anti-hype wave that inevitably follows it? Because then how will one do career development? We are building now for a client and the client wants it, I can feel it in my bones that what we are building is far more complex and will be difficult to maintain. I am no expert in microservices but this hunch is just coming from common sense.
  - slver 5y ago
    In your situation it seems like a non-technical person in charge of technical decisions. Those decisions are by definition poor quality. But the really bad moment is when the developers themselves make those bad choices entirely on their own.
    - unlimit 5y ago
      > In your situation it seems like a non-technical person in charge of technical decisions. Those decisions are by definition poor quality. The client is to blame. The client let go of all the people who knew the technical side of the product and has hired an architect to re-architect everything. And the client is in a hurry.
- jeffbee 5y ago
gravypod 5y ago
> One could not just look into a project in their IDE, but it required to have multiple projects open simultaneously to make sense of all that mess Why not both? You can set things up so your go-to-def can understand your API calls and head to the right place. This is very easy to do with a monorepo + protobuf setup. > How much does it cost to spin 200 services in a cloud provider? Can you do it? Can you also spin up the infrastructure needed to run them? Assuming 256mb RAM/service you're still well within 1 machine territory. Once you get above 1 machine territory you can set things up so that you can: 1. Build really good integration testing tooling so that devs don't really need to interface with all services. In a test spin up everything you need + deps, run an API call, tear everything down. This can be cached if your build system does that. You can run into issues if you have situations where 1 API call hits every service but if you've done that you've already messed up. In those cases the best you can do is mock a step in the chain there, run a few test hits the entire chain before release, but then have devs run against the mock in their integration tests. 2. Hybrid environments. You run a dev cluster that has all of your basic infrastructure that doesn't change much and provide a way for developers to launch new tasks that don't get routed to unless the driver has a feature flag flipped. Essentially you have a "dev" cluster that is continuously delivered from your master repo, each developer has the ability to launch new tasks in this cluster, and they can say "all traffic from alice for FoobarService should go to `{namespace=bob,service=Foobar}`. > As you can imagine, end-to-end tests have similar problems to development environments. Before, it was relatively easy to create a new development environment using virtual machines or containers. It was also fairly simple to create a test suite using Selenium to go through business flows and assert they were working before deploying a new version. Why is it not simple anymore? I've implemented this at more than one company. > Aside from being an obvious single-point-of-failure, defeating some of the service-oriented architecture's principles, there's more. Do you create a user per service? Do you have fine-grained permissions so service A can only read or write from specific tables? What if someone removes an index unintentionally? How do we know how many services are using different tables? What about scaling? Come up with a convention for your company and stick to it. If you can automate it that's better. If you build some way for your task you are running to know "who" it is they can inject that information into other libraries. For example you can inject the following environment variables into a container: FOOBAR_DB_ABCD=pg FOOBAR_DB_ABCD_PASSWORD=... FOOBAR_DB_ABCD_HOST=... FOOBAR_DB_ABCD_PORT=... You can then have some library you write expose a `OpenDatabase("abcd")` that connects in and injects everything. A security operator can then provision accounts and everything transparently. If you generate those env vars from some automated config management tool you don't even have to see the passwords. > Instead of having a monolith getting all of the traffic, now you have a home-made Spring Boot service getting all of it! What could go wrong? Engineers quickly realize this is a mistake, but as there are many customizations, sometimes they cannot substitute this piece for stateless, scale-friendly ones. I don't think this is a single point of failure. This is a single point of failure for a specific subset of your infrastructure and at that it should be a very simple (mostly-pass-through) component. If your mobile gateway dies, your backend one shouldn't. If all of your API gateways die then your integrations to third parties that are required for legal compliance should stay up, etc. > I've seen teams using circuit breakers and then increase the timeouts of an HTTP call to a service downstream. You should always decrease timeouts for operations if you're attempting to retry calls. You can also use a load balancer that already knows the liveness state of all of your instances. Aside from this bit I mostly agree with this section about timeouts, retries, etcs is actually correct. If you are tackling a single problem that's simple don't break it into a distributed system. If you are saying "I want to have X but we should implement a Y" where Y is a completely different thing that doesn't need to talk to X directly, then why not implement it in a separate binary? There's no reason they can't share code to make the burden of operation low.
jillesvangurp 5y ago
People do microservices for the wrong reasons. There are only a few somewhat valid reasons: 1) something has different run time needs than something else. Think CPU, memory, network. Breaking stuff up allows you to make different choices here. Although I have dodged this by simply deploying the same monolith and configure it to do different things. 2) Something needs to be developed by a different team and for whatever reasons you don't want those teams to be too dependent. It's a bad reason but it's valid in a lot of companies where certain teams just need to be engineered around or where there is a fundamental lack of trust between different parts of the org chart. Conway's law is a thing. It's the most common reason to do micro services. 3) You have two things depending on each other (cyclical dependency) and you want to reuse that thing. Extracting it to a third thing is a common way out. It's true for almost any component technology. If you have two components, you'll find a reason to create a third. And a fourth. And so on. However, consider using something less dramatic. E.g. code libraries are a valid choice. Or having an extra module in your source tree. Everything else is just needlessly/prematurely increasing overhead, deployment friction, etc. You get more things to monitor, deploy, manage the roadmap off, worry about, specialize in, etc. Big bloated organizations do micro services because they are big and bloated. Many smart startups keep this nonsense to a minimum. Of course some startups start out being over funded and bloat too early. VC money is great and sometimes requires over engineering like this (i.e. impress the suits). I've heard more than a few CTOs boast their multi cloud strategy and micro service architecture. In my mind that translates to we funnel a lot of VC money to Amazon and pay people full time to do just that. Ridiculous monthly bills and no users or traction is a common pattern in that world.
- jmchuster 5y ago
  Hmm, i don't think our needs fall into any of those three. But our reasons may just be, as you say, the wrong reasons. We split up our services when the majority of changes made to a service can be made independently of everyone else. So then each of our services is a different codebase, and they each have a completely different rate of change from each other. You very rarely would ever make changes across all services as once, only when changing what's being communicated between some set of services. Some services are large, some are small, and many of them fall into the category of -- developed for a while then stabilized and now basically rarely every touched just works. So then the advantage is that you always have a small mental working set. You're focusing on a service at a time, have less to worry about breaking everything else with your changes. And then when you deploy, even if everything goes horribly wrong, it's just your one service that is down, and everything else is up and running, and you'll just have to process your queued messages once you came back online. And then of course each service is smaller, so less code, less tests, faster to compile, faster to run through the pipeline, faster to release. And you're only ever doing rolling deploys on a small sub-section of your infrastructure, never the whole thing at once.
aprdm 5y ago
Really good write up! I think the suite spot in a < 50 eng organization is either a monolithic or a microservice per domain (instead of per functionality) Once you have thousands of engineers, then you either need extreme discipline and a huge team maintaining the "devops" pipeline that everything goes through, or, it's basically everyone for themselves and a "devops" team trying to help others setting standards, best practices and whatnot.
FpUser 5y ago
In places where micro/services were *really* needed people/orgs were implementing those even decades back. I personally was doing it in the 90s. For myself the criteria for making something as a service was simple: it will really cost the organization not to have it as a service. Now as with many things that nothingburger got suddenly overhyped and ended up being shoved into every hole disregarding of any technical rationale.
spaetzleesser 5y ago
I recently got talked into developing a new project with Kubernetes and microservices. It's an interesting journey but the complexity this adds is just enormous. Debugging is hard, refactoring is hard once it touches service boundaries, coordinating releases between services is hard and so on. I highly doubt that we will ever scale to a size where the complexity pays off. I feel this kind architecture of especially appeals to people who like to write only new code instead of understanding existing code. They don't like to read old code to see where new functionality may fit in so they spin up new services. We are complaining about maintenance of old COBOL code but God be with the poor people who in 20 years will have to maintain the monstrosities we are creating today.
- mech422 5y ago
  >>We are complaining about maintenance of old COBOL code but God be with the poor people who in 20 years will have to maintain the monstrosities we are creating today. I'm hoping for some major tax changes right before my retirement so I can dust off my COBOL and get phat lewt! I wonder how many people that have actually written COBOL will be left by 2030? No one mentions it here, but I'm sure there must be programmers deep in the bowels of banks and insurance companies that are learning COBOL even now? Oh - And RPG... wonder if there's any of that left ?
  - yurishimo 5y ago
    There are definitely new devs learning COBOL. I'm not sure where I saw it, but I could have sworn I saw an article or blog post about these mission critical companies paying insane wages and signing bonuses to devs who would sign-on to learn & maintain their infrastructure. And since these jobs would likely be on prem for security reasons, they don't have to pay insane Silicon Valley wages either. Imagine living in Cincinnati making $250k maintaining COBOL? Some people are totally fine with that paradigm and will keep the system running as long as it needs to.
    - mech422 5y ago
      I was down until the on-prem thing.. Still, I wouldn't be surprised about insane wages for COBOL. You make money at the bleeding edge of tech, or way back on the tail end - scarcity breeds profit :-)
cratermoon 5y ago
> Timeouts, retries, and resilience At a previous employer I was responsible for a critical service that was starting to show strain as traffic ramped up. It used Hystrix as the circuit breaker for calls to backend services, including the DB, and at peak times the thread pool would fill and start rejecting additional requests. I was tasked with fixing that. There's a very simple formula for tuning the number of threads: > requests per second at peak when healthy × 99th percentile latency in seconds + some breathing room The catch is that getting good RPS and latency numbers is in a distributed system deployed across three geographically separated datacenters is the opposite of simple. In particular, the legacy of the system meant that we had one write instance of the DB, in one datacenter, meaning that latency was different depending on which DC was the source of the call, so there was no one setting that worked for all instances.
helge9210 5y ago
> I've seen many engineers ignoring these because it's "an edge case", to realize later they have a massive data integrity problem. Massive optimism detected at the part of "to realize later". Also, if you have "edge case" with high (close to 1 but lower than 1) probability of successful outcome, with more and more tries probability of all outcomes being successful moves to zero (multiplication of lower than 1 values) and probability of at least single failed outcome goes to 1 (addition of lower than 1 values). You still have to handle low probability edge cases.
jaredcwhite 5y ago
The problem with the hype around microservices was that it's the web app deployment equivalent of writing device drivers in C. Sure, some people can do it. Some people have to do it. Yet most people shouldn't even make the attempt. I've been on teams where we can't even reliably deploy code over time to one service. The idea of our team maintaining multiple services is madness. That's not a knock on any individual's technical merits. Deploying code to the cloud is just hard, period—and that's even when talking about a traditional monolith! I'm glad the pendulum is swinging back. The microservices pattern is useful for the areas in which it's useful…it just so happens that problem space is way smaller than the hype cycle cared to admit a few years ago.
- dwaite 5y ago
  > I've been on teams where we can't even reliably deploy code over time to one service. The idea of our team maintaining multiple services is madness. That's not a knock on any individual's technical merits. Deploying code to the cloud is just hard, period—and that's even when talking about a traditional monolith! If devops has a good philosophy here it is - if something is hard, you should do it more often. When something is intermittent pain it can be avoided or considered a one-off; if you are say deploying your web app once a week or more people will start to figure out how to automate the manual processes and optimize the slow ones, rewrite troublesome components and come up with strategies for things like incremental database migrations. More generally - everything is a trade-off, and you shouldn't blindly accept more complexity when you don't need the benefits that it is supposed to provide. But sometimes you need to embrace the complexity of doing things the right way when you are hitting up against limitations of doing it the wrong way (in this example - slow, painful, error-prone deployments) Microservices are useful when the monolith becomes too complex for local reasoning and management, so you instead compartmentalize things into components with contracts and reason about those, while you are responsible for managing just your own component. You're taking on complexity (and latency, and additional resource utilization) because your system started to hit up against the limits of everyone working within the same project. If you haven't hit the point where your database is falling over in production because of concurrent loads, or where development is hindered by the infrastructure/dependency or time requirements of doing a local test, or various other problems - you likely don't need to invest in doing microservices.
e67f70028a46fba 5y ago
Putting a network connection between your application abstractions was always a dicey proposition. (See EJB 1.0) Making it the entire basis for application abstraction is lunacy, the sort of extremely clever idiocy that can only occur in the tech world.
- discreteevent 5y ago
  The funny thing is that Martin Fowler had a First Law of Distributed Objects: Don't distribute your objects. But then he switched and helped popularize microservices (he's rowing back a bit nowadays). I can only assume it's because he saw his clients doing it and for a consultant the client is always right. (And I think the client was doing it to indulge their hard to get developers who wanted complexity and the freedom to use their programming language of the week) It feels like a cowboy industry when this is the 'leadership' we have.
  - e67f70028a46fba 5y ago
    Yep. Look at the REST/JSON API debacle that has unfolded over the last two decades. It is trivially obvious that REST is both difficult to implement and largely pointless outside of a hypermedia system, but the thought leaders never said a thing about it.
    - edgyquant 5y ago
      How is REST difficult to implement? And as for pointless, what is a better way to present an API in your opinion?
      - e67f70028a46fba 5y ago
        JSON isn’t a hypermedium so creating a uniform interface is hard. I don’t have a strong opinion on alternatives. GraphQL seems ok for server to server stuff.
- tootie 5y ago
  There are some valid cases for microservices. Specifically when you have divergent scaling needs for different types of services. But almost everything else about them can be solved with good code modularity. I think microservices grew out crummy package managers that either didn't work very well (npm, everything python has tried) or ones that were powerful but no one knew how to use properly (maven, nuget).
deckard1 5y ago
I was going to comment on this very thing in the other thread about software glue[1]. In that article there is a youtube video on Multics vs Unix[2] which really outlines why microservices were always doomed. Someone should coin a new law for how programmers have to rediscover Brooks's law every 5-10 years. The issue with microservices, as it has always been, is that you need an enormous company to brute force the communication pathways and maintenance overhead for it to all work. And by work, I don't mean function efficiently (as the Multics vs Unix video shows). I mean just function. Just work at all. The Multics team had all the devs and the Unix team was two guys doing laps around the Multics team. Because they had the mathematics on their side. Remember the bad old days of memory thrashing? That's what happens to teams that do not have enough bandwidth to properly maintain the dozens of services they are responsible for. Your organization gets frozen. This is what we all get for taking advice from the Googles and Facebooks of the world. Google has like a billion lines of code in a monorepo. They do not do things remotely like 99% of the businesses out there. They are sitting on huge piles of money that lets them be incredibly inefficient for decades. [1] https://news.ycombinator.com/item?id=27482832 https://news.ycombinator.com/item?id=27482832 [2] https://www.youtube.com/watch?v=3Ea3pkTCYx4 https://www.youtube.com/watch?v=3Ea3pkTCYx4
- h43k3r 5y ago
  Googler here. You are absolutely right. I would never ever do something the Google way outside of Google. Google has a dedicated organisation for maintaining all the infrastructure required for and around microservices. We are talking about 500-1000 engineers just improving and maintaining the infrastructure. I don't have to worry about things like distributed tracing, monitoring, authentication, logs, logs search, logs parsing for exceptions, anamoly detection on logs, deployment, release management etc. They are already there and is glued pretty well that 90 percent of engineers don't even have to spend more than a day to understand all that.
- collyw 5y ago
  To me a microservices application is a monolithic application, with unreliable network calls between the components. That can't possibly make things simpler to understand or maintain.
jameshart 5y ago
I guess you just won’t get traction on HN writing ‘Disasters I’ve seen in a monolithic world’. If you have never gone through a significant dependency upgrade on a large monolithic codebase, you might not appreciate the value of microservice architecture. I’ve been at companies where the number one technical achievement for a tech organization for an entire calendar year was ‘successfully moved the app from .net 2 to .net 4’. Have you ever gone through the process of integrating an acquired company’s monolith into an acquiring company’s monolith? It’s futile. With a microservice architecture though I’ve seen acquisitions integrate significant systems (things like billing and auth) in a matter of weeks where monolith projects dragged on for years. Sure, there’s no silver bullets. But there are a lot of problems with monoliths which microservices eliminate. Not without trade offs, naturally.
- dspillett 5y ago
  > I guess you just won’t get traction on HN writing ‘Disasters I’ve seen in a monolithic world’. Though I think mostly that is because the discussion there has been well trodden already. Microservice Architectures are relatively new, less completely explored, and less familiar to many. There are new and interesting mistakes to make (the biggest two being to use this model where it is not at all appropriate, or cargo-culting and not really understanding what its advantages are so not actually taking advantage of them) where most of the "interesting" problems to be had in the monolithic world have been well documented for quite some time (which is why new models pop up, to try remove the potential for these problems in some, many, or even most, cases (anyone who says method X is better in all cases should be treated with deep suspicion)).
- jeffshek 5y ago
  In those organizations, I highly doubt microservices or monolithic decisions would have made a difference. Disorganized management won't magically get fixed if the root problems of ASAP, priority, techinal debt, etc features aren't fixed. The fact that the organization was trying to combine two large monolithic codebase into one is an obvious smell that technical decisions are made by someone non-technical.
  - 5y ago
ferdowsi 5y ago
It's interesting to hear about stability concerns. Overall I think my organization moving to microservices improved our resiliency story. It allowed us to freeze sensitive legacy services and gradually build other surrounding services that incrementally replaced those legacy services with better-performing Go services. Rolling out new services is not onerous due to our Kubernetes platform (which was nowhere near as difficult to build on as some might suggest). Strong service boundaries helped us, they didn't hold us back.
mavelikara 5y ago
A response someone wrote to this article: https://medium.com/productboard-engineering/countering-microservice-disasters-5a8f957803cb https://medium.com/productboard-engineering/countering-micro...
- seibelj 5y ago
  > Three languages > ”At the heart of our Engineering Strategy is a rather simple document. A manifesto if you’d like. We call it Core Engineering Principles.” > Lists 7 fairly complex rules > ”There are about 30 more touching stacks, architecture, and org structure.” I’m dying. This could only be written with a straight face by a microservices enthusiast.
  - zizee 5y ago
    I read the full post linked to by the grand-parent, and I think you are mischaracterizing things. Settling on 3 programming languages is a good compromise between "one language to rule them all", and "anything goes". Being limited a single language's talent pool can be quite limiting, as is only having one language to problems in. Having no limits on language choice also has obvious drawbacks. The "7 fairly complex rules" are not _that_ complex considering they are rules for trying to tame a domain that is inherently complex. If you try and scale any software engineering team much beyond 30 engineers you will need to have similar rules regardless of whether you have a majestic monolith, microservices, or some mix of the two.
sackerhews 5y ago
I was once working with an engineer so hell bent on splitting everything up into microservices that at one point logging became incompatible with his solution. He then argued that our banking solution didn't need logging, because it was so well tested that failure rates would be extremely low. I'm not making this up.
trixie_ 5y ago
Another one for http://microservices.fail/ http://microservices.fail/ There are so many developers who think microservices are the answer to every ill in the world it is infuriating. It’s like one of those ‘nosql is webscale we should use it’ conversions.
ryanthedev 5y ago
I work in a 300+ project monolith. If you think microserivces are an issue. I can’t help you. I have worked in both. Until you realize that both worlds have their pros and cons, just stop.
throwawaaarrgh 5y ago
In the modern world of computing, the one thing I have seen literally everywhere I've been is people implementing systems that they absolutely do not understand. Most microservices I've seen implemented are not actually microservices. Most teams (unless they are developing "simple" web or mobile applications) have no idea who is consuming their services, or how, or if what they've made is working correctly. They frequently don't have an understanding of the different models of releasing software, much less of a complete system architecture, failure modes, consistency models, reliability estimates, performance limits, etc. Mostly what I see time and again is a team of people who just write some code that seems to work for them, and then go home, without ever considering how or if it's working in the real world. I don't know anything about modern computer science education. But it appears that new developers have absolutely no idea how anything but algorithms work. It's like we taught them how hammers and saws and wrenches work, and then told them to go build a skyscraper. There are only two ways I know of for anyone today to correctly build a modern large-scale computer system: 1) Read every single book and blog post, watch every conference talk, and listen to every podcast that exists about building modern large-scale computing systems, and 2) Spend 5+ years making every mistake in the book as you try to build them yourself. It feels like the industry mostly just re-learns the same mistakes over and over like we're in Groundhog's Day (we're the extras, not Bill Murray). But it's equally possible that I just lack perspective and am expecting too much. Maybe the auto industry at the turn of the 20th century also spent decades re-learning the same lessons over and over, as the novelty of mass-producing complex systems continued to elude us. Hell, the new auto companies still don't get it right.
- pfarrell 5y ago
  “There aren’t any new problems; only new engineers.” - Quote I read online a long time ago Agree with all your points, but there is a third way: apprenticeship with older engineers. I advise new engineers to find their balance between not accepting the status quo but also realizing software is way more complicated than they think. For example: successfully versioning an API over time while supporting live deployments is (I think) not something you’re going to learn in school. As a further aside... I’ve heard older attorneys complain about newly minted lawyers. They work to pass school and then the bar. But when they start lawyering, new grads don’t know the first thing about how to actually file something at the courthouse. I’m saying, I don’t think it’s constrained to our industry.
frays 5y ago
Great read, thanks.
deleted 5y ago
[deleted]
tjpnz 5y ago
The E2E testing one is driving me nuts. I've told people time and time again that we'll never be able to have hundreds (or even a dozen) of them working reliably. Yet people still try and to date we've lost even more hours in trying to make them reliable. The idea is seductive but in my experience an exercise in futility. At the same time I've proposed alternatives such as semantic monitoring but people typically recoil in horror when they learn what it means.
- throwawaaarrgh 5y ago
  It depends on what your goals are. At Cisco, we absolutely needed to do E2E testing because we were shipping people physical devices that they relied on to work as intended. We built automation test frameworks and reusable/configurable tests. We built labs full of equipment and time-sharing systems to keep equipment constantly in use. Quality engineers wrote tens of thousands of (mostly) no-code tests, that would each do tens of thousands of permutations of tests. We wrote new test tools and bought high-performance test gear. In the end, we were able to do via automated testing what it would have taken thousands more people to do individually. Your organization and product goals may be totally different and not need that level of testing. But if it is needed, you can do it. It's up to your business to decide how much to invest in it.
mbrodersen 5y ago
If you are not a good enough developer to build monoliths then you are not a good enough developer to build micro-services.

256 points by gregdoesit 5y ago

> Some teams were suffering from servicitis. Even worse than that, it generated a lot of friction while developing. One could not just look into a project in their IDE, but it required to have multiple projects open simultaneously to make sense of all that mess. This is real: I've worked in projects where purely transformational code was offloaded into a "service". Refactoring it into a library reduced lines of code, computational cost, and code complexity dramatically. But wouldn't it be cool if there were a framework where the developer didn't have to demarcate where services started and ended? In principle, any pure asynchronous function could be abstracted out to a service. It would be neat if the compiler did that for me and deployment of the application was more like "deploy the cluster" instead of deploying each individual service.

slver 5y ago
> But wouldn't it be cool if there were a framework where the developer didn't have to demarcate where services started and ended? Erlang/Elixir, for JVM Akka, for .NET Akka.NET.
- jerf 5y ago
  No, it's a bit of a myth that those techs just magically scale everything. They don't make that problem go away at all. You still need to find homes for the services to run on. It does make it a single function call to potentially start up a new service on a remote node, but if you need to be careful about resource use you're hardly any better off than you are with anything else. In a way that can end up being too easy to just add work to systems rather willy-nilly, the ceremony in kubernetes or any other more explicit system can be a good thing.
  - slver 5y ago
    The question wasn't "how to magically scale everything". The question was why not eliminate the difference between in-process async call and a service, and that's basically what Erlang and actors do.
brown 5y ago
Next up, leftpad.io, leftpad as a service.
- kevmo314 5y ago
  Haha someone actually made this: http://left-pad.io/ http://left-pad.io/
  - coding123 5y ago
    This is comedy gold: > `left-pad.io` is 100% REST-compliant as defined by some guy on Hacker News with maximal opinions and minimal evidence.
aprdm 5y ago
Lol, that sounds like Java in 2011... put an annotation on a function and it's a service :) SOAP and all that. Maybe they had something right! It did cause problems when users put something on a tight loop that was actually a remote call but not easy to see. Anyho... what's old is new!
- kitd 5y ago
  The one I was thinking of was SCA. Define the interface and implement the backend, either as a service or library, and the SCA layer in between would handle how to call it.
- rsynnott 5y ago
  Also Java in 1996; remember RMI?
  - ourmandave 5y ago
    I used RMI but put a queue in front of it. One at a time, please wait your turn...
wrnr 5y ago
That is what lots of actor frameworks are trying, first specify a DAG of tranformations and then instantiate that graph on a physical network of computers such that the async boundaries can be scaled independently. It's cumbersome, like writing the same program squared just to get scaling for "free". Personally I think the Persistency-as-a-library and Consistency-as-a-library architecture is a saner alternative.
ai_ja_nai 5y ago
Erlang?
MichaelMoser123 5y ago
servicitis sounds like the absence of an experienced system architect, or the absence of any kind of system architecture as such.

slver 5y ago

Disaster #1 is too small services, and Disaster #4 is huge, shared databases (between many services). Which reaffirms my overall opinion that most people writing services have no idea what's a service and what it encapsulates (it encapsulates its own state, for example, it's basically distributed OOP). BTW, remember when "your service should be 100 lines of code top" was considered a best practice? Why is it so hard for most to resist this hype nonsense wave when its oncoming and it's so hard to resist the anti-hype wave that inevitably follows it? Because hype and anti-hype are simply the oscillation of a sea of empty minds in look for a solution to a problem they don't understand. I've been writing service oriented apps for decades. In my "world" nothing has changed.

unlimit 5y ago
> Why is it so hard for most to resist this hype nonsense wave when its oncoming and it's so hard to resist the anti-hype wave that inevitably follows it? Because then how will one do career development? We are building now for a client and the client wants it, I can feel it in my bones that what we are building is far more complex and will be difficult to maintain. I am no expert in microservices but this hunch is just coming from common sense.
- slver 5y ago
  In your situation it seems like a non-technical person in charge of technical decisions. Those decisions are by definition poor quality. But the really bad moment is when the developers themselves make those bad choices entirely on their own.
  - unlimit 5y ago
    > In your situation it seems like a non-technical person in charge of technical decisions. Those decisions are by definition poor quality. The client is to blame. The client let go of all the people who knew the technical side of the product and has hired an architect to re-architect everything. And the client is in a hurry.
jeffbee 5y ago

gravypod 5y ago

> One could not just look into a project in their IDE, but it required to have multiple projects open simultaneously to make sense of all that mess Why not both? You can set things up so your go-to-def can understand your API calls and head to the right place. This is very easy to do with a monorepo + protobuf setup. > How much does it cost to spin 200 services in a cloud provider? Can you do it? Can you also spin up the infrastructure needed to run them? Assuming 256mb RAM/service you're still well within 1 machine territory. Once you get above 1 machine territory you can set things up so that you can: 1. Build really good integration testing tooling so that devs don't really need to interface with all services. In a test spin up everything you need + deps, run an API call, tear everything down. This can be cached if your build system does that. You can run into issues if you have situations where 1 API call hits every service but if you've done that you've already messed up. In those cases the best you can do is mock a step in the chain there, run a few test hits the entire chain before release, but then have devs run against the mock in their integration tests. 2. Hybrid environments. You run a dev cluster that has all of your basic infrastructure that doesn't change much and provide a way for developers to launch new tasks that don't get routed to unless the driver has a feature flag flipped. Essentially you have a "dev" cluster that is continuously delivered from your master repo, each developer has the ability to launch new tasks in this cluster, and they can say "all traffic from alice for FoobarService should go to `{namespace=bob,service=Foobar}`. > As you can imagine, end-to-end tests have similar problems to development environments. Before, it was relatively easy to create a new development environment using virtual machines or containers. It was also fairly simple to create a test suite using Selenium to go through business flows and assert they were working before deploying a new version. Why is it not simple anymore? I've implemented this at more than one company. > Aside from being an obvious single-point-of-failure, defeating some of the service-oriented architecture's principles, there's more. Do you create a user per service? Do you have fine-grained permissions so service A can only read or write from specific tables? What if someone removes an index unintentionally? How do we know how many services are using different tables? What about scaling? Come up with a convention for your company and stick to it. If you can automate it that's better. If you build some way for your task you are running to know "who" it is they can inject that information into other libraries. For example you can inject the following environment variables into a container: FOOBAR_DB_ABCD=pg FOOBAR_DB_ABCD_PASSWORD=... FOOBAR_DB_ABCD_HOST=... FOOBAR_DB_ABCD_PORT=... You can then have some library you write expose a `OpenDatabase("abcd")` that connects in and injects everything. A security operator can then provision accounts and everything transparently. If you generate those env vars from some automated config management tool you don't even have to see the passwords. > Instead of having a monolith getting all of the traffic, now you have a home-made Spring Boot service getting all of it! What could go wrong? Engineers quickly realize this is a mistake, but as there are many customizations, sometimes they cannot substitute this piece for stateless, scale-friendly ones. I don't think this is a single point of failure. This is a single point of failure for a specific subset of your infrastructure and at that it should be a very simple (mostly-pass-through) component. If your mobile gateway dies, your backend one shouldn't. If all of your API gateways die then your integrations to third parties that are required for legal compliance should stay up, etc. > I've seen teams using circuit breakers and then increase the timeouts of an HTTP call to a service downstream. You should always decrease timeouts for operations if you're attempting to retry calls. You can also use a load balancer that already knows the liveness state of all of your instances. Aside from this bit I mostly agree with this section about timeouts, retries, etcs is actually correct. If you are tackling a single problem that's simple don't break it into a distributed system. If you are saying "I want to have X but we should implement a Y" where Y is a completely different thing that doesn't need to talk to X directly, then why not implement it in a separate binary? There's no reason they can't share code to make the burden of operation low.

jillesvangurp 5y ago

People do microservices for the wrong reasons. There are only a few somewhat valid reasons: 1) something has different run time needs than something else. Think CPU, memory, network. Breaking stuff up allows you to make different choices here. Although I have dodged this by simply deploying the same monolith and configure it to do different things. 2) Something needs to be developed by a different team and for whatever reasons you don't want those teams to be too dependent. It's a bad reason but it's valid in a lot of companies where certain teams just need to be engineered around or where there is a fundamental lack of trust between different parts of the org chart. Conway's law is a thing. It's the most common reason to do micro services. 3) You have two things depending on each other (cyclical dependency) and you want to reuse that thing. Extracting it to a third thing is a common way out. It's true for almost any component technology. If you have two components, you'll find a reason to create a third. And a fourth. And so on. However, consider using something less dramatic. E.g. code libraries are a valid choice. Or having an extra module in your source tree. Everything else is just needlessly/prematurely increasing overhead, deployment friction, etc. You get more things to monitor, deploy, manage the roadmap off, worry about, specialize in, etc. Big bloated organizations do micro services because they are big and bloated. Many smart startups keep this nonsense to a minimum. Of course some startups start out being over funded and bloat too early. VC money is great and sometimes requires over engineering like this (i.e. impress the suits). I've heard more than a few CTOs boast their multi cloud strategy and micro service architecture. In my mind that translates to we funnel a lot of VC money to Amazon and pay people full time to do just that. Ridiculous monthly bills and no users or traction is a common pattern in that world.

jmchuster 5y ago
Hmm, i don't think our needs fall into any of those three. But our reasons may just be, as you say, the wrong reasons. We split up our services when the majority of changes made to a service can be made independently of everyone else. So then each of our services is a different codebase, and they each have a completely different rate of change from each other. You very rarely would ever make changes across all services as once, only when changing what's being communicated between some set of services. Some services are large, some are small, and many of them fall into the category of -- developed for a while then stabilized and now basically rarely every touched just works. So then the advantage is that you always have a small mental working set. You're focusing on a service at a time, have less to worry about breaking everything else with your changes. And then when you deploy, even if everything goes horribly wrong, it's just your one service that is down, and everything else is up and running, and you'll just have to process your queued messages once you came back online. And then of course each service is smaller, so less code, less tests, faster to compile, faster to run through the pipeline, faster to release. And you're only ever doing rolling deploys on a small sub-section of your infrastructure, never the whole thing at once.

aprdm 5y ago

Really good write up! I think the suite spot in a < 50 eng organization is either a monolithic or a microservice per domain (instead of per functionality) Once you have thousands of engineers, then you either need extreme discipline and a huge team maintaining the "devops" pipeline that everything goes through, or, it's basically everyone for themselves and a "devops" team trying to help others setting standards, best practices and whatnot.

FpUser 5y ago

In places where micro/services were *really* needed people/orgs were implementing those even decades back. I personally was doing it in the 90s. For myself the criteria for making something as a service was simple: it will really cost the organization not to have it as a service. Now as with many things that nothingburger got suddenly overhyped and ended up being shoved into every hole disregarding of any technical rationale.

spaetzleesser 5y ago

I recently got talked into developing a new project with Kubernetes and microservices. It's an interesting journey but the complexity this adds is just enormous. Debugging is hard, refactoring is hard once it touches service boundaries, coordinating releases between services is hard and so on. I highly doubt that we will ever scale to a size where the complexity pays off. I feel this kind architecture of especially appeals to people who like to write only new code instead of understanding existing code. They don't like to read old code to see where new functionality may fit in so they spin up new services. We are complaining about maintenance of old COBOL code but God be with the poor people who in 20 years will have to maintain the monstrosities we are creating today.

mech422 5y ago
>>We are complaining about maintenance of old COBOL code but God be with the poor people who in 20 years will have to maintain the monstrosities we are creating today. I'm hoping for some major tax changes right before my retirement so I can dust off my COBOL and get phat lewt! I wonder how many people that have actually written COBOL will be left by 2030? No one mentions it here, but I'm sure there must be programmers deep in the bowels of banks and insurance companies that are learning COBOL even now? Oh - And RPG... wonder if there's any of that left ?
- yurishimo 5y ago
  There are definitely new devs learning COBOL. I'm not sure where I saw it, but I could have sworn I saw an article or blog post about these mission critical companies paying insane wages and signing bonuses to devs who would sign-on to learn & maintain their infrastructure. And since these jobs would likely be on prem for security reasons, they don't have to pay insane Silicon Valley wages either. Imagine living in Cincinnati making $250k maintaining COBOL? Some people are totally fine with that paradigm and will keep the system running as long as it needs to.
  - mech422 5y ago
    I was down until the on-prem thing.. Still, I wouldn't be surprised about insane wages for COBOL. You make money at the bleeding edge of tech, or way back on the tail end - scarcity breeds profit :-)

cratermoon 5y ago

> Timeouts, retries, and resilience At a previous employer I was responsible for a critical service that was starting to show strain as traffic ramped up. It used Hystrix as the circuit breaker for calls to backend services, including the DB, and at peak times the thread pool would fill and start rejecting additional requests. I was tasked with fixing that. There's a very simple formula for tuning the number of threads: > requests per second at peak when healthy × 99th percentile latency in seconds + some breathing room The catch is that getting good RPS and latency numbers is in a distributed system deployed across three geographically separated datacenters is the opposite of simple. In particular, the legacy of the system meant that we had one write instance of the DB, in one datacenter, meaning that latency was different depending on which DC was the source of the call, so there was no one setting that worked for all instances.

helge9210 5y ago

> I've seen many engineers ignoring these because it's "an edge case", to realize later they have a massive data integrity problem. Massive optimism detected at the part of "to realize later". Also, if you have "edge case" with high (close to 1 but lower than 1) probability of successful outcome, with more and more tries probability of all outcomes being successful moves to zero (multiplication of lower than 1 values) and probability of at least single failed outcome goes to 1 (addition of lower than 1 values). You still have to handle low probability edge cases.

jaredcwhite 5y ago

The problem with the hype around microservices was that it's the web app deployment equivalent of writing device drivers in C. Sure, some people can do it. Some people have to do it. Yet most people shouldn't even make the attempt. I've been on teams where we can't even reliably deploy code over time to one service. The idea of our team maintaining multiple services is madness. That's not a knock on any individual's technical merits. Deploying code to the cloud is just hard, period—and that's even when talking about a traditional monolith! I'm glad the pendulum is swinging back. The microservices pattern is useful for the areas in which it's useful…it just so happens that problem space is way smaller than the hype cycle cared to admit a few years ago.

dwaite 5y ago
> I've been on teams where we can't even reliably deploy code over time to one service. The idea of our team maintaining multiple services is madness. That's not a knock on any individual's technical merits. Deploying code to the cloud is just hard, period—and that's even when talking about a traditional monolith! If devops has a good philosophy here it is - if something is hard, you should do it more often. When something is intermittent pain it can be avoided or considered a one-off; if you are say deploying your web app once a week or more people will start to figure out how to automate the manual processes and optimize the slow ones, rewrite troublesome components and come up with strategies for things like incremental database migrations. More generally - everything is a trade-off, and you shouldn't blindly accept more complexity when you don't need the benefits that it is supposed to provide. But sometimes you need to embrace the complexity of doing things the right way when you are hitting up against limitations of doing it the wrong way (in this example - slow, painful, error-prone deployments) Microservices are useful when the monolith becomes too complex for local reasoning and management, so you instead compartmentalize things into components with contracts and reason about those, while you are responsible for managing just your own component. You're taking on complexity (and latency, and additional resource utilization) because your system started to hit up against the limits of everyone working within the same project. If you haven't hit the point where your database is falling over in production because of concurrent loads, or where development is hindered by the infrastructure/dependency or time requirements of doing a local test, or various other problems - you likely don't need to invest in doing microservices.

e67f70028a46fba 5y ago

Putting a network connection between your application abstractions was always a dicey proposition. (See EJB 1.0) Making it the entire basis for application abstraction is lunacy, the sort of extremely clever idiocy that can only occur in the tech world.

discreteevent 5y ago
The funny thing is that Martin Fowler had a First Law of Distributed Objects: Don't distribute your objects. But then he switched and helped popularize microservices (he's rowing back a bit nowadays). I can only assume it's because he saw his clients doing it and for a consultant the client is always right. (And I think the client was doing it to indulge their hard to get developers who wanted complexity and the freedom to use their programming language of the week) It feels like a cowboy industry when this is the 'leadership' we have.
- e67f70028a46fba 5y ago
  Yep. Look at the REST/JSON API debacle that has unfolded over the last two decades. It is trivially obvious that REST is both difficult to implement and largely pointless outside of a hypermedia system, but the thought leaders never said a thing about it.
  - edgyquant 5y ago
    How is REST difficult to implement? And as for pointless, what is a better way to present an API in your opinion?
    - e67f70028a46fba 5y ago
      JSON isn’t a hypermedium so creating a uniform interface is hard. I don’t have a strong opinion on alternatives. GraphQL seems ok for server to server stuff.
tootie 5y ago
There are some valid cases for microservices. Specifically when you have divergent scaling needs for different types of services. But almost everything else about them can be solved with good code modularity. I think microservices grew out crummy package managers that either didn't work very well (npm, everything python has tried) or ones that were powerful but no one knew how to use properly (maven, nuget).

deckard1 5y ago

I was going to comment on this very thing in the other thread about software glue[1]. In that article there is a youtube video on Multics vs Unix[2] which really outlines why microservices were always doomed. Someone should coin a new law for how programmers have to rediscover Brooks's law every 5-10 years. The issue with microservices, as it has always been, is that you need an enormous company to brute force the communication pathways and maintenance overhead for it to all work. And by work, I don't mean function efficiently (as the Multics vs Unix video shows). I mean just function. Just work at all. The Multics team had all the devs and the Unix team was two guys doing laps around the Multics team. Because they had the mathematics on their side. Remember the bad old days of memory thrashing? That's what happens to teams that do not have enough bandwidth to properly maintain the dozens of services they are responsible for. Your organization gets frozen. This is what we all get for taking advice from the Googles and Facebooks of the world. Google has like a billion lines of code in a monorepo. They do not do things remotely like 99% of the businesses out there. They are sitting on huge piles of money that lets them be incredibly inefficient for decades. [1] https://news.ycombinator.com/item?id=27482832 https://news.ycombinator.com/item?id=27482832 [2] https://www.youtube.com/watch?v=3Ea3pkTCYx4 https://www.youtube.com/watch?v=3Ea3pkTCYx4

h43k3r 5y ago
Googler here. You are absolutely right. I would never ever do something the Google way outside of Google. Google has a dedicated organisation for maintaining all the infrastructure required for and around microservices. We are talking about 500-1000 engineers just improving and maintaining the infrastructure. I don't have to worry about things like distributed tracing, monitoring, authentication, logs, logs search, logs parsing for exceptions, anamoly detection on logs, deployment, release management etc. They are already there and is glued pretty well that 90 percent of engineers don't even have to spend more than a day to understand all that.
collyw 5y ago
To me a microservices application is a monolithic application, with unreliable network calls between the components. That can't possibly make things simpler to understand or maintain.

jameshart 5y ago

I guess you just won’t get traction on HN writing ‘Disasters I’ve seen in a monolithic world’. If you have never gone through a significant dependency upgrade on a large monolithic codebase, you might not appreciate the value of microservice architecture. I’ve been at companies where the number one technical achievement for a tech organization for an entire calendar year was ‘successfully moved the app from .net 2 to .net 4’. Have you ever gone through the process of integrating an acquired company’s monolith into an acquiring company’s monolith? It’s futile. With a microservice architecture though I’ve seen acquisitions integrate significant systems (things like billing and auth) in a matter of weeks where monolith projects dragged on for years. Sure, there’s no silver bullets. But there are a lot of problems with monoliths which microservices eliminate. Not without trade offs, naturally.

dspillett 5y ago
> I guess you just won’t get traction on HN writing ‘Disasters I’ve seen in a monolithic world’. Though I think mostly that is because the discussion there has been well trodden already. Microservice Architectures are relatively new, less completely explored, and less familiar to many. There are new and interesting mistakes to make (the biggest two being to use this model where it is not at all appropriate, or cargo-culting and not really understanding what its advantages are so not actually taking advantage of them) where most of the "interesting" problems to be had in the monolithic world have been well documented for quite some time (which is why new models pop up, to try remove the potential for these problems in some, many, or even most, cases (anyone who says method X is better in all cases should be treated with deep suspicion)).
jeffshek 5y ago
In those organizations, I highly doubt microservices or monolithic decisions would have made a difference. Disorganized management won't magically get fixed if the root problems of ASAP, priority, techinal debt, etc features aren't fixed. The fact that the organization was trying to combine two large monolithic codebase into one is an obvious smell that technical decisions are made by someone non-technical.
- 5y ago

ferdowsi 5y ago

It's interesting to hear about stability concerns. Overall I think my organization moving to microservices improved our resiliency story. It allowed us to freeze sensitive legacy services and gradually build other surrounding services that incrementally replaced those legacy services with better-performing Go services. Rolling out new services is not onerous due to our Kubernetes platform (which was nowhere near as difficult to build on as some might suggest). Strong service boundaries helped us, they didn't hold us back.

mavelikara 5y ago

A response someone wrote to this article: https://medium.com/productboard-engineering/countering-microservice-disasters-5a8f957803cb https://medium.com/productboard-engineering/countering-micro...

seibelj 5y ago
> Three languages > ”At the heart of our Engineering Strategy is a rather simple document. A manifesto if you’d like. We call it Core Engineering Principles.” > Lists 7 fairly complex rules > ”There are about 30 more touching stacks, architecture, and org structure.” I’m dying. This could only be written with a straight face by a microservices enthusiast.
- zizee 5y ago
  I read the full post linked to by the grand-parent, and I think you are mischaracterizing things. Settling on 3 programming languages is a good compromise between "one language to rule them all", and "anything goes". Being limited a single language's talent pool can be quite limiting, as is only having one language to problems in. Having no limits on language choice also has obvious drawbacks. The "7 fairly complex rules" are not _that_ complex considering they are rules for trying to tame a domain that is inherently complex. If you try and scale any software engineering team much beyond 30 engineers you will need to have similar rules regardless of whether you have a majestic monolith, microservices, or some mix of the two.

sackerhews 5y ago

I was once working with an engineer so hell bent on splitting everything up into microservices that at one point logging became incompatible with his solution. He then argued that our banking solution didn't need logging, because it was so well tested that failure rates would be extremely low. I'm not making this up.

trixie_ 5y ago

Another one for http://microservices.fail/ http://microservices.fail/ There are so many developers who think microservices are the answer to every ill in the world it is infuriating. It’s like one of those ‘nosql is webscale we should use it’ conversions.

ryanthedev 5y ago

I work in a 300+ project monolith. If you think microserivces are an issue. I can’t help you. I have worked in both. Until you realize that both worlds have their pros and cons, just stop.

throwawaaarrgh 5y ago

In the modern world of computing, the one thing I have seen literally everywhere I've been is people implementing systems that they absolutely do not understand. Most microservices I've seen implemented are not actually microservices. Most teams (unless they are developing "simple" web or mobile applications) have no idea who is consuming their services, or how, or if what they've made is working correctly. They frequently don't have an understanding of the different models of releasing software, much less of a complete system architecture, failure modes, consistency models, reliability estimates, performance limits, etc. Mostly what I see time and again is a team of people who just write some code that seems to work for them, and then go home, without ever considering how or if it's working in the real world. I don't know anything about modern computer science education. But it appears that new developers have absolutely no idea how anything but algorithms work. It's like we taught them how hammers and saws and wrenches work, and then told them to go build a skyscraper. There are only two ways I know of for anyone today to correctly build a modern large-scale computer system: 1) Read every single book and blog post, watch every conference talk, and listen to every podcast that exists about building modern large-scale computing systems, and 2) Spend 5+ years making every mistake in the book as you try to build them yourself. It feels like the industry mostly just re-learns the same mistakes over and over like we're in Groundhog's Day (we're the extras, not Bill Murray). But it's equally possible that I just lack perspective and am expecting too much. Maybe the auto industry at the turn of the 20th century also spent decades re-learning the same lessons over and over, as the novelty of mass-producing complex systems continued to elude us. Hell, the new auto companies still don't get it right.

pfarrell 5y ago
“There aren’t any new problems; only new engineers.” - Quote I read online a long time ago Agree with all your points, but there is a third way: apprenticeship with older engineers. I advise new engineers to find their balance between not accepting the status quo but also realizing software is way more complicated than they think. For example: successfully versioning an API over time while supporting live deployments is (I think) not something you’re going to learn in school. As a further aside... I’ve heard older attorneys complain about newly minted lawyers. They work to pass school and then the bar. But when they start lawyering, new grads don’t know the first thing about how to actually file something at the courthouse. I’m saying, I don’t think it’s constrained to our industry.

frays 5y ago

Great read, thanks.

deleted 5y ago

[deleted]

tjpnz 5y ago

The E2E testing one is driving me nuts. I've told people time and time again that we'll never be able to have hundreds (or even a dozen) of them working reliably. Yet people still try and to date we've lost even more hours in trying to make them reliable. The idea is seductive but in my experience an exercise in futility. At the same time I've proposed alternatives such as semantic monitoring but people typically recoil in horror when they learn what it means.

throwawaaarrgh 5y ago
It depends on what your goals are. At Cisco, we absolutely needed to do E2E testing because we were shipping people physical devices that they relied on to work as intended. We built automation test frameworks and reusable/configurable tests. We built labs full of equipment and time-sharing systems to keep equipment constantly in use. Quality engineers wrote tens of thousands of (mostly) no-code tests, that would each do tens of thousands of permutations of tests. We wrote new test tools and bought high-performance test gear. In the end, we were able to do via automated testing what it would have taken thousands more people to do individually. Your organization and product goals may be totally different and not need that level of testing. But if it is needed, you can do it. It's up to your business to decide how much to invest in it.

mbrodersen 5y ago

If you are not a good enough developer to build monoliths then you are not a good enough developer to build micro-services.