9 ms·

I think my biggest gripe with orgs that adopt microservices is they don't build out any of the testing, CI/CD, monitoring and debugging workflows to support it.

I think my biggest gripe with orgs that adopt microservices is they don't build out any of the testing, CI/CD, monitoring and debugging workflows to support it. It goes from shitty, slow monolithic application that super-pro can debug in a few minutes to.. Slow, shitty disparate services that are managed by different teams who don't share anything, suddenly you've got a Cuckoo's Egg situation where 1 guy needs to get access to all the things to find out what the fuck is happening. Or you just accept it's shitty and slow, and pay a consultancy to rebuild it in New Thing 2.0 in 8 years when accounting forget about the last 3 rebuilds.

lmilcin 6y ago
That is EXACTLY what I have observed, time and time again. If you have trouble managing ONE application what makes you think you will be better at managing multiple? Also, running distributed system is way more complicated than having all logic in a single application. Ideally you want to delay switching to distributed system until it is inevitable that you are not going to be able to fulfill the demand using monolithic service. If your application has problems, don't move to microservices. Remove all unnecessary tasks and let your team focus on solving the problems first and then automate all your development, testing, deployment and maintenance processes. Or call me and I can help you diagnose and plan:)
- bulleyashah 6y ago
  Monoliths are also distributed systems and will run on multiple hosts, most probably co-ordinating on some sort of state (and that state management will need to take care of concurrency, consistency). Some hosts will go down. Service traffic will increase. I understand your point. You are using distributed in the sense of "how is one big work distributed", you probably also hate overly "Object Oriented code" for similar reasons. But distributed systems is a well understood thing in the industry. If I call you and you tell me this, then you're directly responsible for hurting how successful I would be by giving me a misleading sense of what a distributed systems is.
  - pjc50 6y ago
    > Monoliths are also distributed systems and will run on multiple hosts ... not necessarily. Although the big SPOF monolith has gone out of fashion, do not underestimate the throughput possible from one single very fast server.
    - bulleyashah 6y ago
      Well, no matter how fast a single server is, it can't keep getting faster. You might shoot yourself in the foot by optimizing only for single servers because eventually you'll need horizontal scaling and it's better to think about it in the beginning of your architecture.
      - rthomas6 6y ago
        Whatever happened to premature optimization being bad?
      - acdha 6y ago
        > eventually you'll need horizontal scaling This is far from inevitable. There are tons of systems which never grow that much - not everyone works at a growth-oriented startup - or do so in ways which aren’t obvious when initially designing it. Given how easily you can get massive servers these days you can also buy yourself a lot of margin for one developer’s salary part time.
    - NationalPark 6y ago
      Even in a contrived situation where you have a strict cache locality constraint for performance reasons or something, you'd still want to have at least a second host for failover. Now you have a distributed system and a service discovery problem!
  - marcosdumay 6y ago
    > But distributed systems is a well understood thing in the industry. Wait, what? Distributed systems are one of the most active areas on CS currently. That's the opposite of "well understood". It's true that most systems people create are required to be distributed. But they are coordinated by a single database layer that satisfies approximately all the requirements. What remains is an atomic facade that developers can write as if their clients were the only one. There is a huge difference between that and a microservices architecture.
npteljes 6y ago
I think this nails it. It's not the concept's fault if the implementation is half-assed.
- mcny 6y ago
  > It's not the concept's fault if the implementation is half-assed. worse than that. Case in point: a major US national bank interview for a developer position. they talk about moving toward microservices. a simple question since they mention microservices: will someone be able to access and modify data data underneath my service without going through my service? the uneasy answer: yes, it does and will happen. that's the end of the interview as far as I am concerned. If you can access and modify the underlying data store from my micro service, not only it isn't a micro service, it isn't much of a service oriented architecture. this isn't me being a purist, just being practical. If we need to coordinate with five different teams to change the internal implementation of my "microservice", what is the point of doing service oriented architecture? all the downside with zero upside?
  - lifeisstillgood 6y ago
    Thats a distinction worth following up: My take is there are 3 kinds of micro-service * Service Ownership of data - if you want to change customer name, there is only one place to go. * Worker services - they don't really own data, they process something - usually requesting from golden sources. Just worker bees but the thing they do (send out marketing emails to that persons name) is not done by anyone else * Everything else is borked
  - rantwasp 6y ago
    sorry, that’s a distributed monolith. if everyone can mess with the data directly you might as well keep it in one service
kevstev 6y ago
So I actually find that microservices should actually help tremendously here? Service A starts throwing 500s. Service A has a bunch of well defined API calls it makes with known, and logged, requests and responses. These responses should be validated on the way in and produce 400s if they aren't well formed. Most 500s IMHO result from the validations not catching all corner cases, or not handling the downstream errors properly. But in general it should be relatively easy to track down why one, or a series of calls failed. I also find that by having separate distinct services, it puts up a lot of friction to scope creep in that service and also avoids side effect problems- IE you made this call, and little did you know this updated state somewhere you completely didn't expect and now touching this area is considered off limits, or at least scary because it has tentacles in so many different places. Eventually this will absolutely happen IME. No of course not on your team, you are better than that, but eventually teams change, this is now handled by the offshore or other B/C team, or a tyrant manager takes over for a year or two before that is obsessed with hitting the date, hacks or not, etc... But I guess an absolutely critical key to that is having a logging/monitoring/testing/tracing workflow built in. Frameworks can help, Hapi.js makes a lot of this stuff a core concept for example. This is table stakes to be doing "micro" services though and any team that doesn't realize that has no business going near them. Based on the comments here though ignorance around this for teams embracing microservices might be more common than I had imagined.
- acdha 6y ago
  > So I actually find that microservices should actually help tremendously here? Service A starts throwing 500s. Service A has a bunch of well defined API calls it makes with known, and logged, requests and responses. This isn’t wrong - although there is a reasonable concern about expanding interconnection problems – but I think there’s commonly a misattribution problem in these discussions: a team which can produce clean microservices by definition has a good handle on architecture, ownership, quality, business understanding, etc. and would almost certainly bring those same traits to bear successfully for a more monolithic architecture, too. A specific case of this is when people successfully replace an old system with a new one and credit new languages, methodology, etc. more than better understanding of the problem, which is usually the biggest single factor. Fundamentally, I like microservices (scaling & security boundaries) but I think anything trendy encounters the silver bullet problem where people love the idea that there’s this one weird trick they can do rather than invest in the harder problems of culture, training, managing features versus technical debt, etc. Adopting microservices doesn’t mean anyone has to acknowledge that the way they were managing projects wasn’t effective.

lmilcin 6y ago

That is EXACTLY what I have observed, time and time again. If you have trouble managing ONE application what makes you think you will be better at managing multiple? Also, running distributed system is way more complicated than having all logic in a single application. Ideally you want to delay switching to distributed system until it is inevitable that you are not going to be able to fulfill the demand using monolithic service. If your application has problems, don't move to microservices. Remove all unnecessary tasks and let your team focus on solving the problems first and then automate all your development, testing, deployment and maintenance processes. Or call me and I can help you diagnose and plan:)

bulleyashah 6y ago
Monoliths are also distributed systems and will run on multiple hosts, most probably co-ordinating on some sort of state (and that state management will need to take care of concurrency, consistency). Some hosts will go down. Service traffic will increase. I understand your point. You are using distributed in the sense of "how is one big work distributed", you probably also hate overly "Object Oriented code" for similar reasons. But distributed systems is a well understood thing in the industry. If I call you and you tell me this, then you're directly responsible for hurting how successful I would be by giving me a misleading sense of what a distributed systems is.
- pjc50 6y ago
  > Monoliths are also distributed systems and will run on multiple hosts ... not necessarily. Although the big SPOF monolith has gone out of fashion, do not underestimate the throughput possible from one single very fast server.
  - bulleyashah 6y ago
    Well, no matter how fast a single server is, it can't keep getting faster. You might shoot yourself in the foot by optimizing only for single servers because eventually you'll need horizontal scaling and it's better to think about it in the beginning of your architecture.
    - rthomas6 6y ago
      Whatever happened to premature optimization being bad?
    - acdha 6y ago
      > eventually you'll need horizontal scaling This is far from inevitable. There are tons of systems which never grow that much - not everyone works at a growth-oriented startup - or do so in ways which aren’t obvious when initially designing it. Given how easily you can get massive servers these days you can also buy yourself a lot of margin for one developer’s salary part time.
  - NationalPark 6y ago
    Even in a contrived situation where you have a strict cache locality constraint for performance reasons or something, you'd still want to have at least a second host for failover. Now you have a distributed system and a service discovery problem!
- marcosdumay 6y ago
  > But distributed systems is a well understood thing in the industry. Wait, what? Distributed systems are one of the most active areas on CS currently. That's the opposite of "well understood". It's true that most systems people create are required to be distributed. But they are coordinated by a single database layer that satisfies approximately all the requirements. What remains is an atomic facade that developers can write as if their clients were the only one. There is a huge difference between that and a microservices architecture.

npteljes 6y ago

I think this nails it. It's not the concept's fault if the implementation is half-assed.

mcny 6y ago
> It's not the concept's fault if the implementation is half-assed. worse than that. Case in point: a major US national bank interview for a developer position. they talk about moving toward microservices. a simple question since they mention microservices: will someone be able to access and modify data data underneath my service without going through my service? the uneasy answer: yes, it does and will happen. that's the end of the interview as far as I am concerned. If you can access and modify the underlying data store from my micro service, not only it isn't a micro service, it isn't much of a service oriented architecture. this isn't me being a purist, just being practical. If we need to coordinate with five different teams to change the internal implementation of my "microservice", what is the point of doing service oriented architecture? all the downside with zero upside?
- lifeisstillgood 6y ago
  Thats a distinction worth following up: My take is there are 3 kinds of micro-service * Service Ownership of data - if you want to change customer name, there is only one place to go. * Worker services - they don't really own data, they process something - usually requesting from golden sources. Just worker bees but the thing they do (send out marketing emails to that persons name) is not done by anyone else * Everything else is borked
- rantwasp 6y ago
  sorry, that’s a distributed monolith. if everyone can mess with the data directly you might as well keep it in one service

kevstev 6y ago

So I actually find that microservices should actually help tremendously here? Service A starts throwing 500s. Service A has a bunch of well defined API calls it makes with known, and logged, requests and responses. These responses should be validated on the way in and produce 400s if they aren't well formed. Most 500s IMHO result from the validations not catching all corner cases, or not handling the downstream errors properly. But in general it should be relatively easy to track down why one, or a series of calls failed. I also find that by having separate distinct services, it puts up a lot of friction to scope creep in that service and also avoids side effect problems- IE you made this call, and little did you know this updated state somewhere you completely didn't expect and now touching this area is considered off limits, or at least scary because it has tentacles in so many different places. Eventually this will absolutely happen IME. No of course not on your team, you are better than that, but eventually teams change, this is now handled by the offshore or other B/C team, or a tyrant manager takes over for a year or two before that is obsessed with hitting the date, hacks or not, etc... But I guess an absolutely critical key to that is having a logging/monitoring/testing/tracing workflow built in. Frameworks can help, Hapi.js makes a lot of this stuff a core concept for example. This is table stakes to be doing "micro" services though and any team that doesn't realize that has no business going near them. Based on the comments here though ignorance around this for teams embracing microservices might be more common than I had imagined.

acdha 6y ago
> So I actually find that microservices should actually help tremendously here? Service A starts throwing 500s. Service A has a bunch of well defined API calls it makes with known, and logged, requests and responses. This isn’t wrong - although there is a reasonable concern about expanding interconnection problems – but I think there’s commonly a misattribution problem in these discussions: a team which can produce clean microservices by definition has a good handle on architecture, ownership, quality, business understanding, etc. and would almost certainly bring those same traits to bear successfully for a more monolithic architecture, too. A specific case of this is when people successfully replace an old system with a new one and credit new languages, methodology, etc. more than better understanding of the problem, which is usually the biggest single factor. Fundamentally, I like microservices (scaling & security boundaries) but I think anything trendy encounters the silver bullet problem where people love the idea that there’s this one weird trick they can do rather than invest in the harder problems of culture, training, managing features versus technical debt, etc. Adopting microservices doesn’t mean anyone has to acknowledge that the way they were managing projects wasn’t effective.