5 ms·

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm

by vondur 6y ago

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

staticassertion 6y ago
Feels like you've just described every single bug ever.
- m463 6y ago
  Even at the start I'm sure Grace Hopper didn't anticipate a computer with a moth in it. https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k.jpg https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k....
- deleted 6y ago
  [deleted]
Nextgrid 6y ago
Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but I see a lot of companies (including startups with very few customers) going down the microservices route and exposing themselves to such a risk when there is no major upside beyond giving engineers lots of toys to play with and slapping the "microservices" and related buzzwords on their careers page.
- deleted 6y ago
  [deleted]
- bstar77 6y ago
  I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame large scale deployments that has sensible failover processes. The benefits you get are services that can be scaled independently, deployments that only affect isolated pieces of code, horizontal scaling, dockerized environemnts, etc. All of these advantages should exist in well designed systems, but systems that have been executed hastily will likely have critical problems crop up at some point.
  - please-reread 6y ago
    Did you mean boogeyman? https://www.dictionary.com/browse/boogeyman https://www.dictionary.com/browse/boogeyman
    - arthurcolle 6y ago
      Obviously he means boogeyman. Even on your page, it suggests another spelling. What a useless comment.
  - Nextgrid 6y ago
    I think that all other things considered equal, a microservices architecture has more moving parts and is more complex to learn than a monolith. The same applies for Kubernetes, it's one extra layer of abstraction and moving parts that you need to understand and keep into account when developing or making changes to an existing system. I am not saying that microservices is a problem for Slack (at their scale it can make sense), but I am expressing my overall concern about smaller companies going down the same route when their scale or the problem being solved doesn't justify it and they end up having to deal with the (self-inflicted) problems of a massively distributed system with no major upside as well as my personal opinion of why I feel uneasy working on systems where I don't have a full overview of how it works and its potential failure modes. When it comes to the benefits of microservices I am not sure whether those are all worthwhile considering the overhead and extra complexity of development on a microservices architecture. In a monolithic application, most data and functions you might need are just a function call away and you typically have one or a handful of databases to interact with, often abstracted away by an ORM. In a microservices architecture, you suddenly need to worry about serialization, authentication and communication between services (and its failure modes, etc) and might require coordinated changes across several services, each of which might use a different language/framework and deployment process. In terms of getting started, it has always been easier for me to work on a monolith where the codebase makes up for bad/no documentation because my IDE can resolve the majority of the symbols and allows me to see where the data I need lives and where it's being used. In a microservices architecture all of that goes out the window and you need to do a lot more manual "discovery" work searching through the documentation (if there is documentation, which is not a given) and manually figuring out the RPC calls because IDEs typically can't resolve cross-service communications. Running a monolithic application locally is a lot easier than a microservices architecture. For the former you can typically get away with just a database and cache server all running natively. The latter pretty much imposes a container-based stack where you are now running 10 databases, caches, reverse proxies and everything involved around service discovery, which adds yet another layer of abstraction and makes you spend more time on this useless plumbing than actually getting work done and delivering business value.
dahfizz 6y ago
Microservices are really cool™
- Polylactic_acid 6y ago
  This is far from a micro service problem. I work on a medium sized monolithic rails app and regularly I have issues due to not understanding the whole situation. Even when working in a small part of the app I know super well sometimes other devs change things so the way it works in my mind is not how it works anymore.
jzoch 6y ago
Its a load balancer bug. Thats it. Everything else functioned totally normally from what I can see. I think people are blowing this out of proportion saying everything is so complex or its all microservices oh no! This exact bug could have bit a large monolithic app running on a VM.
- rclayton 6y ago
  My sentiments exactly.
- a2tech 6y ago
  No, its not a load balancer bug. It was a bug with the home built program they built for managing config load ins TO the load balancer. The load balancer did exactly what it was instructed to do--but the slack developed app for helping the load balancer deal with their overly complicated back end discovery problems failed to keep the load balancers updated with accurate information on backend servers available for serving.
lloydatkinson 6y ago
Possibly, but I think it's equally if not more likely a result of "agile" and teams not communicating due to product owners and the like not wanting other teams to work together etc.
ueli 6y ago
Woods' Theorem: As the complexity of a system increases, the accuracy of any single agent's own model of that system decreases rapidly. from: https://snafucatchers.github.io/ https://snafucatchers.github.io/

by vondur 6y ago

staticassertion 6y ago

Feels like you've just described every single bug ever.

m463 6y ago
Even at the start I'm sure Grace Hopper didn't anticipate a computer with a moth in it. https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k.jpg https://upload.wikimedia.org/wikipedia/commons/8/8a/H96566k....
deleted 6y ago
[deleted]

Nextgrid 6y ago

Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system. To be fair to Slack, at their scale, lots of moving parts might make sense, but I see a lot of companies (including startups with very few customers) going down the microservices route and exposing themselves to such a risk when there is no major upside beyond giving engineers lots of toys to play with and slapping the "microservices" and related buzzwords on their careers page.

deleted 6y ago
[deleted]
bstar77 6y ago
I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer. Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame large scale deployments that has sensible failover processes. The benefits you get are services that can be scaled independently, deployments that only affect isolated pieces of code, horizontal scaling, dockerized environemnts, etc. All of these advantages should exist in well designed systems, but systems that have been executed hastily will likely have critical problems crop up at some point.
- please-reread 6y ago
  Did you mean boogeyman? https://www.dictionary.com/browse/boogeyman https://www.dictionary.com/browse/boogeyman
  - arthurcolle 6y ago
    Obviously he means boogeyman. Even on your page, it suggests another spelling. What a useless comment.
- Nextgrid 6y ago
  I think that all other things considered equal, a microservices architecture has more moving parts and is more complex to learn than a monolith. The same applies for Kubernetes, it's one extra layer of abstraction and moving parts that you need to understand and keep into account when developing or making changes to an existing system. I am not saying that microservices is a problem for Slack (at their scale it can make sense), but I am expressing my overall concern about smaller companies going down the same route when their scale or the problem being solved doesn't justify it and they end up having to deal with the (self-inflicted) problems of a massively distributed system with no major upside as well as my personal opinion of why I feel uneasy working on systems where I don't have a full overview of how it works and its potential failure modes. When it comes to the benefits of microservices I am not sure whether those are all worthwhile considering the overhead and extra complexity of development on a microservices architecture. In a monolithic application, most data and functions you might need are just a function call away and you typically have one or a handful of databases to interact with, often abstracted away by an ORM. In a microservices architecture, you suddenly need to worry about serialization, authentication and communication between services (and its failure modes, etc) and might require coordinated changes across several services, each of which might use a different language/framework and deployment process. In terms of getting started, it has always been easier for me to work on a monolith where the codebase makes up for bad/no documentation because my IDE can resolve the majority of the symbols and allows me to see where the data I need lives and where it's being used. In a microservices architecture all of that goes out the window and you need to do a lot more manual "discovery" work searching through the documentation (if there is documentation, which is not a given) and manually figuring out the RPC calls because IDEs typically can't resolve cross-service communications. Running a monolithic application locally is a lot easier than a microservices architecture. For the former you can typically get away with just a database and cache server all running natively. The latter pretty much imposes a container-based stack where you are now running 10 databases, caches, reverse proxies and everything involved around service discovery, which adds yet another layer of abstraction and makes you spend more time on this useless plumbing than actually getting work done and delivering business value.

dahfizz 6y ago

Microservices are really cool™

Polylactic_acid 6y ago
This is far from a micro service problem. I work on a medium sized monolithic rails app and regularly I have issues due to not understanding the whole situation. Even when working in a small part of the app I know super well sometimes other devs change things so the way it works in my mind is not how it works anymore.

jzoch 6y ago

Its a load balancer bug. Thats it. Everything else functioned totally normally from what I can see. I think people are blowing this out of proportion saying everything is so complex or its all microservices oh no! This exact bug could have bit a large monolithic app running on a VM.

rclayton 6y ago
My sentiments exactly.
a2tech 6y ago
No, its not a load balancer bug. It was a bug with the home built program they built for managing config load ins TO the load balancer. The load balancer did exactly what it was instructed to do--but the slack developed app for helping the load balancer deal with their overly complicated back end discovery problems failed to keep the load balancers updated with accurate information on backend servers available for serving.

lloydatkinson 6y ago

Possibly, but I think it's equally if not more likely a result of "agile" and teams not communicating due to product owners and the like not wanting other teams to work together etc.

ueli 6y ago

Woods' Theorem: As the complexity of a system increases, the accuracy of any single agent's own model of that system decreases rapidly. from: https://snafucatchers.github.io/ https://snafucatchers.github.io/