6 ms·

> However, your codebase has now to deal with network and multiple processes. Here's the thing I see repeatedly called out as a negative, but it's a positive!

> However, your codebase has now to deal with network and multiple processes. Here's the thing I see repeatedly called out as a negative, but it's a positive! Processes and networks are amazing abstractions. They force you to not share memory on a single system, they encourage you to focus on how you communicate, they give you scale and failure isolation, for force you to handle the fact that a called subroutine might fail because it's across a network. > f your codebase has failed to achieve modularity with tools such as functions and packages, it will not magically succeed by adding network layers and binary boundaries inside it Functions allow shared state, they don't isolate errors. Processes over networks do. That's a massive difference. If you read up on the fundamental papers regarding software reliability this is something that's brought up ad nauseum. > (this might be the reason why the video game industry is still safe from this whole microservice trend). Performance is more complex than this. For a video game system latency might be the dominating criteria. For a data processing service it might be throughput, or the ability to scale up and down. For many, microservices have the performance characteristics that they need, because many tasks are not latency sensitive, or the latency sensitive part can be handled separately. > would argue that by having to anticipate the traffic for each microservice specifically, we will face more problem because one part of the app can't compensate for another one. I would argue that if you're manually scaling things then you're doing it wrong. Your whole system should grow and shrink has needed.

mpfundstein 6y ago
> Processes and networks are amazing abstractions. They force you to not share memory on a single system, they encourage you to focus on how you communicate, they give you scale and failure isolation, for force you to handle the fact that a called subroutine might fail because it's across a network. Stop on! Its so much easier to enforce separation of concern if there is an actual physical barrier. Its just to easy to just slip bad decision through peer review. So I totally agree
- mattupstate 6y ago
  Stop off! Drop a bunch "microservices" into the same network without any access control and you don't have a physical barrier at all! In fact, it becomes even harder to have a clue as to what's interfacing with what unless you can observe your inter process traffic.
  - staticassertion 6y ago
    This is a completely unrelated issue. Microservices have nothing to do with access control. If anything, they should lend themselves to more fine grained control, but again, microservice architecture says nothing about it.
    - maattdd 6y ago
      This is exactly the point of the original article by Monzo. Microservices don't provide any tool to deal with access control, thus you have to implement it yourself. I rather avoid that when I can.
    - mattupstate 6y ago
      Unrelated? I' beg to differ. More to the point, all systems have an access control model. It could be explicit or implicit depending on how the system is built. Regardless, it defines the barriers and boundaries between everything in the system. The comment suggests that just by separating things in the popular sense of microservices results in a barrier to enforce separation of concerns. That's how I read it, at least, and I find that to be is misleading. This is the crux I see around popular discourse of microservices. It's often presented without a broader context.
  - yellowapple 6y ago
    > without any access control The point, I reckon, is that you should be implementing access control.
- karatestomp 6y ago
  > Stop on! Not sure if typo or joke about out-of-order message processing in distributed systems. > just to easy to just slip And maybe also about the impossibility of exactly-once message delivery?
amelius 6y ago
> They force you to not share memory on a single system, they encourage you to focus on how you communicate, they give you scale and failure isolation, for force you to handle the fact that a called subroutine might fail because it's across a network. The problem: distributed systems are hard to get right. Better stay away from them unless you really need them, AND you have the time/resources to implement them correctly. The benefits of microservices are a bad excuse, most of the time.
- anyfoo 6y ago
  Exactly, distributed systems are hard. Very hard. It's an entire field of study. Don't opt for something very, very hard unless you have to.
  - tonyarkles 6y ago
    Heh, I have a masters degree focused on distributed systems. My thesis was about tracing and debugging in microservice-style systems. I generally write monoliths, on purpose. The massive overheads and debugging nightmares are not worth it most of the time. Global state still gets pushed out into backend services (redis, postgres) and I can still scale horizontally all day, but there’s no crazy chain of backend interservice HTTP requests to cause no end of chaos :)
    - harpratap 6y ago
      Do you mind if you can link me to your thesis? I am actually looking into tracing systems right now.
      - tonyarkles 6y ago
        Honestly, you're probably better off looking at https://opentracing.io/ https://opentracing.io/. For a bit of context, I started working on my thesis in 2009 when there weren't many commercial tools available for doing this kind of tracing; by time I was done, there were a ton of open source and commercial tools available. Thesis is here: https://harvest.usask.ca/handle/10388/ETD-2015-06-2185 https://harvest.usask.ca/handle/10388/ETD-2015-06-2185
mixedCase 6y ago
> for force you to handle the fact that a called subroutine might fail because it's across a network. That just adds one failure mode to the list of failure modes people ignore due to the happy-path development that languages with "unchecked exceptions as default error handling" encourage. > Functions allow shared state, they don't isolate errors. Processes over networks do. That's a massive difference. Except not, because "just dump that on a database/kv-store" is an all-too-common workaround chosen as an easy way out. This problem is instead tackled by things such as functionally pure languages such as Haskell and Rust's borrow checker, and only up to a certain degree at which point it's still back into the hands of the programmer's experience; though they do help a ton.
- staticassertion 6y ago
  > That just adds one failure mode to the list of failure modes people ignore due to the happy-path development that languages with "unchecked exceptions as default error handling" encourage. There are only two meaningful failure modes - persistent and transient. So adding another transient failure (network partition) is not extra work to handle. > Except not, because "just dump that on a database/kv-store" is an all-too-common workaround chosen as an easy way out. Just to be clear, microservices are not just separate binaries on a network. If you're not following the actual patterns of microservice architecture... you're just complaining about something else.
  - unchar1 6y ago
    >Just to be clear, microservices are not just separate binaries on a network. If you're not following the actual patterns of microservice architecture... you're just complaining about something else So what you're saying is that the way to avoid this problem in a microservice architecture, is to be disciplined and follow the right patterns. Then couldn't I just follow the same patterns in a modular monolith (eg: avoid shared state, make sure errors are handled properly, etc) and get the bulk of the benefits, without having to introduce network related problems into the mix?
    - fennecfoxen 6y ago
mic47 6y ago
> Functions allow shared state, they don't isolate errors. So why not use Haskell (or other pure language)? It's pure, so functions don't share state. And you don't have to replace function call with network call.
- staticassertion 6y ago
  You'll never achieve the level of operational stability of an air gap that you would without a strictly in process system. Shared state is part of it. Isolation of failure is another. Your Haskell code can still blow up. Immutability makes that way easier to recover from, but it's just one part. Of course, microservices are much more than processes over a network, they're a discipline. And I think one can go much further than microservices as well - microservice architecture doesn't tell you how processes should communicate, it's more focused on how they're split up, and leaves the protocol to you. Good protocol design is critical.
twic 6y ago
> If you read up on the fundamental papers regarding software reliability this is something that's brought up ad nauseum. I don't believe there are any papers that show that adding network hops to an application makes it more reliable. I would be extremely interested in any references you could provide.
- staticassertion 6y ago
  Not the network bit, the separation of state and the process abstraction. https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf Here's one paper that discusses building reliable systems from two abstractions - the process and transaction.
deleted 6y ago
[deleted]
m463 6y ago
I'm reminded of Jeff Bezos and his push for services over web protocols. this is the closest reference I can find: http://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf http://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf
- redis_mlc 6y ago
  The point of his edict was that he wanted to reduce silos, not to "adopt web prootcols."
DSMan195276 6y ago
> Functions allow shared state, they don't isolate errors. Processes over networks do. That's a massive difference. > > If you read up on the fundamental papers regarding software reliability this is something that's brought up ad nauseum. I think you missed the point - just using separate processes does not guarantee you separate errors and state between services, there's lots of ways to get 'around' that. What if the two services talk to the same database/service? What if there's weird co-dependencies and odd workarounds for shared state? What if one service failing means the entire thing grinds to a halt or data is screwed up? Now that said, yes, if you use good development practices and have a good architecture microservices can work quite well, but if you were capable of that you probably wouldn't have created a non-microservice ball of mud. And if you're currently unable to fix your existing ball of mud, attempting to make it distributed is likely going to result in you adding more distributed mud instead. In other words, the problem here isn't really a technical one, it's a process one. And using their current failing processes to make microservices is just going to make worse mud, because they haven't yet figured out how to deal with their existing mud.
- staticassertion 6y ago
  > just using separate processes Microservice architecture is not just using separate processes.
  - DSMan195276 6y ago
    If you're talking about 'over a network', sorry I left that out, but I'd argue it's largely implied. That said, if you add 'over a network' there nothing that I said changes.
    - staticassertion 6y ago
      My point is that microservice architecture is not just the singular pattern of "code talking over the internet", but a collection of patterns and techniques, focusing on when and where to split up code, and to focusing on the communication strategy you use. You can 'get around' microservice architecture by not doing it. The point is that if you're familiar with it it's a lot easier to 'accidentally' be successful - or at least, that's the proposition.
peterwwillis 6y ago
> They force you to not share memory on a single system, So instead people use a single SQL database between 20 microservices. > give you scale and failure isolation, Only if you configure and test them properly, and they actually tend to increase failure and make it harder to isolate its origin (hello distributed tracing) > force you to handle the fact that a called subroutine might fail because it's across a network They don't force that at all. It's good when people do handle that, but often they don't. > I would argue that if you're manually scaling things then you're doing it wrong And I would argue that if people are given a default choice of doing the wrong thing, they will do that wrong thing, until someone forces them not to. Microservices allow people to make hundreds of bad decisions because nobody else can tell if they're making bad decisions unless they audit all of the code. Usually the only people auditing code are the people writing it, and usually they have no special incentive to make the best decisions, they just need to ship a feature.
- daotoad 6y ago
  If your teams are defaulting to doing the wrong thing in every case, you are having a crisis of leadership. The problem might lie in your technical leadership (e.g. making bad design decisions and failing to learn from experience) or from product leadership (e.g. demanding impossibly short turnaround on work, leading to reliance on solutions that are purely short term focused).
  - peterwwillis 6y ago
    Well, I'd answer that with two things 1) good leadership is probably rare, and 2) doing the wrong thing isn't always wrong. In my experience, the best leadership comes from people who don't hyper-focus on one thing, but are constantly thinking about many different things, mostly centered around the customer experience (because that's the whole point of this thing: to make/sell/run a product, not to have perfect software craftsmanship). I find those people rare and usually not working at lower levels of a product. In a sense, doing the wrong thing is perfectly fine if you don't need to be doing the right thing to provide a great customer experience. Of course maintainability, availability, recoverability, extensibility, etc are all necessary considerations, but even within these you can often do the wrong thing and still be fine. I have seen some truly ugly stuff in production yet the customers seemed happy. (though there were probably some chronically sleepless engineers keeping it alive, which sucks and shouldn't happen) I don't like microservices because of how poorly they're usually implemented by default. But at the same time, even poorly implemented microservices give some great benefits (more rapid/agile work, reusability, separation of concerns, BU independence), even if you're struggling against some harmful aspects.
deleted 6y ago
[deleted]
pjmlp 6y ago
Plenty of languages allow to write modular code, all the way back to early 80's. The developers that aren't able to write modular code, are just going to write spaghetti network code, with the added complexity of distributed computing.

mpfundstein 6y ago

> Processes and networks are amazing abstractions. They force you to not share memory on a single system, they encourage you to focus on how you communicate, they give you scale and failure isolation, for force you to handle the fact that a called subroutine might fail because it's across a network. Stop on! Its so much easier to enforce separation of concern if there is an actual physical barrier. Its just to easy to just slip bad decision through peer review. So I totally agree

mattupstate 6y ago
Stop off! Drop a bunch "microservices" into the same network without any access control and you don't have a physical barrier at all! In fact, it becomes even harder to have a clue as to what's interfacing with what unless you can observe your inter process traffic.
- staticassertion 6y ago
  This is a completely unrelated issue. Microservices have nothing to do with access control. If anything, they should lend themselves to more fine grained control, but again, microservice architecture says nothing about it.
  - maattdd 6y ago
    This is exactly the point of the original article by Monzo. Microservices don't provide any tool to deal with access control, thus you have to implement it yourself. I rather avoid that when I can.
  - mattupstate 6y ago
    Unrelated? I' beg to differ. More to the point, all systems have an access control model. It could be explicit or implicit depending on how the system is built. Regardless, it defines the barriers and boundaries between everything in the system. The comment suggests that just by separating things in the popular sense of microservices results in a barrier to enforce separation of concerns. That's how I read it, at least, and I find that to be is misleading. This is the crux I see around popular discourse of microservices. It's often presented without a broader context.
- yellowapple 6y ago
  > without any access control The point, I reckon, is that you should be implementing access control.
karatestomp 6y ago
> Stop on! Not sure if typo or joke about out-of-order message processing in distributed systems. > just to easy to just slip And maybe also about the impossibility of exactly-once message delivery?

amelius 6y ago

> They force you to not share memory on a single system, they encourage you to focus on how you communicate, they give you scale and failure isolation, for force you to handle the fact that a called subroutine might fail because it's across a network. The problem: distributed systems are hard to get right. Better stay away from them unless you really need them, AND you have the time/resources to implement them correctly. The benefits of microservices are a bad excuse, most of the time.

anyfoo 6y ago
Exactly, distributed systems are hard. Very hard. It's an entire field of study. Don't opt for something very, very hard unless you have to.
- tonyarkles 6y ago
  Heh, I have a masters degree focused on distributed systems. My thesis was about tracing and debugging in microservice-style systems. I generally write monoliths, on purpose. The massive overheads and debugging nightmares are not worth it most of the time. Global state still gets pushed out into backend services (redis, postgres) and I can still scale horizontally all day, but there’s no crazy chain of backend interservice HTTP requests to cause no end of chaos :)
  - harpratap 6y ago
    Do you mind if you can link me to your thesis? I am actually looking into tracing systems right now.
    - tonyarkles 6y ago
      Honestly, you're probably better off looking at https://opentracing.io/ https://opentracing.io/. For a bit of context, I started working on my thesis in 2009 when there weren't many commercial tools available for doing this kind of tracing; by time I was done, there were a ton of open source and commercial tools available. Thesis is here: https://harvest.usask.ca/handle/10388/ETD-2015-06-2185 https://harvest.usask.ca/handle/10388/ETD-2015-06-2185

mixedCase 6y ago

> for force you to handle the fact that a called subroutine might fail because it's across a network. That just adds one failure mode to the list of failure modes people ignore due to the happy-path development that languages with "unchecked exceptions as default error handling" encourage. > Functions allow shared state, they don't isolate errors. Processes over networks do. That's a massive difference. Except not, because "just dump that on a database/kv-store" is an all-too-common workaround chosen as an easy way out. This problem is instead tackled by things such as functionally pure languages such as Haskell and Rust's borrow checker, and only up to a certain degree at which point it's still back into the hands of the programmer's experience; though they do help a ton.

staticassertion 6y ago
> That just adds one failure mode to the list of failure modes people ignore due to the happy-path development that languages with "unchecked exceptions as default error handling" encourage. There are only two meaningful failure modes - persistent and transient. So adding another transient failure (network partition) is not extra work to handle. > Except not, because "just dump that on a database/kv-store" is an all-too-common workaround chosen as an easy way out. Just to be clear, microservices are not just separate binaries on a network. If you're not following the actual patterns of microservice architecture... you're just complaining about something else.
- unchar1 6y ago
  >Just to be clear, microservices are not just separate binaries on a network. If you're not following the actual patterns of microservice architecture... you're just complaining about something else So what you're saying is that the way to avoid this problem in a microservice architecture, is to be disciplined and follow the right patterns. Then couldn't I just follow the same patterns in a modular monolith (eg: avoid shared state, make sure errors are handled properly, etc) and get the bulk of the benefits, without having to introduce network related problems into the mix?
  - fennecfoxen 6y ago

mic47 6y ago

> Functions allow shared state, they don't isolate errors. So why not use Haskell (or other pure language)? It's pure, so functions don't share state. And you don't have to replace function call with network call.

staticassertion 6y ago
You'll never achieve the level of operational stability of an air gap that you would without a strictly in process system. Shared state is part of it. Isolation of failure is another. Your Haskell code can still blow up. Immutability makes that way easier to recover from, but it's just one part. Of course, microservices are much more than processes over a network, they're a discipline. And I think one can go much further than microservices as well - microservice architecture doesn't tell you how processes should communicate, it's more focused on how they're split up, and leaves the protocol to you. Good protocol design is critical.

twic 6y ago

> If you read up on the fundamental papers regarding software reliability this is something that's brought up ad nauseum. I don't believe there are any papers that show that adding network hops to an application makes it more reliable. I would be extremely interested in any references you could provide.

staticassertion 6y ago
Not the network bit, the separation of state and the process abstraction. https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf Here's one paper that discusses building reliable systems from two abstractions - the process and transaction.

deleted 6y ago

[deleted]

m463 6y ago

I'm reminded of Jeff Bezos and his push for services over web protocols. this is the closest reference I can find: http://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf http://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf

redis_mlc 6y ago
The point of his edict was that he wanted to reduce silos, not to "adopt web prootcols."

DSMan195276 6y ago

> Functions allow shared state, they don't isolate errors. Processes over networks do. That's a massive difference. > > If you read up on the fundamental papers regarding software reliability this is something that's brought up ad nauseum. I think you missed the point - just using separate processes does not guarantee you separate errors and state between services, there's lots of ways to get 'around' that. What if the two services talk to the same database/service? What if there's weird co-dependencies and odd workarounds for shared state? What if one service failing means the entire thing grinds to a halt or data is screwed up? Now that said, yes, if you use good development practices and have a good architecture microservices can work quite well, but if you were capable of that you probably wouldn't have created a non-microservice ball of mud. And if you're currently unable to fix your existing ball of mud, attempting to make it distributed is likely going to result in you adding more distributed mud instead. In other words, the problem here isn't really a technical one, it's a process one. And using their current failing processes to make microservices is just going to make worse mud, because they haven't yet figured out how to deal with their existing mud.

staticassertion 6y ago
> just using separate processes Microservice architecture is not just using separate processes.
- DSMan195276 6y ago
  If you're talking about 'over a network', sorry I left that out, but I'd argue it's largely implied. That said, if you add 'over a network' there nothing that I said changes.
  - staticassertion 6y ago
    My point is that microservice architecture is not just the singular pattern of "code talking over the internet", but a collection of patterns and techniques, focusing on when and where to split up code, and to focusing on the communication strategy you use. You can 'get around' microservice architecture by not doing it. The point is that if you're familiar with it it's a lot easier to 'accidentally' be successful - or at least, that's the proposition.

peterwwillis 6y ago

> They force you to not share memory on a single system, So instead people use a single SQL database between 20 microservices. > give you scale and failure isolation, Only if you configure and test them properly, and they actually tend to increase failure and make it harder to isolate its origin (hello distributed tracing) > force you to handle the fact that a called subroutine might fail because it's across a network They don't force that at all. It's good when people do handle that, but often they don't. > I would argue that if you're manually scaling things then you're doing it wrong And I would argue that if people are given a default choice of doing the wrong thing, they will do that wrong thing, until someone forces them not to. Microservices allow people to make hundreds of bad decisions because nobody else can tell if they're making bad decisions unless they audit all of the code. Usually the only people auditing code are the people writing it, and usually they have no special incentive to make the best decisions, they just need to ship a feature.

daotoad 6y ago
If your teams are defaulting to doing the wrong thing in every case, you are having a crisis of leadership. The problem might lie in your technical leadership (e.g. making bad design decisions and failing to learn from experience) or from product leadership (e.g. demanding impossibly short turnaround on work, leading to reliance on solutions that are purely short term focused).
- peterwwillis 6y ago
  Well, I'd answer that with two things 1) good leadership is probably rare, and 2) doing the wrong thing isn't always wrong. In my experience, the best leadership comes from people who don't hyper-focus on one thing, but are constantly thinking about many different things, mostly centered around the customer experience (because that's the whole point of this thing: to make/sell/run a product, not to have perfect software craftsmanship). I find those people rare and usually not working at lower levels of a product. In a sense, doing the wrong thing is perfectly fine if you don't need to be doing the right thing to provide a great customer experience. Of course maintainability, availability, recoverability, extensibility, etc are all necessary considerations, but even within these you can often do the wrong thing and still be fine. I have seen some truly ugly stuff in production yet the customers seemed happy. (though there were probably some chronically sleepless engineers keeping it alive, which sucks and shouldn't happen) I don't like microservices because of how poorly they're usually implemented by default. But at the same time, even poorly implemented microservices give some great benefits (more rapid/agile work, reusability, separation of concerns, BU independence), even if you're struggling against some harmful aspects.

deleted 6y ago

[deleted]

pjmlp 6y ago

Plenty of languages allow to write modular code, all the way back to early 80's. The developers that aren't able to write modular code, are just going to write spaghetti network code, with the added complexity of distributed computing.