5 ms·

> You push data consistency concerns out of the database and between service boundaries. Sing that from the rooftops. That is exactly my observation as well. A

> You push data consistency concerns out of the database and between service boundaries. Sing that from the rooftops. That is exactly my observation as well. All the vanilla "track some resource"-style webapps I've worked on were never designed to cope with a consistency boundary that spans across service boundaries. Turning a monolith into distributed services is hard for that reason - you have to redesign your data access to ensure that consistency boundaries don't span across multiple services. If you don't do that, then you have to learn to cope with eventual consistency; in my experience, most people just don't think that way. I know I have trouble with it. Surely I'm not the only one.

noir_lord 8y ago
They also throw Single Source of Truth out the window way too often.
stickfigure 8y ago
It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path. I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. I think it's one of those things that gives the illusion of productivity - "I did all this work, and now I have left-pad-as-a-service running! Look at the little green status light on the cool dashboard we spent the last couple months building!" Programmers get excited about little happily running services, these are "real" to them. Customers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.
- ThJ 8y ago
  When I was designing a microservices architecture for a former employee, the justification for it was security. We needed separation of the components in the system because we were going to be handling money. It used ZeroMQ for communication, because I wanted something more lightweight and falut-resistant for message passing (in such a sensitive application, we didn't want to trust the network). Although it was of course fun to design, microservices weren't my first choice. It just made sense to us to use it in that particular scenario.
- hosh 8y ago
  Sometimes, I've seen a lack of regard for data consistency within a monolith. That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions. On the other hand, I've seen some pretty flagrant disregard for it just because there are not atomicity guarantees. Microservices makes reasoning on that harder.
  - stefano 8y ago
    > That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions I'd argue that's on the developer, if he was the one to choose a database that doesn't support transactions, and then didn't implement application-level transactions (which is very hard to do correctly).
    - shub 8y ago
      Txns are just hard in general but mature RDBMS have spent a huge number of man-years on getting it right. After the time I've spent on Couchbase, Mongo, Elasticsearch, I doubt I'll ever use something non-transactional for anything OLTP or OLTP-adjacent. If Postgres or MySQL can't scale up to handle it, make them...or get a new job. Scale is a PITA.
  - thermodynthrway 8y ago
    I've seen it on monoliths too. We run a pretty large app just fine with the "nuclear option". Database transaction isolation level serial. Makes it impossible to have inconsistencies on the database level, and on some RDBMS like PostGres the performance impact is small
- madamelic 8y ago
  >It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path. If two microservices have to share databases, they shouldn't be microservices. One microservice should have write access to one database and preferably, all read requests run through that microservice for exactly the reason you mentioned. >I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. If you're running microservices between regions and communicating with each other outside of the network it is living in, you're probably doing it wrong. Microservices shouldn't have to incur the cost of going from SF to China and back. If one lives in SF, all should and you can co-locate the entire ecosystem (+1 for "only huge companies with big requirements should do microservices") >ustomers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them. Again, that is an example of microservices gone wrong. You'll have the same amount of changes even in a monolith and I'd argue adding new features is safer in microservices (No worries of causing side effects, etc). I will give you +1 on that anyway because I designed a "microservice" that ended up being 3 microservices because of dumb requirements. It probably could've been a monolith quite happily.
  - stickfigure 8y ago
    My problem with microservices is the word 'micro'. It should just be services. Problem domains (along with organizational structures) inherently create natural architectural boundaries... certain bits of data, computation, transactional logic, and programming skill just naturally "clump" together. Microservices ignore this natural order. The main driving architectural principle seems to be "I'm having trouble with my event-driven dynamically-typed metaprogrammed ball-of-mud, so we need more services!".
    - neuromantik8086 8y ago
      It took me longer than it probably should have to realize that the term microservices was made by analogy to the term microkernel. I think that part of my main issue with the "microservices" is that it conflates highly technical semantics with word forms that are a bit more wishy washy in meaning (i.e., a service is something formed more of human perception, not rooted directly in operating system abstractions).
- w4tson 8y ago
  Yes! But it’s worse again. Because some bright spark comes along and says “hey there’s some commonality between these n services. I’ll make a shared library that will remove that duplicated code. Then before you know it there are a dozen more shared libraries and you have the distributed monolith. Then either have to Stand up every micro service every time integration test or make changes and hope for the best.
- thermodynthrway 8y ago
  I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" that comb the database for various inconsistencies that inevitably show up. You can't have consistent microservices without distributed transactions. If a service gets called, and inside that call, it calls 3 others, you need to have a roll back mechanism that handles any of them failing in any order. If you write to the first service and the second two fail, you need to write a second "undo" call to keep consistent. Worse, this "undo state" needs to be kept transactionally consistent in case it's your service that dies after the first call. In reality, nobody does this, so they're always one service crash away from the whole system corrupting the hell out of itself. Since the state is distributed, good luck making everything right again. Microservices are insane. Nobody that knows database concepts well should go near them
  - davewritescode 8y ago
    Data consistency is one of those things that sounds like it matters but often doesn't. There's not much in Netflix's platform that screams data consistency is an ultra high priority. Any application that deploys active/active multi-region is by definition going to encounter scenarios where data loss is possible. There's just no way around CAP. I'd venture a guess that most applications have all sorts of race conditions that could cause data corruption. The fact of the matter is that almost nobody notices or even cares.
    - stickfigure 8y ago
      A little birdie who worked at Grab once told me that they pretty much don't use transactions anywhere. So... maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them. The next time you're standing in a monsoon waiting for the "coming" car that never comes, remember this post :-) Aside from that, I've found that even in non-mission-critical scenarios ("it's just porn!") it's incredibly convenient to have a limited number of states the system can be in. It makes debugging easier and reduces the number of edge cases ("why is this null??") you have to handle.
radisb 8y ago
I remember using BEA Tuxedo 15 years ago and I can't help wondering if theoretically one could do microservices using it or some OLTP to achieve consistency when needed. Has it progressed over the years? Is there any free alternative to it? Or is it dead tech? When I used it it was hard and too much extra work, but it did the job successfully.

noir_lord 8y ago

They also throw Single Source of Truth out the window way too often.

stickfigure 8y ago

It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path. I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. I think it's one of those things that gives the illusion of productivity - "I did all this work, and now I have left-pad-as-a-service running! Look at the little green status light on the cool dashboard we spent the last couple months building!" Programmers get excited about little happily running services, these are "real" to them. Customers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.

ThJ 8y ago
When I was designing a microservices architecture for a former employee, the justification for it was security. We needed separation of the components in the system because we were going to be handling money. It used ZeroMQ for communication, because I wanted something more lightweight and falut-resistant for message passing (in such a sensitive application, we didn't want to trust the network). Although it was of course fun to design, microservices weren't my first choice. It just made sense to us to use it in that particular scenario.
hosh 8y ago
Sometimes, I've seen a lack of regard for data consistency within a monolith. That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions. On the other hand, I've seen some pretty flagrant disregard for it just because there are not atomicity guarantees. Microservices makes reasoning on that harder.
- stefano 8y ago
  > That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions I'd argue that's on the developer, if he was the one to choose a database that doesn't support transactions, and then didn't implement application-level transactions (which is very hard to do correctly).
  - shub 8y ago
    Txns are just hard in general but mature RDBMS have spent a huge number of man-years on getting it right. After the time I've spent on Couchbase, Mongo, Elasticsearch, I doubt I'll ever use something non-transactional for anything OLTP or OLTP-adjacent. If Postgres or MySQL can't scale up to handle it, make them...or get a new job. Scale is a PITA.
- thermodynthrway 8y ago
  I've seen it on monoliths too. We run a pretty large app just fine with the "nuclear option". Database transaction isolation level serial. Makes it impossible to have inconsistencies on the database level, and on some RDBMS like PostGres the performance impact is small
madamelic 8y ago
>It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path. If two microservices have to share databases, they shouldn't be microservices. One microservice should have write access to one database and preferably, all read requests run through that microservice for exactly the reason you mentioned. >I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. If you're running microservices between regions and communicating with each other outside of the network it is living in, you're probably doing it wrong. Microservices shouldn't have to incur the cost of going from SF to China and back. If one lives in SF, all should and you can co-locate the entire ecosystem (+1 for "only huge companies with big requirements should do microservices") >ustomers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them. Again, that is an example of microservices gone wrong. You'll have the same amount of changes even in a monolith and I'd argue adding new features is safer in microservices (No worries of causing side effects, etc). I will give you +1 on that anyway because I designed a "microservice" that ended up being 3 microservices because of dumb requirements. It probably could've been a monolith quite happily.
- stickfigure 8y ago
  My problem with microservices is the word 'micro'. It should just be services. Problem domains (along with organizational structures) inherently create natural architectural boundaries... certain bits of data, computation, transactional logic, and programming skill just naturally "clump" together. Microservices ignore this natural order. The main driving architectural principle seems to be "I'm having trouble with my event-driven dynamically-typed metaprogrammed ball-of-mud, so we need more services!".
  - neuromantik8086 8y ago
    It took me longer than it probably should have to realize that the term microservices was made by analogy to the term microkernel. I think that part of my main issue with the "microservices" is that it conflates highly technical semantics with word forms that are a bit more wishy washy in meaning (i.e., a service is something formed more of human perception, not rooted directly in operating system abstractions).
w4tson 8y ago
Yes! But it’s worse again. Because some bright spark comes along and says “hey there’s some commonality between these n services. I’ll make a shared library that will remove that duplicated code. Then before you know it there are a dozen more shared libraries and you have the distributed monolith. Then either have to Stand up every micro service every time integration test or make changes and hope for the best.
thermodynthrway 8y ago
I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" that comb the database for various inconsistencies that inevitably show up. You can't have consistent microservices without distributed transactions. If a service gets called, and inside that call, it calls 3 others, you need to have a roll back mechanism that handles any of them failing in any order. If you write to the first service and the second two fail, you need to write a second "undo" call to keep consistent. Worse, this "undo state" needs to be kept transactionally consistent in case it's your service that dies after the first call. In reality, nobody does this, so they're always one service crash away from the whole system corrupting the hell out of itself. Since the state is distributed, good luck making everything right again. Microservices are insane. Nobody that knows database concepts well should go near them
- davewritescode 8y ago
  Data consistency is one of those things that sounds like it matters but often doesn't. There's not much in Netflix's platform that screams data consistency is an ultra high priority. Any application that deploys active/active multi-region is by definition going to encounter scenarios where data loss is possible. There's just no way around CAP. I'd venture a guess that most applications have all sorts of race conditions that could cause data corruption. The fact of the matter is that almost nobody notices or even cares.
  - stickfigure 8y ago
    A little birdie who worked at Grab once told me that they pretty much don't use transactions anywhere. So... maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them. The next time you're standing in a monsoon waiting for the "coming" car that never comes, remember this post :-) Aside from that, I've found that even in non-mission-critical scenarios ("it's just porn!") it's incredibly convenient to have a limited number of states the system can be in. It makes debugging easier and reduces the number of edge cases ("why is this null??") you have to handle.

radisb 8y ago

I remember using BEA Tuxedo 15 years ago and I can't help wondering if theoretically one could do microservices using it or some OLTP to achieve consistency when needed. Has it progressed over the years? Is there any free alternative to it? Or is it dead tech? When I used it it was hard and too much extra work, but it did the job successfully.