8 ms·

The last microservice architecture I worked on consisted of 7 python repositories that shared one "standard library" repository. Something as simple as adding a

The last microservice architecture I worked on consisted of 7 python repositories that shared one "standard library" repository. Something as simple as adding an additional argument to a function required a PR to 7 repos and sign off on each. When it came to release we had to do a mass docker swarm restart and spin up because the giant entanglement of micro-services was really just a monolithic uncontrolled Cthulhu in disguise. The business revolved around filling out a form and PDF generations of said form. I felt like I got no work done in a year and so I left

umpalumpaaa 5y ago
Have you considered not sharing code between services?
- glouwbug 5y ago
  Yes, I came to learn that microservices would probably greatly excel at sending JSON strings back and forth. Then I wondered why we weren't just all using Erlang
  - umpalumpaaa 5y ago
    I don’t understand why I got the downvotes… I was taught that sharing code between services should only be done in circumstances in which there is already a strong connection between them - for example by being owned by the same team. In the organization where I initial learned how to deal with microservices even direct calls via HTTP between services was a no-go and only used in rare circumstances and if then only temporary. I am not sure if this is the way to do it since I have a sample size of one - but we had about 500 services to manage and in the end it worked okay and I never saw something you described. That’s why I wanted to know if you have considered just not sharing code.
    - archduck 5y ago
      Regarding why you got the downvotes, I have a guess: "Have you (maybe) considered not ..." usually gets followed by something that's transparently stupid or incompetent or otherwise undesirable in the given context. For example: "Have you considered NOT publicly yelling at your coworkers?" Obviously that's not what you were saying, but it may have read like it. It's a fantastic INDIRECT way of telling someone they did something stupid. And because stupid actions map so well onto their doers, it's also a fantastic way to insult people. As an aside: this sort of indirection - asking a question to assert something - it's not peculiar to English, but it's suuper common in English-language expressions (which is, in fact, due to the habitual indirection of the English - at least according to a linguistics professor I once had.) The kicker is: because it's an ironic construction, it’s likely going to require more effort to process: the presence of irony means that what is actually meant by the speaker is not encoded literally, but rather must be interpreted - derived from what WAS said - and irony (and other ways of flouting literal language) yields a disjunction (x || y || z). It puts the listener into a role where their recognition of intent and ability to joke around are tested - and the correct interpretation is up in the air. So while it's possible to say this expression to a friend when they screw up ("Have you considered NOT showing up to work drunk?"), it's also a great way to make an enemy ("Have you considered NOT being a fuckup that nobody likes?"). Same construction. (In fact, this later example is especially sinister, because any answer you give just makes YOU assert what you're being slandered with. "No" - I haven't considered not being a fuckup? / "Yes" - I did in fact consider not being a fuckup? You get the idea. It can be wielded in a very mean way.) And your comment wasn't intended this way at all, and in fact it's obvious when you read it that straight talk and no irony is meant, but it shares enough of the signification with the ironic phrase that it sets off alarm bells. Especially in an online reply to someone, where trolling is sport! The context is working against you here. And the context is the first tool we have when interpreting communication, because it's already available to our cognition before we receive any given message. And brains are energy misers, using heuristics to filter shit out that doesn't look like a good reward for the necessary energy expenditure. So I bet that the downvoters saw 1) a short comment, and short is by the way much more likely to be low effort and unconstructive; and 2) all the native speakers' downvoting brains instantly recognized the same construction as the ironic idiom talked about above, and then they instantly said fuck it, this person is being a dick, without even really reading it. Since irony (unless it’s expected) requires more processing effort (both in detecting WHETHER irony is present AND what the ironic speaker means, just to get to the point where you can start assessing the semantic content), these two things were enough to justify jumping the interpretive gun. I wouldn't be surprised if your question was flagged by MOST native speakers’ brains’ asshole detection systems. And so, the actual substance of your question, which, processed in its entirety, have indicated that you were asking a bona fide, relevant, even microservices-essential question - got booted out. And then the overzealous downvoters succumbed to the urge to smash the downvote button without a second thought and never looked back. I found myself in the same place, actually - thinking "they're being a dick," then moving on. Probably took less than a second. Often when people in online conversations say why the downvotes? they're trolls acting in bad faith, which pisses me off because I always go reread what they wrote. So I did. You weren't calling anyone stupid! Still, I had to stare at it a bit before I realized it was an unintentional collision with the syntax of sarcasm. Maybe like if I wrote "Was hast du denn?" and really truly meant "what do you have?" instead of "what's wrong with you?" If you don't recognize it for what it is, it might look like an honest question which expects an honest answer. Oh. Another thing to consider: after a while spent in any given semiotic context, our cognition adjusts and those cognitive heuristics that reject hard things become even lazier. Our pattern of behavior when reading on screens starts with reading, but then it turns to skimming and scanning if given long enough. I'd bet money that if your brief, but bona fide contribution had been one of the first comments, your downvote percentage would be a LOT lower. Halfway down a long page, the skippers are skipping and the downvoters downvoting. That doesn't mean that they intended to or are even aware of the switch in reading modes. At the top of the page, information is still shiny and new, and worth interpreting, speculating on the speaker and their intentions, etc. After a while, nobody gives any comment a second chance. Anyway, I don't get to revisit all the shit I learned at university enough, so it felt good to write all that. Are you still here? Maybe everyone's already moved the fuck on. Brain got bored? Would serve me right, ha! So, here I go, back to my job, where instead of working in cognitive linguistics, text semiotics, pragmatics, philosophy of language, etc., I'm writing boring-ass JavaScript all day, for an application ... being built on microservices.
abledon 5y ago
One upside in dealing with this mess(and understanding all the intricacies, pains etc..) were in is being able to laugh at the kafkaesque situations people find themselves in now, like your story or other blogposts.
pkulak 5y ago
So... you worked on something terrible that people called a "microservice architecture"? Once a pattern gets popular, people start writing nasty code in the style, and then the pattern takes the reputation hit and people move on to the next thing (or just back to the last thing). Rinse, repeat. My company uses microservices; deploys restart one service and PRs are one repo at a time. There's a shared library, but it's versioned and there's nothing compelling you to keep on the bleeding edge.
- jmchuster 5y ago
  Is this something a lot of people are missing with their microservice implementation? You need to be able to deploy each microservice independently of everyone else. If you have a change that's spread across multiple services, you should be able to just do them one-by-one in order. If you want to rollout a shared library, you should be able to just update your services one-by-one. If a coordinated rollout is required, then doesn't that kind of defeat the whole point of doing microservices?
  - dragonwriter 5y ago
    > If a coordinated rollout is required, then doesn't that kind of defeat the whole point of doing microservices? If a coordinated rollout is requires for anything but a change of service API (and then only the service and it's direct clients should be impacted, and even then a decent deprecation policy should eliminate the need for close coordination), you aren't doing microservices, because loose coupling is part of the definition of the pattern.
  - Gigachad 5y ago
    Its a big anti pattern I have seen that a change requires simultaneous merges on multiple repos and there is basically no way to avoid possible outages/broken systems in the middle period while its deploying.
- dragonwriter 5y ago
  > Once a pattern gets popular, people start writing nasty code in the style, Once it gets popular, people start “implementing the pattern” based on rumor, bad descriptions from unqualified (and sometimes ill-motivated) intermediaries, and some boss and/or tech lead’s fever dreams, and blaming it on the pattern. (If the pattern is useful enough, it will be later invented under a new name to escape from the accumulated cruft, which will briefly succeed only to succumb to the same process until another cycle passes.)
abledon 5y ago
but boy could your pdf generator system scale. Boy could it scale...
- ris 5y ago
  Except in my experience most attempts at microservices scale extremely poorly because they have nowhere near enough precision in their APIs to get just the information you need and so most applications end up fetching way, way more data than they need and whittling it down manually - and often having to stitch together the results from multiple calls. That of course leads you down the path of creating an "oddly specific" API that caters exactly to the needs of the clients, down to the point that small changes in the client's needs require changes in the upstream service(s) - so your "separation of concerns" has become a joke.
  - lIIIllllIIII 5y ago
    Is it a common thing for "proper" microservices to have microservices retrieve data from other services via APIs? I've worked on "microservices" systems before that do this. They were mostly shit. The system I'm working on at the moment has each service subscribe to events and maintain its own database. The only comms is via events. It works pretty well and everything really is nicely separated, but it feels a bit haphazard.
root_axis 5y ago
> I worked on consisted of 7 python repositories that shared one "standard library" repository. Something as simple as adding an additional argument to a function required a PR to 7 repos and sign off on each This is an engineering process failure, not a failure of microservices or shared dependencies. You should be versioning your shared library, that way you only need to make a deployment to the service that requires the update, leaving the others pegged at the previous version until a business or engineering need motivates the upgrade.
- paskozdilar 5y ago
  While we're at the topic of engineering process failures - is there any research/literature about software engineering in general that describes all the hows and whys of large-scale programming like that?
paskozdilar 5y ago
I have had similar experience building microservices that used shared repositories. The PR paperwork was so bad that at one point I've made all my services self-contained, just to avoid having to modify my own code in two different places and synchronize the changes. The whole problem, I think, comes from the "split the code" cargo-cult. We need to think about why we're splitting the code, and use that why to figure out when to split code. IMHO, code separation arises naturally from modular programming - once your code is mature enough, it becomes just a piece of glue around a set of libraries that you can just rip out and put in their own repos, provided that they're useful enough.
AndyPa32 5y ago
So, then you have been working on a distributed monolith and not a proper microservice architecture. Just saying.
willhoyle 5y ago
This is blowing my mind. I felt like you were specifically talking about my place of work. Turns out (based on your github link in your profile), you did work there haha! The 7 repos are gone now and a k8s cluster has replaced the swarm service. Deployments are a bit easier to manage now. It's still such slow development process to add a simple API endpoint, especially if that API has to call other DAOs. It's crazy because I feel like it's completely normalized here that dev work takes 300% longer than it should for simple features.

umpalumpaaa 5y ago

Have you considered not sharing code between services?

glouwbug 5y ago
Yes, I came to learn that microservices would probably greatly excel at sending JSON strings back and forth. Then I wondered why we weren't just all using Erlang
- umpalumpaaa 5y ago
  I don’t understand why I got the downvotes… I was taught that sharing code between services should only be done in circumstances in which there is already a strong connection between them - for example by being owned by the same team. In the organization where I initial learned how to deal with microservices even direct calls via HTTP between services was a no-go and only used in rare circumstances and if then only temporary. I am not sure if this is the way to do it since I have a sample size of one - but we had about 500 services to manage and in the end it worked okay and I never saw something you described. That’s why I wanted to know if you have considered just not sharing code.
  - archduck 5y ago
    Regarding why you got the downvotes, I have a guess: "Have you (maybe) considered not ..." usually gets followed by something that's transparently stupid or incompetent or otherwise undesirable in the given context. For example: "Have you considered NOT publicly yelling at your coworkers?" Obviously that's not what you were saying, but it may have read like it. It's a fantastic INDIRECT way of telling someone they did something stupid. And because stupid actions map so well onto their doers, it's also a fantastic way to insult people. As an aside: this sort of indirection - asking a question to assert something - it's not peculiar to English, but it's suuper common in English-language expressions (which is, in fact, due to the habitual indirection of the English - at least according to a linguistics professor I once had.) The kicker is: because it's an ironic construction, it’s likely going to require more effort to process: the presence of irony means that what is actually meant by the speaker is not encoded literally, but rather must be interpreted - derived from what WAS said - and irony (and other ways of flouting literal language) yields a disjunction (x || y || z). It puts the listener into a role where their recognition of intent and ability to joke around are tested - and the correct interpretation is up in the air. So while it's possible to say this expression to a friend when they screw up ("Have you considered NOT showing up to work drunk?"), it's also a great way to make an enemy ("Have you considered NOT being a fuckup that nobody likes?"). Same construction. (In fact, this later example is especially sinister, because any answer you give just makes YOU assert what you're being slandered with. "No" - I haven't considered not being a fuckup? / "Yes" - I did in fact consider not being a fuckup? You get the idea. It can be wielded in a very mean way.) And your comment wasn't intended this way at all, and in fact it's obvious when you read it that straight talk and no irony is meant, but it shares enough of the signification with the ironic phrase that it sets off alarm bells. Especially in an online reply to someone, where trolling is sport! The context is working against you here. And the context is the first tool we have when interpreting communication, because it's already available to our cognition before we receive any given message. And brains are energy misers, using heuristics to filter shit out that doesn't look like a good reward for the necessary energy expenditure. So I bet that the downvoters saw 1) a short comment, and short is by the way much more likely to be low effort and unconstructive; and 2) all the native speakers' downvoting brains instantly recognized the same construction as the ironic idiom talked about above, and then they instantly said fuck it, this person is being a dick, without even really reading it. Since irony (unless it’s expected) requires more processing effort (both in detecting WHETHER irony is present AND what the ironic speaker means, just to get to the point where you can start assessing the semantic content), these two things were enough to justify jumping the interpretive gun. I wouldn't be surprised if your question was flagged by MOST native speakers’ brains’ asshole detection systems. And so, the actual substance of your question, which, processed in its entirety, have indicated that you were asking a bona fide, relevant, even microservices-essential question - got booted out. And then the overzealous downvoters succumbed to the urge to smash the downvote button without a second thought and never looked back. I found myself in the same place, actually - thinking "they're being a dick," then moving on. Probably took less than a second. Often when people in online conversations say why the downvotes? they're trolls acting in bad faith, which pisses me off because I always go reread what they wrote. So I did. You weren't calling anyone stupid! Still, I had to stare at it a bit before I realized it was an unintentional collision with the syntax of sarcasm. Maybe like if I wrote "Was hast du denn?" and really truly meant "what do you have?" instead of "what's wrong with you?" If you don't recognize it for what it is, it might look like an honest question which expects an honest answer. Oh. Another thing to consider: after a while spent in any given semiotic context, our cognition adjusts and those cognitive heuristics that reject hard things become even lazier. Our pattern of behavior when reading on screens starts with reading, but then it turns to skimming and scanning if given long enough. I'd bet money that if your brief, but bona fide contribution had been one of the first comments, your downvote percentage would be a LOT lower. Halfway down a long page, the skippers are skipping and the downvoters downvoting. That doesn't mean that they intended to or are even aware of the switch in reading modes. At the top of the page, information is still shiny and new, and worth interpreting, speculating on the speaker and their intentions, etc. After a while, nobody gives any comment a second chance. Anyway, I don't get to revisit all the shit I learned at university enough, so it felt good to write all that. Are you still here? Maybe everyone's already moved the fuck on. Brain got bored? Would serve me right, ha! So, here I go, back to my job, where instead of working in cognitive linguistics, text semiotics, pragmatics, philosophy of language, etc., I'm writing boring-ass JavaScript all day, for an application ... being built on microservices.

abledon 5y ago

One upside in dealing with this mess(and understanding all the intricacies, pains etc..) were in is being able to laugh at the kafkaesque situations people find themselves in now, like your story or other blogposts.

pkulak 5y ago

So... you worked on something terrible that people called a "microservice architecture"? Once a pattern gets popular, people start writing nasty code in the style, and then the pattern takes the reputation hit and people move on to the next thing (or just back to the last thing). Rinse, repeat. My company uses microservices; deploys restart one service and PRs are one repo at a time. There's a shared library, but it's versioned and there's nothing compelling you to keep on the bleeding edge.

jmchuster 5y ago
Is this something a lot of people are missing with their microservice implementation? You need to be able to deploy each microservice independently of everyone else. If you have a change that's spread across multiple services, you should be able to just do them one-by-one in order. If you want to rollout a shared library, you should be able to just update your services one-by-one. If a coordinated rollout is required, then doesn't that kind of defeat the whole point of doing microservices?
- dragonwriter 5y ago
  > If a coordinated rollout is required, then doesn't that kind of defeat the whole point of doing microservices? If a coordinated rollout is requires for anything but a change of service API (and then only the service and it's direct clients should be impacted, and even then a decent deprecation policy should eliminate the need for close coordination), you aren't doing microservices, because loose coupling is part of the definition of the pattern.
- Gigachad 5y ago
  Its a big anti pattern I have seen that a change requires simultaneous merges on multiple repos and there is basically no way to avoid possible outages/broken systems in the middle period while its deploying.
dragonwriter 5y ago
> Once a pattern gets popular, people start writing nasty code in the style, Once it gets popular, people start “implementing the pattern” based on rumor, bad descriptions from unqualified (and sometimes ill-motivated) intermediaries, and some boss and/or tech lead’s fever dreams, and blaming it on the pattern. (If the pattern is useful enough, it will be later invented under a new name to escape from the accumulated cruft, which will briefly succeed only to succumb to the same process until another cycle passes.)

abledon 5y ago

but boy could your pdf generator system scale. Boy could it scale...

ris 5y ago
Except in my experience most attempts at microservices scale extremely poorly because they have nowhere near enough precision in their APIs to get just the information you need and so most applications end up fetching way, way more data than they need and whittling it down manually - and often having to stitch together the results from multiple calls. That of course leads you down the path of creating an "oddly specific" API that caters exactly to the needs of the clients, down to the point that small changes in the client's needs require changes in the upstream service(s) - so your "separation of concerns" has become a joke.
- lIIIllllIIII 5y ago
  Is it a common thing for "proper" microservices to have microservices retrieve data from other services via APIs? I've worked on "microservices" systems before that do this. They were mostly shit. The system I'm working on at the moment has each service subscribe to events and maintain its own database. The only comms is via events. It works pretty well and everything really is nicely separated, but it feels a bit haphazard.

root_axis 5y ago

> I worked on consisted of 7 python repositories that shared one "standard library" repository. Something as simple as adding an additional argument to a function required a PR to 7 repos and sign off on each This is an engineering process failure, not a failure of microservices or shared dependencies. You should be versioning your shared library, that way you only need to make a deployment to the service that requires the update, leaving the others pegged at the previous version until a business or engineering need motivates the upgrade.

paskozdilar 5y ago
While we're at the topic of engineering process failures - is there any research/literature about software engineering in general that describes all the hows and whys of large-scale programming like that?

paskozdilar 5y ago

I have had similar experience building microservices that used shared repositories. The PR paperwork was so bad that at one point I've made all my services self-contained, just to avoid having to modify my own code in two different places and synchronize the changes. The whole problem, I think, comes from the "split the code" cargo-cult. We need to think about why we're splitting the code, and use that why to figure out when to split code. IMHO, code separation arises naturally from modular programming - once your code is mature enough, it becomes just a piece of glue around a set of libraries that you can just rip out and put in their own repos, provided that they're useful enough.

AndyPa32 5y ago

So, then you have been working on a distributed monolith and not a proper microservice architecture. Just saying.

willhoyle 5y ago

This is blowing my mind. I felt like you were specifically talking about my place of work. Turns out (based on your github link in your profile), you did work there haha! The 7 repos are gone now and a k8s cluster has replaced the swarm service. Deployments are a bit easier to manage now. It's still such slow development process to add a simple API endpoint, especially if that API has to call other DAOs. It's crazy because I feel like it's completely normalized here that dev work takes 300% longer than it should for simple features.