6 ms·

Yup I've seen it at just about every company I've worked for. A cluster of microservices that is really just a distributed monolith. Is it really a microservice

Yup I've seen it at just about every company I've worked for. A cluster of microservices that is really just a distributed monolith. Is it really a microservice if changing one part of it requires making echoing changes to every service downstream? Is it really a microservice if one part breaking causes the entire system to fail spectacularly? I feel like the pendulum is starting to swing the other way as companies learn that microservices aren't the end-all solution to their process problems, and that prioritizing new features every sprint and constantly pushing housekeeping, bugsquashing, and system improvements will result in a broken collection of microservices just as quickly as it resulted in a broken monolith.

411111111111111 4y ago
> Is it really a microservice if one part breaking causes the entire system to fail spectacularly? No, that's a distributed monolith not a microservice
cogman10 4y ago
> prioritizing new features every sprint and constantly pushing housekeeping, bugsquashing, and system improvements will result in a broken collection of microservices just as quickly as it resulted in a broken monolith. Now, HTF do you convince a C-Level of this truth? Seems like there is never any time for any sort of housekeeping until the house is literally on fire and the whole world is looking at our smoke pillar. Even then, they'll usually be like "Ok, take 2 weeks to fix all our architecture problems and then get back to work on new features!". As SOON as the fire is doused, they want to move on rather than cleaning and removing the flammable material and the flame that keep starting the fire.
- mixedCase 4y ago
  You don't, it's not his job to care, it's yours. And just like you don't ask permission for writing down a line of feature code, you don't ask permission for maintenance work. You just do it and schedule it accordingly to business needs to the best of your ability. And make sure not to apologize for doing your work or treat maintenance work as "unimportant" because that is how it will be perceived. It does get tricky when you have a manager defining tasks and you have yet to determine whether their job is to be a technical manager (and is adequately competent) or a nerd babysitter/translator. In the former case it's good to run maintenance projects by them if they need active, dedicated time as opposed to something you can mostly do during downtime. In case it's the other kind of "manager", and you've already tried and failed to convince them to prioritize basic stuff, you have to just put it inside other estimates and rope in the rest of your team to do the same and/or polish your résumé.
  - pojzon 4y ago
    > You don't, it's not his job to care, it's yours. Last time I checked it was the job of Project Manager to decide which tasks land on the sprint backlog as he decides the priorities of what has to be delivered. Literally written in the job description. As a developer you can create a plan/tasks about decreasing technical dept, but its never your work to prioritize that, coz you dont know whats the priority of upper management..
    - polotics 4y ago
      dot dot dot Hello! you do know the priority of upper management: they prioritize their own personal best interest, and as much as the board and CEO manage the incentive structure, this will be aligned with their fat bonuses. so, dot dot dot. do the same: if something creates pain for you, dispose of it.
      - cogman10 4y ago
        I've got two choices: - I can work with the current architecture and ever inflating accruing debt (yet slowly inflating) feature times which ultimately is acceptable by PM. This will get me pay raises, bonuses, and praise. - I can work on the annoying stuff which product does NOT want me working on, does decrease feature times, and does decrease debt. This will get me chastised, if I get any praise from my local team members, I don't get it from the people paying me. Working around the system is punished. Working through the system is rewarded. I get my fat bonuses by making my PO/PM happy. I don't get fat bonuses making my life easier.
    - cogman10 4y ago
      Bingo. My quarterly deliverables are defined by the product guy above me. His priorities are defined by the product org. Ultimately, the CTO/CEO are the ones that set and tell them to allocate stuff. I can RAISE issues that need to be addressed to product, but generally speaking, dev doesn't have a seat at the table when it comes to prioritizing stuff. Instead, that's all driven by sales. Sure the company SAYS they do, but the practice isn't there. Dev requests are routinely ignored to the point that dev stops making them. Perhaps this is just my org that's dysfunctional, doesn't feel that way though.
- jasonwatkinspdx 4y ago
  > Now, HTF do you convince a C-Level of this truth? Seems like there is never any time for any sort of housekeeping until the house is literally on fire and the whole world is looking at our smoke pillar. I've had minor successes with advocating for a tik / tok approach, ie every couple iterations you have one iteration where new feature work is banned vs cleanup, bugs long ignored in the backlog, etc. I haven't found a simple way to explain this, but product centric executives do seem to see "ok, they get one iteration of what they want in trade for me getting what I want in the others" as a reasonable trade to keep the dev team happy.
  - native_samples 4y ago
    It's probably the predictability and bounded costs that help. I've been on both sides of this. I've been on projects with tech debt, but I've also had lots of experiences where new developers join a project and immediately, before even seeing the code at all, announce that they enjoy "making code clean" and "cleaning up tech debt" and where is the tech debt they can refactor away? Or they'll join, spend literally 20 minutes reading the first file they come across and then start pronouncing the decisions made by their new collegues 18 months ago as obviously "legacy" or "hacky". That's usually a sign of trouble. The whole concept of tech debt or "cleanness" is very under-defined. One man's tech debt is another man's pragmatic solution, or even clean design. The last company I worked at is basically being killed by this problem. The technical leadership is weak and agrees to what the engineers say too easily. Theoretically there's a product team but they aren't technical enough to argue with the devs. The moment the company started getting product/market fit and making good sales the devs announced the product - all of three years old - was riven with tech debt and would need a massive rewrite (it didn't, I worked on it and it was fine). Three years later their grand rewrite still didn't launch. Utterly pathetic. The correct solution would have been to crack down on that sort of thing and tell devs who wanted to rewrite the product that they were welcome to do that, somewhere else. Or, they could get back to adding features that would actually make users happier. I think a lot of companies have had that experience - the sort of devs who are obsessed with "clean code" and "tech debt" often exhibit extremely poor judgement and can end up making things worse. Especially if the product has a naturally limited lifespan anyway due to e.g. pace of change in the industry, it can be fatal to spend too much time on meta-coding.
- mperham 4y ago
  Farmers have to let fields lie fallow every N seasons, gardens must be weeded. You could easily argue that if a CTO or VP Eng doesn't already understand and agree with this, they aren't qualified for the job.
- fleddr 4y ago
  You need psychological engineering... My method is radical transparency. I simply keep refactoring notes which I turn into user stories on the backlog. I don't take the PM seat as to when to do these user stories, yet have built up the expectation that it's an ongoing process and recurring thing. The transparency creates a rational set. Clarity about what is currently wrong and what the benefit of the improvement is. It's kind of hard to argue against rationalism. When you do this calmly and with precision, you instill a perpetual feeling of guilt every time the PM tries to ignore it. Seed planted. 1-0 for you. It's all in the open, the entire team sees it. That's different from a forgettable backroom discussion. The PM looks somewhat neglectful to everybody. That's 2-0. You need to have/build a reputation of being constructive. Thinking along with the PM instead of being stubborn. So you let go of refactoring for two sprints because there really is an important business deadline, and then you bring refactoring back to the agenda. This demonstrates you're reasonable and flexible, whilst if the PM is to continue to ignore refactoring, they will stand out as the unreasonable. 3-0. Continued ignoring of technical debt will inevitably lead to disaster, affecting real world results. Who will get the blame? Not you, you have your bases covered and it's on record whom ignored all the warnings. 4-0. These are all mind tricks to raise the cost for the PM of ignoring technical debt. Ideally it's not needed and you just agree to reserve 20% of your capacity for it, to end the discussion once and for all. Of course, one may also encounter a job hopping sociopath PM that doesn't care about any of the above. In that case, just inflate your user stories and do what you can. To conclude, in a truly healthy organization one would flip the question. PMs have a tendency to be feature fetishists. It's not even their fault as internal organizations richly reward whoever has the most ambitious feature agenda, whilst everybody thinks that the PM that "fixes the basics" is a slacker. The user of the product, in the meanwhile, doesn't seem to be a topic of concern. Rather than requiring extreme evidence for a software engineer to do the basics of their job, ask a PM per feature what its justification is. Show me the business case. They can't. I've been in this industry for 2 decades and the consistent experience is that every long term software project ultimately ends up in 80-90% of non-value or negative value, also known as "unrewarded complexity". If I were PM, I'd spent 50% of each sprint on refactoring, simplification, and improving the core of the product that delivers business value, rather than building unrewarded complexity. For the other 50%, I'd take the team to the bar, which I consider a good use of time. You'd find me a ridiculous PM, but I'm deadly serious that the product outcome would be better. You can't have technical debt when you don't do loans. Plus it's more fun.
- urthor 4y ago
  C suite is fully aware. C suite has, for many a year, launched 5 underfunded software projects and canned the 4 that don't gain traction. The goal is to build tech debt until the house is on fire, because they don't expect the majority of projects to survive. Any truly important is done via external vendors & specialised product companies. Internal corporate applications simply don't matter much.
pojzon 4y ago
No architectural design is an „End of all best solution”. Best solution is to get very smart ppl and listen to them. Thats the part most companies fail at. Until Software Engineering becomes a real ENGINEERING, companies will do half-ass job just to push something out and make quick bucks. Rarely you gonna see nowadays a craftsmanship brilliance.
- ryanmcbride 4y ago
  Exactly

411111111111111 4y ago

> Is it really a microservice if one part breaking causes the entire system to fail spectacularly? No, that's a distributed monolith not a microservice

cogman10 4y ago

> prioritizing new features every sprint and constantly pushing housekeeping, bugsquashing, and system improvements will result in a broken collection of microservices just as quickly as it resulted in a broken monolith. Now, HTF do you convince a C-Level of this truth? Seems like there is never any time for any sort of housekeeping until the house is literally on fire and the whole world is looking at our smoke pillar. Even then, they'll usually be like "Ok, take 2 weeks to fix all our architecture problems and then get back to work on new features!". As SOON as the fire is doused, they want to move on rather than cleaning and removing the flammable material and the flame that keep starting the fire.

mixedCase 4y ago
You don't, it's not his job to care, it's yours. And just like you don't ask permission for writing down a line of feature code, you don't ask permission for maintenance work. You just do it and schedule it accordingly to business needs to the best of your ability. And make sure not to apologize for doing your work or treat maintenance work as "unimportant" because that is how it will be perceived. It does get tricky when you have a manager defining tasks and you have yet to determine whether their job is to be a technical manager (and is adequately competent) or a nerd babysitter/translator. In the former case it's good to run maintenance projects by them if they need active, dedicated time as opposed to something you can mostly do during downtime. In case it's the other kind of "manager", and you've already tried and failed to convince them to prioritize basic stuff, you have to just put it inside other estimates and rope in the rest of your team to do the same and/or polish your résumé.
- pojzon 4y ago
  > You don't, it's not his job to care, it's yours. Last time I checked it was the job of Project Manager to decide which tasks land on the sprint backlog as he decides the priorities of what has to be delivered. Literally written in the job description. As a developer you can create a plan/tasks about decreasing technical dept, but its never your work to prioritize that, coz you dont know whats the priority of upper management..
  - polotics 4y ago
    dot dot dot Hello! you do know the priority of upper management: they prioritize their own personal best interest, and as much as the board and CEO manage the incentive structure, this will be aligned with their fat bonuses. so, dot dot dot. do the same: if something creates pain for you, dispose of it.
    - cogman10 4y ago
      I've got two choices: - I can work with the current architecture and ever inflating accruing debt (yet slowly inflating) feature times which ultimately is acceptable by PM. This will get me pay raises, bonuses, and praise. - I can work on the annoying stuff which product does NOT want me working on, does decrease feature times, and does decrease debt. This will get me chastised, if I get any praise from my local team members, I don't get it from the people paying me. Working around the system is punished. Working through the system is rewarded. I get my fat bonuses by making my PO/PM happy. I don't get fat bonuses making my life easier.
  - cogman10 4y ago
    Bingo. My quarterly deliverables are defined by the product guy above me. His priorities are defined by the product org. Ultimately, the CTO/CEO are the ones that set and tell them to allocate stuff. I can RAISE issues that need to be addressed to product, but generally speaking, dev doesn't have a seat at the table when it comes to prioritizing stuff. Instead, that's all driven by sales. Sure the company SAYS they do, but the practice isn't there. Dev requests are routinely ignored to the point that dev stops making them. Perhaps this is just my org that's dysfunctional, doesn't feel that way though.
jasonwatkinspdx 4y ago
> Now, HTF do you convince a C-Level of this truth? Seems like there is never any time for any sort of housekeeping until the house is literally on fire and the whole world is looking at our smoke pillar. I've had minor successes with advocating for a tik / tok approach, ie every couple iterations you have one iteration where new feature work is banned vs cleanup, bugs long ignored in the backlog, etc. I haven't found a simple way to explain this, but product centric executives do seem to see "ok, they get one iteration of what they want in trade for me getting what I want in the others" as a reasonable trade to keep the dev team happy.
- native_samples 4y ago
  It's probably the predictability and bounded costs that help. I've been on both sides of this. I've been on projects with tech debt, but I've also had lots of experiences where new developers join a project and immediately, before even seeing the code at all, announce that they enjoy "making code clean" and "cleaning up tech debt" and where is the tech debt they can refactor away? Or they'll join, spend literally 20 minutes reading the first file they come across and then start pronouncing the decisions made by their new collegues 18 months ago as obviously "legacy" or "hacky". That's usually a sign of trouble. The whole concept of tech debt or "cleanness" is very under-defined. One man's tech debt is another man's pragmatic solution, or even clean design. The last company I worked at is basically being killed by this problem. The technical leadership is weak and agrees to what the engineers say too easily. Theoretically there's a product team but they aren't technical enough to argue with the devs. The moment the company started getting product/market fit and making good sales the devs announced the product - all of three years old - was riven with tech debt and would need a massive rewrite (it didn't, I worked on it and it was fine). Three years later their grand rewrite still didn't launch. Utterly pathetic. The correct solution would have been to crack down on that sort of thing and tell devs who wanted to rewrite the product that they were welcome to do that, somewhere else. Or, they could get back to adding features that would actually make users happier. I think a lot of companies have had that experience - the sort of devs who are obsessed with "clean code" and "tech debt" often exhibit extremely poor judgement and can end up making things worse. Especially if the product has a naturally limited lifespan anyway due to e.g. pace of change in the industry, it can be fatal to spend too much time on meta-coding.
mperham 4y ago
Farmers have to let fields lie fallow every N seasons, gardens must be weeded. You could easily argue that if a CTO or VP Eng doesn't already understand and agree with this, they aren't qualified for the job.
fleddr 4y ago
You need psychological engineering... My method is radical transparency. I simply keep refactoring notes which I turn into user stories on the backlog. I don't take the PM seat as to when to do these user stories, yet have built up the expectation that it's an ongoing process and recurring thing. The transparency creates a rational set. Clarity about what is currently wrong and what the benefit of the improvement is. It's kind of hard to argue against rationalism. When you do this calmly and with precision, you instill a perpetual feeling of guilt every time the PM tries to ignore it. Seed planted. 1-0 for you. It's all in the open, the entire team sees it. That's different from a forgettable backroom discussion. The PM looks somewhat neglectful to everybody. That's 2-0. You need to have/build a reputation of being constructive. Thinking along with the PM instead of being stubborn. So you let go of refactoring for two sprints because there really is an important business deadline, and then you bring refactoring back to the agenda. This demonstrates you're reasonable and flexible, whilst if the PM is to continue to ignore refactoring, they will stand out as the unreasonable. 3-0. Continued ignoring of technical debt will inevitably lead to disaster, affecting real world results. Who will get the blame? Not you, you have your bases covered and it's on record whom ignored all the warnings. 4-0. These are all mind tricks to raise the cost for the PM of ignoring technical debt. Ideally it's not needed and you just agree to reserve 20% of your capacity for it, to end the discussion once and for all. Of course, one may also encounter a job hopping sociopath PM that doesn't care about any of the above. In that case, just inflate your user stories and do what you can. To conclude, in a truly healthy organization one would flip the question. PMs have a tendency to be feature fetishists. It's not even their fault as internal organizations richly reward whoever has the most ambitious feature agenda, whilst everybody thinks that the PM that "fixes the basics" is a slacker. The user of the product, in the meanwhile, doesn't seem to be a topic of concern. Rather than requiring extreme evidence for a software engineer to do the basics of their job, ask a PM per feature what its justification is. Show me the business case. They can't. I've been in this industry for 2 decades and the consistent experience is that every long term software project ultimately ends up in 80-90% of non-value or negative value, also known as "unrewarded complexity". If I were PM, I'd spent 50% of each sprint on refactoring, simplification, and improving the core of the product that delivers business value, rather than building unrewarded complexity. For the other 50%, I'd take the team to the bar, which I consider a good use of time. You'd find me a ridiculous PM, but I'm deadly serious that the product outcome would be better. You can't have technical debt when you don't do loans. Plus it's more fun.
urthor 4y ago
C suite is fully aware. C suite has, for many a year, launched 5 underfunded software projects and canned the 4 that don't gain traction. The goal is to build tech debt until the house is on fire, because they don't expect the majority of projects to survive. Any truly important is done via external vendors & specialised product companies. Internal corporate applications simply don't matter much.

pojzon 4y ago

No architectural design is an „End of all best solution”. Best solution is to get very smart ppl and listen to them. Thats the part most companies fail at. Until Software Engineering becomes a real ENGINEERING, companies will do half-ass job just to push something out and make quick bucks. Rarely you gonna see nowadays a craftsmanship brilliance.

ryanmcbride 4y ago
Exactly