9 ms·

Here's a question: if you have a team and an internal culture that managed to create a legacy code mess in as little as six months in their existing app, how is

Here's a question: if you have a team and an internal culture that managed to create a legacy code mess in as little as six months in their existing app, how is giving them a microservices architecture and Docker to play with going to help? If you give that team that structure, it seems to me you're highly unlikely to end up with beautifully bounded context and highly likely to end up with even more of a mess than you started with.

lostInTheWoods3 10y ago
Agreed. Cleaning and correcting a code base is only part of the solution. Putting in place the policies and culture to make sure the code stays clean is the other part.
biocomputation 10y ago
Depending on what you're doing, the right decisions might only be visible in hindsight. To this point, you might scaffold out a system to get something running and then incrementally / organically improve the system. I regard this as 'getting the system to the point where it can be dogfooded and/or criticized in a helpful way'. At this point, despite what you might have running, it may be exceedingly clear that you should have used a different strategy. Maybe you like the overall results, but you need things to be implemented in a different way. For example, I'm a C++ programmer. It's not always obvious at first that you need a specific set of virtual functions to make something work really nicely. It's impossible to always know these sorts of things in advance.
- ben_jones 10y ago
  What if microservices are poorly architected such that they are actually coupled? It's worth thinking about such an outcome especially if microservices are being seen as a way of making dysfunctional teams functional.
  - biocomputation 10y ago
    You're right, I'm just saying that it's not always possible to see these problems in advance, and that it can be very difficult to try and get something done if you apply too many rules to a nascent system ( too many rules == design by committee ).
- Ericson2314 10y ago
  I assume you are saying this is why refactoring is so important? If so I absolutely agree. Designing good design/abstractions like anything else takes practice. Armchair critiques won't cut it. If one's unwilling to fix bad design once it's identified, they're not practicing good design, they're practicing cobbling together something on a rickety foundation. How's that going to help next time one rights something from scratch?
  - biocomputation 10y ago
    Yes, to me, 'software development' is 'refactoring', just like some authors say that 'writing' is actually 'rewriting'.
jdavis703 10y ago
It's easier to just start over again with one small part. I've worked at two places with massive monolithic tech debt ridden codebases. It was nearly impossible to rewrite anything because everything touched everything. Now the place I currently work has micro services. Some are legacy tech debt that no one wants to touch. But if needed they could probably be rewritten in a couple of months. It's the difference between selling AAA bonds and junk bonds. You're probably far more likely to pay off those low interest AAA ones than the junk bonds.
- hinkley 10y ago
  The tradeoff though is that you can't get a handle on how expensive a particular call is because the domino effect of knock-on calls is very difficult to see. You can get a system that 'works' on test data but doesn't scale to production (from a user's perspective, this code doesn't work, even though the dev team will insist that it does)
  - computerex 10y ago
    Engineering is about trade offs. Microservices are a fairly recent innovation, and there are many kinks that need to be worked out. Another issue is the overhead cost of maintaining a large number of decoupled, independent services. It's plausible to have hundreds or thousands of microservices, and the tools to visualize/manage these systems haven't yet been created.
    - erichocean 10y ago
      > Microservices are a fairly recent innovation Yeah, no. Micro-services are neither recent, nor innovative. Fun fact: a friend of my consults for large pensions funds in The Netherlands. One of the projects he was brought in on earlier this year to get an outside perspective on was this gigantic monument to micro-services that had been built up over a two year period by ~200 developers, at a cost of around 50 million euros. After two months, every developer was fired, the project cancelled, and the entire 50 million euro cost written off. But it had great micro-services, hundreds and hundreds of them!
hinkley 10y ago
That's a very excellent question, to which I have most of an answer, if not 'the' answer. Daylight is the best antiseptic. It's not giving them a microservices architecture and Docker that solves problems. It's expecting them to use it that causes problems to be solved, or at least formally identified. I do a lot of root cause analysis whether my boss asks for it or not. Human factors are almost always in play. A significant source of bugs that make it into late QA or even production are caused by wishful thinking about the scope and impact of code changes. Things that should have been red flags are brushed off because you can't prove that their code caused the problem, and as long as there is reasonable doubt about the source, some people won't look at code they already declared done. When the code is developed and tested in an isolated system then the only source of state changes on that system were caused by the new code. It takes a real set of brass ones or an extremely dense skull to deflect concerns about problems seen on a system that only contains your code changes. People either shape up or get labeled as untrustworthy. The former result is preferable, but at least with the latter you get predictability out of the system.
- 0xfaded 10y ago
  I would argue it would be simpler to introduce test coverage tools that automatically call out bad test coverage during code review. Microservices remind me of a particular C++ protocol implementation I wrote as a novice programmer. Since the protocol was structured, I had high level classes broken down into simpler classes. e.g. virtual class Reader { void read(Buffer *buf); }; class TCharacter: public Reader { TString *name; TList<TStat> *stats; ... }; class TStat: public Reader { TWord32 *remaining; TWord8 *percent; }; class TWord32: public Reader { uint32_t v; }; class TWord8: public Reader { uint8_t v; }; And on, and on, and on, and on ... I thought it was cute that down to the simplest PODT, every type satisfied a well defined `Reader` interface. I hand wrote many constructors, deconstructors, read and several other interface definitions for each type. Elegant, arguably. Inefficient, yes. Overly complex, undoubtedly. Every argument for microservices has in some way reminded me of this code, with it's well defined interfaces and overly verbose implementation. My current strategy is to utilise the crap out of language features to safely achieve what I want in the minimal amount of code possible. In C++ this would be done by initialising the protocol structure classes straight out of the memory buffer, using #pragma pack as required. No error prone constructors / deconstructors / read() required. Haskell and Scala are two languages I use a lot, and each have powerful type systems which I can use to prevent myself from doing something stupid. Programmable macros are great for removing boilerplate. Want less technical debt? Write less code. In my own controversial opinion, if somebody can't work out how to modify my code because they can't work out how to correctly update the type definitions, that person has no business modifying my code. Problem solved! Interfaces are well defined and modifiable only by people who truly understand the code.
maxxxxx 10y ago
Yeah, that's the 'shiny new toy' fallacy. In every project there are points when it's clear that something wasn't designed right. Some people decide to get deeper into their mess and some people decide to refactor. It's a judgement call and not an easy one. I don't think I'd want to work with microservices and Docker containers designed by the same people who messed up another codebase.
- sbov 10y ago
  Considering management has a very large role in most messed up codebases I've seen, even if they fire 100% of the engineering team who coded it you probably are working with (for) the same people who messed up a codebase.
  - maxxxxx 10y ago
    Management is a part of the team that messed up the codebase, not only the engineers. That's why I am always wary of managers bringing in new methods (microservices, scrum, whatever). They had a big part in previous failures and if they don't admit their own failures nothing will change.
    - collyw 10y ago
      The majority of managers don't give a shit about code quality. A fairly significant percentage of developers don't give a shit either.
  - mseebach 10y ago
    That's a pretty dangerous mindset, and one that seems to be pretty popular around here, that all failure is automatically the fault of management. Yes, management plays a role, that's almost tautological, but it takes two to tango, and at the end of the day, the dev team wrote the messed up code. Grandly messing up in just six months, even if encouraged by management, takes willing participation from the developers. Also, I've seen plenty of developers who were perfectly capable of screwing up codebases without being induced to do so by management (I've also seen lots of inept management, don't get me wrong. But I wrote the bad code, they didn't.)
beat 10y ago
Doesn't need to be an "and". Just getting the development/ops environments into a virtualized architecture like Docker can make a tremendous difference in a monolithic app. Quite often, the technical debt is due to configuration differences in manually managed environments - development, QA, and production can differ in small or large ways. I don't know how many times I've seen things like developing on Windows and deploying on Unix, or testing on a single server but deploying into a cluster. Docker (or Vagrant, or others) can fully automate provisioning and configuration, simplifying and enforcing consistency between environments. Likewise, microservices from monoliths can be done without containers. Just pick a chunk of the monolith that can be pried loose, and start with that. This can solve all sorts of problems.
gvb 10y ago
"Corgibytes often introduces new clients to Docker, making it much easier and faster to set up new developer environments." (emphasis mine) She isn't advocating using Docker for the target environment, she is advocating using Docker to set up a standard development environment that allows the developers to be productive[1] in 10 minutes rather than 10 hours. I've been in the situation where the development environment is a poorly documented, massively multi-step steaming pile. The result is I spent a lot of time chasing down those responsible for the steaming pile (the "rockstar" in the department) asking them how to get the development environment working. Typically they pull the keyboard away from me, type a bunch, and then say "there, it works." Which it does. For a while. Rinse and repeat for every developer. Regularly. [1] Well, at least building the system rather than staring at mysterious very broken error messages.
- marcosdumay 10y ago
  Hum... You are trading an opaque irreproducible environment for another opaque environment that you reproduce by copying verbatim. Technically, it's progress. But I can't imagine how people get satisfied by this.
  - Arcsech 10y ago
    I'm not too experienced with Docker, but aren't Docker images built using Dockerfiles, which are essentially glorified shell scripts? In that case, wouldn't the Dockerfile be a relatively transparent way to see what's needed to reproduce an environment?
    - lallysingh 10y ago
      Yes, yes they are. And they're written down where you can read them yourself. And maybe even version controlled!
      - GrinningFool 10y ago
        On the other hand, tracing back a dockerfile that pulls in an obscure one-off base image, then repeating that all the way up the chain, you can find yourself left with a dozen Dockerfiles that you need to read in the right order to know what you have. That's if you can even find the original Dockerfiles - I recently ran into a situation where a container image was in the registry, but the original Dockerfile was completely lost. Docker/containers in general move the complexity to a different plane, but they only move it - they don't eliminate it. Care needs to be taken to avoid situations just like the one above - or any of a number of other pitfalls - or you end up with an environment that's far more complex to understand than a VM housing a monolithic dev environment.
benologist 10y ago
It's not even "technical debt" until someone thinks "technically ... we could have done this more easily in X". Now it's a question of developer comfort and huge investment will be made to make sure everyone building the product are having the best experience doing so. All the old code and knowledge is now debt because it can't come with us on this wonderful journey.
- codingmyway 10y ago
  It's only debt if you are paying for it going forward. The thought should be "things would be easier going forward if we'd done X". If it does the job and doesn't create issues then you just purchased it without taking on debt. This is easier if it is isolated/loosely coupled. There's no point going back to improve it unless it saves you time/money in the future. If you make a better one to get the 'best experience' you are throwing away time and money for a slightly better mousetrap.
  - benologist 10y ago
    It's ironic because we're trying to build software others will depend on, meanwhile the software that we depend on tries to avoid the kind of disruptive change that accompanies rewritten-with-X... that's why we can depend on it.
mscottford 10y ago
You're right to say that a microservices architecture and Docker won't solve the problem. That's not an accurate view of what Corgibytes advocates for. Microservices and Docker are tactical tools that can help many teams, but they don't work everywhere. What does work is assisting that team to slowly clean up the mess that they've built[1] by focusing on incremental improvements. [1]: And we find that almost all messes were created for very understandable reasons.
matwood 10y ago
One of the main causes of technical debt and bad code overall is breaking out of the conventions of the code base. Good code reviews can help this, and solid leads of course, but things can still slip through. Like or hate microservices, one thing they do is maintain a clear, hard boundary on the interfaces. There is no let me just reach into this class here, or call that there. It does work to keep people honest when the pressure is on. Will it turn bad programmers into good ones? No, but I think the base assumption is that you're not necessarily dealing with bad programmers.
- collyw 10y ago
  Personally I would say that crappy abstractions (or no abstractions) are what makes code shitty.

lostInTheWoods3 10y ago

Agreed. Cleaning and correcting a code base is only part of the solution. Putting in place the policies and culture to make sure the code stays clean is the other part.

biocomputation 10y ago

Depending on what you're doing, the right decisions might only be visible in hindsight. To this point, you might scaffold out a system to get something running and then incrementally / organically improve the system. I regard this as 'getting the system to the point where it can be dogfooded and/or criticized in a helpful way'. At this point, despite what you might have running, it may be exceedingly clear that you should have used a different strategy. Maybe you like the overall results, but you need things to be implemented in a different way. For example, I'm a C++ programmer. It's not always obvious at first that you need a specific set of virtual functions to make something work really nicely. It's impossible to always know these sorts of things in advance.

ben_jones 10y ago
What if microservices are poorly architected such that they are actually coupled? It's worth thinking about such an outcome especially if microservices are being seen as a way of making dysfunctional teams functional.
- biocomputation 10y ago
  You're right, I'm just saying that it's not always possible to see these problems in advance, and that it can be very difficult to try and get something done if you apply too many rules to a nascent system ( too many rules == design by committee ).
Ericson2314 10y ago
I assume you are saying this is why refactoring is so important? If so I absolutely agree. Designing good design/abstractions like anything else takes practice. Armchair critiques won't cut it. If one's unwilling to fix bad design once it's identified, they're not practicing good design, they're practicing cobbling together something on a rickety foundation. How's that going to help next time one rights something from scratch?
- biocomputation 10y ago
  Yes, to me, 'software development' is 'refactoring', just like some authors say that 'writing' is actually 'rewriting'.

jdavis703 10y ago

It's easier to just start over again with one small part. I've worked at two places with massive monolithic tech debt ridden codebases. It was nearly impossible to rewrite anything because everything touched everything. Now the place I currently work has micro services. Some are legacy tech debt that no one wants to touch. But if needed they could probably be rewritten in a couple of months. It's the difference between selling AAA bonds and junk bonds. You're probably far more likely to pay off those low interest AAA ones than the junk bonds.

hinkley 10y ago
The tradeoff though is that you can't get a handle on how expensive a particular call is because the domino effect of knock-on calls is very difficult to see. You can get a system that 'works' on test data but doesn't scale to production (from a user's perspective, this code doesn't work, even though the dev team will insist that it does)
- computerex 10y ago
  Engineering is about trade offs. Microservices are a fairly recent innovation, and there are many kinks that need to be worked out. Another issue is the overhead cost of maintaining a large number of decoupled, independent services. It's plausible to have hundreds or thousands of microservices, and the tools to visualize/manage these systems haven't yet been created.
  - erichocean 10y ago
    > Microservices are a fairly recent innovation Yeah, no. Micro-services are neither recent, nor innovative. Fun fact: a friend of my consults for large pensions funds in The Netherlands. One of the projects he was brought in on earlier this year to get an outside perspective on was this gigantic monument to micro-services that had been built up over a two year period by ~200 developers, at a cost of around 50 million euros. After two months, every developer was fired, the project cancelled, and the entire 50 million euro cost written off. But it had great micro-services, hundreds and hundreds of them!

hinkley 10y ago

That's a very excellent question, to which I have most of an answer, if not 'the' answer. Daylight is the best antiseptic. It's not giving them a microservices architecture and Docker that solves problems. It's expecting them to use it that causes problems to be solved, or at least formally identified. I do a lot of root cause analysis whether my boss asks for it or not. Human factors are almost always in play. A significant source of bugs that make it into late QA or even production are caused by wishful thinking about the scope and impact of code changes. Things that should have been red flags are brushed off because you can't prove that their code caused the problem, and as long as there is reasonable doubt about the source, some people won't look at code they already declared done. When the code is developed and tested in an isolated system then the only source of state changes on that system were caused by the new code. It takes a real set of brass ones or an extremely dense skull to deflect concerns about problems seen on a system that only contains your code changes. People either shape up or get labeled as untrustworthy. The former result is preferable, but at least with the latter you get predictability out of the system.

0xfaded 10y ago
I would argue it would be simpler to introduce test coverage tools that automatically call out bad test coverage during code review. Microservices remind me of a particular C++ protocol implementation I wrote as a novice programmer. Since the protocol was structured, I had high level classes broken down into simpler classes. e.g. virtual class Reader { void read(Buffer *buf); }; class TCharacter: public Reader { TString *name; TList<TStat> *stats; ... }; class TStat: public Reader { TWord32 *remaining; TWord8 *percent; }; class TWord32: public Reader { uint32_t v; }; class TWord8: public Reader { uint8_t v; }; And on, and on, and on, and on ... I thought it was cute that down to the simplest PODT, every type satisfied a well defined `Reader` interface. I hand wrote many constructors, deconstructors, read and several other interface definitions for each type. Elegant, arguably. Inefficient, yes. Overly complex, undoubtedly. Every argument for microservices has in some way reminded me of this code, with it's well defined interfaces and overly verbose implementation. My current strategy is to utilise the crap out of language features to safely achieve what I want in the minimal amount of code possible. In C++ this would be done by initialising the protocol structure classes straight out of the memory buffer, using #pragma pack as required. No error prone constructors / deconstructors / read() required. Haskell and Scala are two languages I use a lot, and each have powerful type systems which I can use to prevent myself from doing something stupid. Programmable macros are great for removing boilerplate. Want less technical debt? Write less code. In my own controversial opinion, if somebody can't work out how to modify my code because they can't work out how to correctly update the type definitions, that person has no business modifying my code. Problem solved! Interfaces are well defined and modifiable only by people who truly understand the code.

maxxxxx 10y ago

Yeah, that's the 'shiny new toy' fallacy. In every project there are points when it's clear that something wasn't designed right. Some people decide to get deeper into their mess and some people decide to refactor. It's a judgement call and not an easy one. I don't think I'd want to work with microservices and Docker containers designed by the same people who messed up another codebase.

sbov 10y ago
Considering management has a very large role in most messed up codebases I've seen, even if they fire 100% of the engineering team who coded it you probably are working with (for) the same people who messed up a codebase.
- maxxxxx 10y ago
  Management is a part of the team that messed up the codebase, not only the engineers. That's why I am always wary of managers bringing in new methods (microservices, scrum, whatever). They had a big part in previous failures and if they don't admit their own failures nothing will change.
  - collyw 10y ago
    The majority of managers don't give a shit about code quality. A fairly significant percentage of developers don't give a shit either.
- mseebach 10y ago
  That's a pretty dangerous mindset, and one that seems to be pretty popular around here, that all failure is automatically the fault of management. Yes, management plays a role, that's almost tautological, but it takes two to tango, and at the end of the day, the dev team wrote the messed up code. Grandly messing up in just six months, even if encouraged by management, takes willing participation from the developers. Also, I've seen plenty of developers who were perfectly capable of screwing up codebases without being induced to do so by management (I've also seen lots of inept management, don't get me wrong. But I wrote the bad code, they didn't.)

beat 10y ago

Doesn't need to be an "and". Just getting the development/ops environments into a virtualized architecture like Docker can make a tremendous difference in a monolithic app. Quite often, the technical debt is due to configuration differences in manually managed environments - development, QA, and production can differ in small or large ways. I don't know how many times I've seen things like developing on Windows and deploying on Unix, or testing on a single server but deploying into a cluster. Docker (or Vagrant, or others) can fully automate provisioning and configuration, simplifying and enforcing consistency between environments. Likewise, microservices from monoliths can be done without containers. Just pick a chunk of the monolith that can be pried loose, and start with that. This can solve all sorts of problems.

gvb 10y ago

"Corgibytes often introduces new clients to Docker, making it much easier and faster to set up new developer environments." (emphasis mine) She isn't advocating using Docker for the target environment, she is advocating using Docker to set up a standard development environment that allows the developers to be productive[1] in 10 minutes rather than 10 hours. I've been in the situation where the development environment is a poorly documented, massively multi-step steaming pile. The result is I spent a lot of time chasing down those responsible for the steaming pile (the "rockstar" in the department) asking them how to get the development environment working. Typically they pull the keyboard away from me, type a bunch, and then say "there, it works." Which it does. For a while. Rinse and repeat for every developer. Regularly. [1] Well, at least building the system rather than staring at mysterious very broken error messages.

marcosdumay 10y ago
Hum... You are trading an opaque irreproducible environment for another opaque environment that you reproduce by copying verbatim. Technically, it's progress. But I can't imagine how people get satisfied by this.
- Arcsech 10y ago
  I'm not too experienced with Docker, but aren't Docker images built using Dockerfiles, which are essentially glorified shell scripts? In that case, wouldn't the Dockerfile be a relatively transparent way to see what's needed to reproduce an environment?
  - lallysingh 10y ago
    Yes, yes they are. And they're written down where you can read them yourself. And maybe even version controlled!
    - GrinningFool 10y ago
      On the other hand, tracing back a dockerfile that pulls in an obscure one-off base image, then repeating that all the way up the chain, you can find yourself left with a dozen Dockerfiles that you need to read in the right order to know what you have. That's if you can even find the original Dockerfiles - I recently ran into a situation where a container image was in the registry, but the original Dockerfile was completely lost. Docker/containers in general move the complexity to a different plane, but they only move it - they don't eliminate it. Care needs to be taken to avoid situations just like the one above - or any of a number of other pitfalls - or you end up with an environment that's far more complex to understand than a VM housing a monolithic dev environment.

benologist 10y ago

It's not even "technical debt" until someone thinks "technically ... we could have done this more easily in X". Now it's a question of developer comfort and huge investment will be made to make sure everyone building the product are having the best experience doing so. All the old code and knowledge is now debt because it can't come with us on this wonderful journey.

codingmyway 10y ago
It's only debt if you are paying for it going forward. The thought should be "things would be easier going forward if we'd done X". If it does the job and doesn't create issues then you just purchased it without taking on debt. This is easier if it is isolated/loosely coupled. There's no point going back to improve it unless it saves you time/money in the future. If you make a better one to get the 'best experience' you are throwing away time and money for a slightly better mousetrap.
- benologist 10y ago
  It's ironic because we're trying to build software others will depend on, meanwhile the software that we depend on tries to avoid the kind of disruptive change that accompanies rewritten-with-X... that's why we can depend on it.

mscottford 10y ago

You're right to say that a microservices architecture and Docker won't solve the problem. That's not an accurate view of what Corgibytes advocates for. Microservices and Docker are tactical tools that can help many teams, but they don't work everywhere. What does work is assisting that team to slowly clean up the mess that they've built[1] by focusing on incremental improvements. [1]: And we find that almost all messes were created for very understandable reasons.

matwood 10y ago

One of the main causes of technical debt and bad code overall is breaking out of the conventions of the code base. Good code reviews can help this, and solid leads of course, but things can still slip through. Like or hate microservices, one thing they do is maintain a clear, hard boundary on the interfaces. There is no let me just reach into this class here, or call that there. It does work to keep people honest when the pressure is on. Will it turn bad programmers into good ones? No, but I think the base assumption is that you're not necessarily dealing with bad programmers.

collyw 10y ago
Personally I would say that crappy abstractions (or no abstractions) are what makes code shitty.