4 ms·

I'm now working on a codebase which is quite large (13 micro-services required to run the main product); all containerized to run on Kubernetes. The learning cu

I'm now working on a codebase which is quite large (13 micro-services required to run the main product); all containerized to run on Kubernetes. The learning curve was quite steep but luckily, I was familiar with most of the tech so that made it easier (I guess that's why they hired me). The project has been around for over 10 years so it has a lot of legacy code and different repos have different code styles, engine versions and compatibility requirements. The biggest challenge is that it used to be maintained by a large team and now there are just 2 developers. Also, the dev environment isn't fully automated so it takes like 20 minutes just to launch all the services locally for development. The pace of work means that automating this hasn't been a priority. It's a weird experience working on such project because I know for a fact that it would be possible to create the entire project from scratch using only 1 to 3 services max and we would get much better performance, reliability, maintainability etc... But the company wouldn't be willing to foot the cost of a refactor so we have to move at steady snail's pace. The slow pace is because of the point mentioned in the article; the systems are all intertwined and you need to understand how they integrate with one another in order to make any change. It's very common that something works locally but doesn't work when deployed to staging because things are complicated on the infrastructure side with firewall rules, integration with third-party services, build process, etc... Also, because there are so many repos with different coding styles and build requirements, it's hard to keep track of everything because some bug fixes or features I implement touch on like 4 different repos at the same time and because deployment isn't fully automated, it creates a lot of room for error... Common issues include forgetting to push one's changes or forgetting to make a PR on one of the repos. Or sometimes the PR for one of the repos was merged but not deployed... Or there was a config or build issue with one of the repos that was missed because it contained some code which did not meet the compatibility requirements of that repo...

jimbokun 2y ago
If you can’t change, test and deploy each service independently from the others, you don’t really have separate services. You just have a Ball of Mud held together with HTTP calls.
- jongjong 2y ago
  We can test and change them independently, some features just require changing just one repo but most features affect multiple repos. But yes, essentially, it's a ball of mud in our case anyway because the separation between the two most important microservices is unclear (cannot be explained simply, there is no clear logic/responsibility separation besides the fact that one is an older codebase and the other is newer). I've worked on a lot of projects in my career and this one has one of the most complex/chaotic architectures I've seen yet. Surprisingly, it recovers from service downtimes and reboots pretty well. The main issues are maintainability, deployment and configuration. It's often the case that local env does not match staging when building features.

jimbokun 2y ago

If you can’t change, test and deploy each service independently from the others, you don’t really have separate services. You just have a Ball of Mud held together with HTTP calls.

jongjong 2y ago
We can test and change them independently, some features just require changing just one repo but most features affect multiple repos. But yes, essentially, it's a ball of mud in our case anyway because the separation between the two most important microservices is unclear (cannot be explained simply, there is no clear logic/responsibility separation besides the fact that one is an older codebase and the other is newer). I've worked on a lot of projects in my career and this one has one of the most complex/chaotic architectures I've seen yet. Surprisingly, it recovers from service downtimes and reboots pretty well. The main issues are maintainability, deployment and configuration. It's often the case that local env does not match staging when building features.