6 ms·
Taking on Good Technical Debt
- swatcoder 4y ago> I'm an avid supporter of Kubernetes (see About me), but I would never suggest that a small team adopt it from the start. Even with expert knowledge of Kubernetes, taking it on as a dependency still creates loads of maintenance debt. Somewhere along the way, it became fashionable to build out your devops on day one as though you’ll be Google tomorrow, but to write your code without any architectural foresight as though the only thing that matters is tomorrow’s deployment. It’s always nice to see people advocating towards a thoughtful balance.
- mikepurvis 4y agoI'm only a light k8s user, but isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure? Like, there's no point spinning up EC2 nodes just to run microk8s or k0s on them— have ansible scripts or whatever that just run your application directly. But the calculus changes if instead of EC2 nodes you can have a helm chart that deploys directly to a managed kubernetes services like EKS or GKE?
- nicoburns 4y agoThat's still a bunch more complicated than other hosted solutions. I once worked for a 4 year old company whose entire infrastructure consisted of a managed postgres instance + a moderately sized single VM (and a duplicated setup as a staging environment).
- colechristensen 4y agoEh, I've found kubernetes more complicated than the interface of a cloud provider, you always have to do the cloud provider as well for some things, and the benefits of the pre-baked helm charts or whatever else just aren't there until the relatively small difference in efficiency in each way of doing things makes a real difference as a result of scale. If you have a lot of experience with any interface, you can do it really efficiently, but if you don't, kubernetes can make your experience worse than anything you'd come up with on a cloud provider.
- irjustin 4y agoCan't you run K8 on Fargate so you don't have to manage EC2 nodes themselves? Even with ECS which is way simpler than K8 for small teams using Fargate has been incredibly beneficial in terms of debt/time. Then next step would be to move the entire application onto Lambda so we're not paying for downtime, but that's a whole headache and a half.
- makeitdouble 4y agoFun with k8s starts when you realize it has no LTS and gets updated on a fixed schedule. Any given version is only officialy supported for about 1 year and a half. Then you'll need non trivial amount of knowledge of how it works to avoid many pitfalls (the helm chart will probably only go so far, it will still need small tweaks for your specific use case) So it's a combination of upfront time investment, ongoing tuning, and recurring maintenance events.
- p_l 4y agoHonestly, I find Helm to be a trap that is going to reduce your capability to run service reliably - or understand it. It's probably one of the most annoying bits in kubernetes ecosystem
- alexalx666 4y agoWhat are the benefits of k8s over docker-compose for small/medium business?
- maccard 4y agoAll of the cloud providers offer k8s as a service, for one.
- alexalx666 4y agoI should specify I was thinking about VPS. Coming from AWS to Linode VPS with Caddy + Docker felt great, Im wondering at what point (N requests per second ?) it becomes tech dept
- maccard 4y agoYeah that's definitely something that needs to be specified. > Coming from AWS to Linode VPS with Caddy + Docker felt great I guess that depends what you were using with AWS, but Fargate + k8s is a pretty amazing workflow. > Im wondering at what point (N requests per second ?) it becomes tech dept I make these calls for our org, and I wouldn't call it tech debt based on the performance or scale, but because of maintenance. As an example, if you needed to hire a new person/replacement person for your team, would it be easier to find someone who can maintain k8s or docker compose? The other area it shines for is the ecosystem. If you need to add monitoring to your app (e.g. with Prometheus), adding it to a k8s cluster is `helm install prometheus`
- alexalx666 4y agothanks for detailed response! how big is your org? My concern is that there is a cost to magic of adding deps like that. Complexity goes up and it is harder to understand what went wrong and where are the performance bottlenecks. I guess my question is at what org scale the tradeoffs are right
- 4y ago
- lucideer 4y ago> isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure? That's a big part of many k8s pitches I've seen but in my experience it hasn't been representative of reality (or at least for every part of infrastructure you no longer need to run there's a new part of k8s architecture/config to learn & manage). For me the selling points are: 1. For the time-cost of upfront complexity and setup, you get extremely easy and quick subsequent addition of infra/tooling & scaling thereof. 2. There are certain patterns of tooling that k8s enables (relatively easily), that would have been difficult/impossible with more traditional setups. I don't think it makes infra management overhead less, it just reorganises it so it's more upfront and then less later while scaling.
- deleted 4y ago[deleted]
- al_borland 4y agoMy boss was trying to push me to use some new CICD pipeline the company built, just so he could understand what CICD meant. I kept ignoring him, because it just didn't seem needed and saw no need to complicate things. To build the servers I was running on I wrote an Ansible job to install everything and copy the code down. All I had to do was add a couple flags to that to allow it to just deploy new changes. Instead of getting into the CICD pipeline I just took a few minute to make a little web front end for myself to kick off the job and watch the job status (we have Tower, but it's painfully slow). I can now deploy in 18 seconds with 1 click. Seems good enough. I'm sure I could set up a web hook to eliminate the click, but it's not a big deal.
- maccard 4y agoIve been banging this drum on here for a while, but we use AWS with ECS. I can spin up an ecs cluster with one runner in less than 10 minutes, and setting up a CICD pipeline is a handful of clicks on GitHub actions, or about 10 lines of pipeline .yml
- al_borland 4y agoI went through a React course that used Github and some online service to setup CICD in a few clicks with some service. It was painless. At my company, it would involve going to weekly meetings to get all the updates and changes happening as they keep developing it, one-off meetings to get onboarded and setup... I'm pretty sure they also designed the system for Kubernetes, so I figured it would also mean containerizing the site, as it has been running on VMs for almost 10 years. Those weekly meetings also mean constant changes so things keep working (which I'm not dealing with on a different site that is going though CICD). All of this in the midst of a lot of bureaucracy. It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google. It likely would have taken several months for little to no tangible benefit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited. It took 8 months for someone to bother to tell me where I could see build status/errors, at first they just told me to "wait about 10 minutes". There is no documentation anywhere. It's all tribal knowledge and no one seems to have the time to share it. The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.
- Chris_Newton 4y agoThis is surely resume (or consultant) driven development in action. High Scalability™ is like an epic side quest in the game of making software. The final boss at the end of the quest, Yagni, can only be defeated by those who have reached a large enough scale already. Many attempt the side quest too early and as a result get so distracted that they suffer delays or even lose the whole game.
- p_l 4y agoIt really depends on what you're going to run and what experience you have. Personally, I would run k8s from the day I expect to run the code as separate deployment to running it from my IDE. I have the skills to do so, and I found I lose way less time doing so than trying to do "classic" server setup, even if it's just one server.
- Deritio 4y agoA small team should use the central k8s platform if it exists and it's the company base infrastructure layer. Everything else would only mean less golden path support, less synergies etc.
- lucideer 4y agoBy "small team" I assume they meant a start-up - i.e the team is the company's entire dev. As someone who works in a small team within a large corp where we use k8s for everything large and small, I did not read that statement as applying to me.
- quickthrower2 4y agoGood APIs between your code (that means programmatic interfaces, not necessarily over the network) help alot. If you have a subsystem conform to a good API and is well behaved it can be rewritten. It is allowed to have some tech debt. If the subsystem is sprawled through other subsystems that tech debt doesn't get paid down because now you need an expert in all those areas to fix it. I swear I have worked places where there is code no one currently at the company can understand. At best you micro-fix for bugs and back the hell away!
- jef_leppard 4y agoGood APIs and interfaces are everything. I call it "the hotdog principle" when I am mentoring folks who are new to the concept. A good API is like a hot dog: Hot dogs are delicious. That is all you need to know about them. You don't want to know whats inside them or details of how they get made. You will be sad if you do. It's funny and gross enough to stick most of the time.
- quickthrower2 4y agoDon’t tell me how the sausage was made! That said sometimes abstractions leak. I was using Firebase and getting the public Url for a blob given it’s path is an async call! God knows why.
- eftychis 4y agoHere we go again! Trying to use a financial term to describe "bad" past decisions. I do not see this term used when we build bridges or houses -- we either say they were time or financial constraints or it was a bad job. And certainly intermediate (temporary or not) supporting constructions are not viewed as debt. Nobody built the pyramids in a day. Here is a different attempt on definition: An occurrence on debt is when a higher up decides that a product/construction must be ready when it clearly is not, and thus convinces/bullies the engineers and put their signature on it. Analogies can be a great tool to communicate or for prose but don't take them literally.
- louwrentius 4y agoI agree with you but your definition was never “take shortcuts that reduce quality”. Ward Cunningham was explicit about this. “TD was about taking the time to incorporate new understanding and learnings into the code, if you don’t you accrue TD”. Unfortunately your description is what has become the common usage of TD, and now the term is meaningless. Frankly speaking, today, TD is just a neutral term to externalize blame/responsibility.
- eftychis 4y agoYes, I was trying to give a contemporary definition. But I do find it interesting that it is a notion in our field. I agree with you and your disappointment. Thank you for the comment.
- louwrentius 4y agoYes indeed, thank you too.
- keithnz 4y agothis seems wrong, the examples given aren't examples of technical debt. Starting a monorepo isn't technical debt, changing to something else later isn't technical debt. Technical debt would happen when doing that change you delay a part of the migration, or leave out some testing because its too tricky, or anything you are delaying till some time later that ideally should be done now.
- selcuka 4y agoIt depends. If you knew that you would certainly need a distributed architecture in the future but you still implemented it as a monorepo, then it's technical debt. Obviously it's not if the requirements have changed.
- keithnz 4y agothat's not really technical debt, if mono repo makes sense at the moment, and you will change later, it's not technical debt, that's just a choice, there's no debt. While you are using the monorepo, there's nothing to "pay off". If after you make the change, but you don't do it fully, then you start having things to "pay off".
- selcuka 4y ago> if mono repo makes sense at the moment That's the opposite of what I wrote. My scenario was you knew that it didn't make sense for that project but you still did it because of time/budget/resource reasons. You start paying off the day you can't scale it anymore.
- keithnz 4y agotime/budget/resource reasons means it DOES make sense. It's like saying you want a Ferrari, but at the moment all you can afford is a bicycle... its a start, it gets you going, and it maybe not what you want long term, and you can't realistically go long distance, but it gets you started. They are just strategic decisions, not technical debt.
- KingOfCoders 4y agoPlease stop using that term. The idea was to communicate "technical stuff that needs to be done" to product management and sometimes have a lever to rework the shortcuts taken due to business pressure - in that case it should be called product debt, and not tech debt. Product took out the money (earlier release), not technology. And needs to pay it back with scheduling development work. What I see is that the term makes technology look bad, doesn't work as a lever and demotivates developers. Be a craftman, don't do technical debt. For time to market create a zerocode/crappy MVP that you throw away. Schedule real technical debt like DB migrations and do them as needed.
- jayceedenton 4y agoI do agree that the term feels like a misnomer, but mainly because it gives an air of respectability to simple bad engineering with great cost. The idea is that debt is accrued intentionally and strategically, but everyone is so used to constantly referring to tech debt that just doing a job well has been completely forgotten. If there are real problems, can we just have a list of tasks to improve things? Maybe it's just boredom but the term 'tech debt' doesn't feel that it's doing much good any more. > For time to market create a zerocode/crappy MVP that you throw away. Unfortunately what tends to happen is that as the business sees early success and starts to grow, there is never a stopping point where people are happy to start again and build from scratch 'properly'. That crappy MVP will be extended, adjusted and corrected, until it's something no longer throwaway but also not well built/tested/hosted/monitored/etc. Maybe slow down to go faster is a better approach.
- KingOfCoders 4y ago"That crappy MVP will be extended, adjusted and corrected, until it's something no longer throwaway" Yes and from talking to my coachees, this is one of their main problems. I urge them to throw away the MVP when they got funded.
- Deritio 4y agoTechnical debt is if you purposely don't write unit tests because you don't have the time for it. Or clean code because no one will ever touch it again. Or a feature you only need once and just write it somehow. And then all this shit stays in your product and starts dragging you down. This is totally different than when you start with a monolith and know that you might want to rework it later but accept potential future work. And you should start with a modular monolith anyway. I don't think it's reasonable to try to redefine the term technical debt.