7 ms·

It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but

by 9dev 11mo ago

It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues: You want straightforward, self-contained deployments for one, instead of uploading files onto your single server. If the process crashes or your harddisk dies, you want redundancy so even those twelve customers can still access the application. You want a CI pipeline, so the junior developer can't just break prod because they forgot to run the tests before pushing. You want proper secret management, so the database credentials aren't just accessible to everyone. You want a caching layer, so you're not surprised by a rogue SQL query that takes way too long, or a surge of users that exhaust the database connections because you never bothered to add proper pooling. Adding guardrails to protect your team from itself mandates some complexity, but just hand-waving that away as unnecessary is a bad answer. At least if you're not working as part of a team.

omnicognate 11mo ago
Conway's Law: > Organizations which design systems... are constrained to produce designs which are copies of the communication structures of these organizations.
- whilenot-dev 11mo ago
  Tell this to a company of 4 engineers that created a system with 40 microservices, deployed as one VM image, to be running on 1 machine.
  - omnicognate 11mo ago
    LOL, perhaps the communication structure there was "silent, internalised turmoil".
    - whilenot-dev 11mo ago
      Probably =), or Conway's law was always about the lower ends of the communication nodes in a company graph. I think it's time we also always include the upper ends of our cognitive limits of multitasking when we design systems in relation to organization structures.
  - noir_lord 11mo ago
    They wouldn't have time to hear it because they'd be trying to fix their local dev environment. I worked for a company that had done pretty much that - not fun at all (for extra fun half the microservices where in a language only half the dev team had even passing familiarity with). You need someone in charge with "taste" enough to not allow that to happen or it can happen.
- hinkley 11mo ago
  Lesser known trick: reorganize your teams so the code isn’t batshit.
  - noir_lord 11mo ago
    That does imply that the people in the business with the authority to do that know how to do that and they in my experience don't - they can't solve a problem they don't understand and are unwilling to delegate it to someone who can understand it. The same pattern repeats across multiple companies - it comes down to trust and delegation, if the people with the power are unwilling to delegate bad things happen.
Freak_NL 11mo ago
> You want a CI pipeline, so the junior developer can't just break prod because they forgot to run the tests before pushing. Make them part of your build first. Tagging a release? Have a documented process (checklist) that says 'run this, do that'. Like how in a Java Maven build you would execute `mvn release:prepare` and `mvn release:perform`, which will execute all tests as well as do the git tagging and anything else that needs doing. Scale up to a CI pipeline once that works. It is step one for doing that anyway.
- BlindEyeHalo 11mo ago
  Why not do a CI pipeline from the beginning instead of relying on trust that no one ever forgets to run a check, considering adding CI is trivial with gitlab or github.
  - 9dev 11mo ago
    Because it adds friction, and whoever introduces that CI pipeline will be the one getting messages from annoyed developers, saying "your pipeline isn't working again". It's definitely a source of complexity on its own, so something you want to consider first.
    - spoiler 11mo ago
      U agree it adds a bit of complexity, but all code adds complexity. Maybe interacted with CIs too much and it's Stockholm syndrome, but they are there to help tame and offload complexity, not just complexity for complexity'a sake
      - 9dev 11mo ago
        > they are there to help tame and offload complexity, not just complexity for complexity'a sake Theoretically. Practically, you're hunting for the reason why your GitHub token doesn't allow you to install a private package from another repository in your org during the build, then you learn you need a classic personal access token tied to an individual user account to interact with GitHub's own package registry, you decide that that sounds brittle and after some pondering, you figure that you can just create a GitHub app that you install in your org and write a small action that uses the GitHub API to create an on-demand token with the correct scopes, and you just need to bundle that so you can use it in your pipeline, but that requires a node_modules folder in your repository, and… Oh! Could it be that you just added complexity for complexity's sake?
isodev 11mo ago
I'm not sure why your architecture needs to be complex to support CI pipelines and proper workflow for change management. And some of these guidelines have grown into satus quo common recipes. Take your starting database for example, the guideline is always "sqlite only for testing, but for production you want Postgres" - it's misleading and absolutely unnecessary. These defaults have also become embedded into PaaS services e.g. the likes of Fly or Scaleway - having a disk attached to a VM instance where you can write data is never a default and usually complicated or expensive to setup. All while there is nothing wrong with a disk that gets backed up - it can support most modern mid sized apps out there before you need block storage and what not.
- hinkley 11mo ago
  Years ago we had someone who wanted to make sure that two deployments were mutually exclusive. Can’t recall why now, but something with a test environment and bootstrapping so no redundancy. I just set one build agent up with a tag that both plans required. The simplest thing that could possibly work.
- 9dev 11mo ago
  I've been involved in bootstrapping the infrastructure for several companies. You always start small, and add more components over time. I dare say, on the projects I was involved, we were fairly successful in balancing complexity, but some things really just make sense. Using a container orchestration tool spares you from tending to actual Linux servers, for example, that need updates and firewalls and IP addresses and managing SSH keys properly. The complexity is still there, but it shifts somewhere else. Looking at the big picture, that might mean your knowledge requirements ease on the systems administration stuff, and tighten on the cloud provider/IaC end; that might be a good trade off if you're working with a team of younger software engineers that don't have a strong Linux background, for example, which I assume is pretty common these days. Or, consider redundancy: Your customers likely expect your service to not have an outage. That's a simple requirement, but very hard to get right, especially if you're using a single server that provides your application. Just introducing multiple copies of the app running in parallel comes with changes required in the app (you can't assume replica #1 will handle the first and second request—except if you jump through sticky session hoops, which is a rabbit hole on its own), in your networking (HTTP requests to the domain must be sent to multiple destinations), and your deployment process (artefacts must go to multiple places, restarts need to be choreographed). Many teams (in my experience) that have a disdain for complex solutions will choose their own, bespoke way of solving these issues one by one, only to end up in a corner of their own making. I guess what I'm saying is pretty mundane actually—solve the right problem at the right time, but no later.
pjc50 11mo ago
I think that's a slightly different set of things to what OP is complaining about though. They're much more reasonable, but also "outside" of the application. Having secret management or CI (pretty much mandatory!) does not dictate the architecture of the application at all. (except the caching layer. Remember the three hard problems of computer science, of which cache invalidation is one.) Still hoping for a good "steelman" demonstration of microservices for something that isn't FAANG-sized.
- hinkley 11mo ago
  Cache invalidation is replacing one logical thing with a new version of the same logical thing. So technically that’s also naming things. Doubly so when you put them in a kv store.
  - kragen 11mo ago
    That angle seems potentially insightful, and I'm going to have to think about it, but to me, cache invalidation seems more like replacing one logical thing with nothing. It may or may not get replaced with a new version of the same logical thing later if that's required.
    - cluckindan 11mo ago
      To me, cache invalidation is not strictly about either replacing or removing cache entries. Rather, cache invalidation is the process of determining which cache entries are stale and need to be replaced/removed. It gets hairy when determining that depends on users, user group memberships AND per-user permissions, access TTL, multiple types of timestamps and/or revision numbering, and especially when the cache entries are composite as in contain data from multiple database entities, where some are e.g. representing a hierarchy and may not even have direct entity relationships with the cached data.
      - kragen 11mo ago
        Yes—and, in many cases, ensuring that you don't use entries which become outdated during your computation.
macspoofing 11mo ago
>It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues: You want straightforward, self-contained deployments for one, instead of uploading files onto your single server ... You can get all that with a monolith server and a Postgres backend.
- tnel77 11mo ago
  In this job market, how am I supposed to get hired without the latest buzzwords on my resume? I can’t just have monolithic server and Postgres! (Sarcasm)
  - chistev 11mo ago
    Indicating sarcasm ruins the sarcasm
    - reactordev 11mo ago
      Squint or pretend it’s not there. This crowd is hit or miss on picking it up o’ naturál.
    - sfn42 11mo ago
      If you don't make it clear people will think you're serious. Sarcasm doesn't work online, If I write something like "Donald Trump is the best president ever" you don't have any way of knowing whether I'm being sarcastic or I'm just really really stupid. Only people who know me can make that judgement, and basically nobody on here knows me. So I either have to avoid sarcasm or make it clear that I'm being sarcastic.
      - chistev 11mo ago
        Context of the comment would tell
    - tnel77 11mo ago
      Sadly, it is missed on a lot of people. Without the disclaimer, I would then have a bunch of serious replies “educating” me about my life choices.
zelphirkalt 11mo ago
Sure, but most of that doesn't make it into the final production thing on the server. CI? Nope. Tests? Nope. The management of the secrets (not the secrets themselves)? Nope. Caching? OK that one does. Rate limits? Maybe, but could be another layer outside the normal services' implementation.
lelanthran 11mo ago
> If the process crashes or your harddisk dies, you want redundancy so even those twelve customers can still access the application. That's fine, 6 of them are test accounts :-) > It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues If you have an entire organisation dedicated to 6 users, those users had better be ultra profitable. > If the process crashes or your harddisk dies, you want redundancy so even those twelve customers can still access the application Can be done simply by a sole company owner; no need for tools that make sense in an organisation (K8s, etc) > You want a CI pipeline, so the junior developer can't just break prod because they forgot to run the tests before pushing. A deployment script that includes test runners is fine for focused product. You can even do it using a green/blue strategy if you can afford the extra $5-$10/m for an extra VPS. > You want proper secret management, so the database credentials aren't just accessible to everyone. Sure, but you don't need to deploy a full-on secrets-manager product for this. > You want a caching layer, so you're not surprised by a rogue SQL query that takes way too long, or a surge of users that exhaust the database connections because you never bothered to add proper pooling. Meh. The caching layer is not to protect you against rogue SQL queries taking too long; that's not what a cache is for, after all. As for proper pooling, what's wrong with using the pool that came with your tech stack? Do you really need to spend time setting up a different product for pooling? > dding guardrails to protect your team from itself mandates some complexity, but just hand-waving that away as unnecessary is a bad answer. I agree with that; the key is knowing when those things are needed, and TBH unless you're doing a B2C product, or have an extremely large B2B client, those things are unnecessary. Whatever happened to "profile, then optimise"?

by 9dev 11mo ago

omnicognate 11mo ago

Conway's Law: > Organizations which design systems... are constrained to produce designs which are copies of the communication structures of these organizations.

whilenot-dev 11mo ago
Tell this to a company of 4 engineers that created a system with 40 microservices, deployed as one VM image, to be running on 1 machine.
- omnicognate 11mo ago
  LOL, perhaps the communication structure there was "silent, internalised turmoil".
  - whilenot-dev 11mo ago
    Probably =), or Conway's law was always about the lower ends of the communication nodes in a company graph. I think it's time we also always include the upper ends of our cognitive limits of multitasking when we design systems in relation to organization structures.
- noir_lord 11mo ago
  They wouldn't have time to hear it because they'd be trying to fix their local dev environment. I worked for a company that had done pretty much that - not fun at all (for extra fun half the microservices where in a language only half the dev team had even passing familiarity with). You need someone in charge with "taste" enough to not allow that to happen or it can happen.
hinkley 11mo ago
Lesser known trick: reorganize your teams so the code isn’t batshit.
- noir_lord 11mo ago
  That does imply that the people in the business with the authority to do that know how to do that and they in my experience don't - they can't solve a problem they don't understand and are unwilling to delegate it to someone who can understand it. The same pattern repeats across multiple companies - it comes down to trust and delegation, if the people with the power are unwilling to delegate bad things happen.

Freak_NL 11mo ago

> You want a CI pipeline, so the junior developer can't just break prod because they forgot to run the tests before pushing. Make them part of your build first. Tagging a release? Have a documented process (checklist) that says 'run this, do that'. Like how in a Java Maven build you would execute `mvn release:prepare` and `mvn release:perform`, which will execute all tests as well as do the git tagging and anything else that needs doing. Scale up to a CI pipeline once that works. It is step one for doing that anyway.

BlindEyeHalo 11mo ago
Why not do a CI pipeline from the beginning instead of relying on trust that no one ever forgets to run a check, considering adding CI is trivial with gitlab or github.
- 9dev 11mo ago
  Because it adds friction, and whoever introduces that CI pipeline will be the one getting messages from annoyed developers, saying "your pipeline isn't working again". It's definitely a source of complexity on its own, so something you want to consider first.
  - spoiler 11mo ago
    U agree it adds a bit of complexity, but all code adds complexity. Maybe interacted with CIs too much and it's Stockholm syndrome, but they are there to help tame and offload complexity, not just complexity for complexity'a sake
    - 9dev 11mo ago
      > they are there to help tame and offload complexity, not just complexity for complexity'a sake Theoretically. Practically, you're hunting for the reason why your GitHub token doesn't allow you to install a private package from another repository in your org during the build, then you learn you need a classic personal access token tied to an individual user account to interact with GitHub's own package registry, you decide that that sounds brittle and after some pondering, you figure that you can just create a GitHub app that you install in your org and write a small action that uses the GitHub API to create an on-demand token with the correct scopes, and you just need to bundle that so you can use it in your pipeline, but that requires a node_modules folder in your repository, and… Oh! Could it be that you just added complexity for complexity's sake?

isodev 11mo ago

I'm not sure why your architecture needs to be complex to support CI pipelines and proper workflow for change management. And some of these guidelines have grown into satus quo common recipes. Take your starting database for example, the guideline is always "sqlite only for testing, but for production you want Postgres" - it's misleading and absolutely unnecessary. These defaults have also become embedded into PaaS services e.g. the likes of Fly or Scaleway - having a disk attached to a VM instance where you can write data is never a default and usually complicated or expensive to setup. All while there is nothing wrong with a disk that gets backed up - it can support most modern mid sized apps out there before you need block storage and what not.

hinkley 11mo ago
Years ago we had someone who wanted to make sure that two deployments were mutually exclusive. Can’t recall why now, but something with a test environment and bootstrapping so no redundancy. I just set one build agent up with a tag that both plans required. The simplest thing that could possibly work.
9dev 11mo ago
I've been involved in bootstrapping the infrastructure for several companies. You always start small, and add more components over time. I dare say, on the projects I was involved, we were fairly successful in balancing complexity, but some things really just make sense. Using a container orchestration tool spares you from tending to actual Linux servers, for example, that need updates and firewalls and IP addresses and managing SSH keys properly. The complexity is still there, but it shifts somewhere else. Looking at the big picture, that might mean your knowledge requirements ease on the systems administration stuff, and tighten on the cloud provider/IaC end; that might be a good trade off if you're working with a team of younger software engineers that don't have a strong Linux background, for example, which I assume is pretty common these days. Or, consider redundancy: Your customers likely expect your service to not have an outage. That's a simple requirement, but very hard to get right, especially if you're using a single server that provides your application. Just introducing multiple copies of the app running in parallel comes with changes required in the app (you can't assume replica #1 will handle the first and second request—except if you jump through sticky session hoops, which is a rabbit hole on its own), in your networking (HTTP requests to the domain must be sent to multiple destinations), and your deployment process (artefacts must go to multiple places, restarts need to be choreographed). Many teams (in my experience) that have a disdain for complex solutions will choose their own, bespoke way of solving these issues one by one, only to end up in a corner of their own making. I guess what I'm saying is pretty mundane actually—solve the right problem at the right time, but no later.

pjc50 11mo ago

I think that's a slightly different set of things to what OP is complaining about though. They're much more reasonable, but also "outside" of the application. Having secret management or CI (pretty much mandatory!) does not dictate the architecture of the application at all. (except the caching layer. Remember the three hard problems of computer science, of which cache invalidation is one.) Still hoping for a good "steelman" demonstration of microservices for something that isn't FAANG-sized.

hinkley 11mo ago
Cache invalidation is replacing one logical thing with a new version of the same logical thing. So technically that’s also naming things. Doubly so when you put them in a kv store.
- kragen 11mo ago
  That angle seems potentially insightful, and I'm going to have to think about it, but to me, cache invalidation seems more like replacing one logical thing with nothing. It may or may not get replaced with a new version of the same logical thing later if that's required.
  - cluckindan 11mo ago
    To me, cache invalidation is not strictly about either replacing or removing cache entries. Rather, cache invalidation is the process of determining which cache entries are stale and need to be replaced/removed. It gets hairy when determining that depends on users, user group memberships AND per-user permissions, access TTL, multiple types of timestamps and/or revision numbering, and especially when the cache entries are composite as in contain data from multiple database entities, where some are e.g. representing a hierarchy and may not even have direct entity relationships with the cached data.
    - kragen 11mo ago
      Yes—and, in many cases, ensuring that you don't use entries which become outdated during your computation.

macspoofing 11mo ago

>It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues: You want straightforward, self-contained deployments for one, instead of uploading files onto your single server ... You can get all that with a monolith server and a Postgres backend.

tnel77 11mo ago
In this job market, how am I supposed to get hired without the latest buzzwords on my resume? I can’t just have monolithic server and Postgres! (Sarcasm)
- chistev 11mo ago
  Indicating sarcasm ruins the sarcasm
  - reactordev 11mo ago
    Squint or pretend it’s not there. This crowd is hit or miss on picking it up o’ naturál.
  - sfn42 11mo ago
    If you don't make it clear people will think you're serious. Sarcasm doesn't work online, If I write something like "Donald Trump is the best president ever" you don't have any way of knowing whether I'm being sarcastic or I'm just really really stupid. Only people who know me can make that judgement, and basically nobody on here knows me. So I either have to avoid sarcasm or make it clear that I'm being sarcastic.
    - chistev 11mo ago
      Context of the comment would tell
  - tnel77 11mo ago
    Sadly, it is missed on a lot of people. Without the disclaimer, I would then have a bunch of serious replies “educating” me about my life choices.

zelphirkalt 11mo ago

Sure, but most of that doesn't make it into the final production thing on the server. CI? Nope. Tests? Nope. The management of the secrets (not the secrets themselves)? Nope. Caching? OK that one does. Rate limits? Maybe, but could be another layer outside the normal services' implementation.

lelanthran 11mo ago

> If the process crashes or your harddisk dies, you want redundancy so even those twelve customers can still access the application. That's fine, 6 of them are test accounts :-) > It's sure a corny stance to hold if you're navigating an infrastructure nightmare daily, but in my opinion, much of the complexity addresses not technical, but organisational issues If you have an entire organisation dedicated to 6 users, those users had better be ultra profitable. > If the process crashes or your harddisk dies, you want redundancy so even those twelve customers can still access the application Can be done simply by a sole company owner; no need for tools that make sense in an organisation (K8s, etc) > You want a CI pipeline, so the junior developer can't just break prod because they forgot to run the tests before pushing. A deployment script that includes test runners is fine for focused product. You can even do it using a green/blue strategy if you can afford the extra $5-$10/m for an extra VPS. > You want proper secret management, so the database credentials aren't just accessible to everyone. Sure, but you don't need to deploy a full-on secrets-manager product for this. > You want a caching layer, so you're not surprised by a rogue SQL query that takes way too long, or a surge of users that exhaust the database connections because you never bothered to add proper pooling. Meh. The caching layer is not to protect you against rogue SQL queries taking too long; that's not what a cache is for, after all. As for proper pooling, what's wrong with using the pool that came with your tech stack? Do you really need to spend time setting up a different product for pooling? > dding guardrails to protect your team from itself mandates some complexity, but just hand-waving that away as unnecessary is a bad answer. I agree with that; the key is knowing when those things are needed, and TBH unless you're doing a B2C product, or have an extremely large B2B client, those things are unnecessary. Whatever happened to "profile, then optimise"?