11 ms·

Why is everything so scalable?

• radarsat1 1y ago
  > scalability needs a whole bunch of complexity I am not sure this is true. Complexity is a function of architecture. Scalability can be achieved by abstraction, it doesn't necessarily imply highly coupled architecture, in fact scalability benefits from decoupling as much as possible, which effectively reduces complexity. If you have a simple job to do that fits in an AWS Lambda, why not deploy it that way, scalability is essentially free. But the real advantage is that by writing it as a Lambda you are forced to think of it in stateless terms. On the other hand if suddenly it needs to coordinate with 50 other Lambdas or services, then you have complexity -- usually scalability will suffer in this case, as things become more and more synchronous and interdependent. > The monolith is composed of separate modules (modules which all run together in the same process). It's of course great to have a modular architecture, but whether or not they run in the same process should be an implementation detail. Barriers should be explicit. By writing it all depending on local, synchronous, same-process logic, you are likely building in all sorts of implicit barriers that will become hidden dangers when suddenly you do need to scale. And by the way that's one of the reasons we think about scaling in advance, is that when the need comes, it comes quickly. It's not that you should scale early. But if you're designing a system architecture, I think it's better to think about scaling, not because you need it, but because doing so forces you to modularize, decouple, and make synchronization barriers explicit. If done correctly, this will lead to a better, more robust system even when it's small. Just like premature optimization -- it's better not to get caught up doing it too early, but you still want to design your system so that you'll be able to do it later when needed, because that time will come, and the opportunity to start over is not going to come as easily as you might imagine.
  • CaptainOfCoit 1y ago
    > It's of course great to have a modular architecture, but whether or not they run in the same process should be an implementation detail It should be, but I think "microservices" somehow screwed up that. Many developers think "modular architecture == separate services communicating via HTTP/network that can be swapped", failing to realize you can do exactly what you're talking about. It doesn't really matter what the barrier is, as long as it's clear, and more often than not, network seems to be the default barrier when it doesn't have to be.
    • worldsayshi 1y ago
      > network seems to be the default barrier when it doesn't have to be. But if you want to use off the shelf solutions to your problems it often is. You can't very well do 'from keycloak import login_page'.
  • dapperdrake 1y ago
    The complexity that makes money is all the essential complexity of the problem domain. The "complexity in the architecture" can only add to that (and often does). This is the part that is about math as a language for patterns as well as research for finding counter-examples. It’s not an engineering problem yet. Once you have product market fit, then it becomes and engineering problem.
  • saidinesh5 1y ago
    > If you have a simple job to do that fits in an AWS Lambda, why not deploy it that way, scalability is essentially free. But the real advantage is that by writing it as a Lambda you are forced to think of it in stateless terms. What you are describing is already the example of premature optimization. The moment you are thinking of a job in terms of "fits in an AWS Lambda" you are automatically stuck with "Use S3 to store the results" and "use a queue to manage the jobs" decisions. You don't even know if that job is the bottleneck that needs to scale. For all you know, writing a simple monolithic script to deploy onto a VM/server would be a lot simpler deployment. Just use the ram/filesystem as the cache. Write the results to the filesystem/database. When the time comes to scale you know exactly which parts of your monolith are the bottleneck that need to be split. For all you know - you can simply replicate your monolith, shard the inputs and the scaling is already done. Or just use the DB's replication functionality. To put things into perspective, even a cheap raspberry pi/entry level cloud VM gives you thousands of postgres queries per second. Most startups I worked at NEVER hit that number. Yet their deployment stories started off with "let's use lambdas, s3, etc..". That's just added complexity. And a lot of bills - if it weren't for the "free cloud credits".
    • bpicolo 1y ago
      > The moment you are thinking of a job in terms of "fits in an AWS Lambda" you are automatically stuck with "Use S3 to store the results" and "use a queue to manage the jobs" decisions. I think the most important one you get is that inputs/outputs must always be < 6mb in size. It makes sense as a limitation for Lambda's scalability, but you will definitely dread it the moment a 6.1mb use case makes sense for your application.
      • hedora 1y ago
        The counterargument to this point is also incredibly weak: It forces you to have clean interfaces to your functions, and to think about where the application state lives, and how it's passed around inside your application. That's equivalent to paying attention in software engineering 101. If you can't get those things right on one machine, you're going to be in world of hurt dealing with something like lambda.
• CaptainOfCoit 1y ago
  I've seen startups killed because of one or two "influential" programmers deciding they need to start architecturing the project for 1000TPS and 10K daily users, as "that's the proper way to build scalable software", while the project itself hasn't even found product-market fit yet and barely has users. Inevitably, the project needs to make a drastic change which now is so painful to do because it no longer fits the perfect vision the lead(s) had. Cue programmers blaming the product team for "always changing their mind" as they discover what users actually need, and the product team blaming developers for being hesitant to do changes, and when programmers agree, it takes a long time to undo the perfect architecture they've spent weeks fine-tuning against some imaginary future user-base.
  • the8472 1y ago
    1000TPS isn't that much? Engineer for low latency and with a 10ms budget that'd be 10 cores if it were CPU-bound, less in practice since usually part of the time is spent in IO wait.
    • drob518 1y ago
      And with CPUs now being shipped with 100+ cores, you can brute force that sucker a long way.
    • CaptainOfCoit 1y ago
      > 1000TPS isn't that much? Why does that matter? My argument is: Engineer for what you know, leave the rest for when you know better, which isn't before you have lots of users.
      • the8472 1y ago
        What I'm saying is that "building for 1000TPS" is not what gets you an overengineered 5-layer microservice architecture. If you build for a good user experience (which includes low latency) you get that not-that-big scale without sharding.
    • hedora 1y ago
• acron0 1y ago
  Ugh, there is just something so satisfying about developer cynicism. It gives me that warm, fuzzy feeling. I basically agree with most of what the author is saying here, and I think that my feeling is that most developers are at least aware that they should resist technical self-pleasure in pursuit of making sure the business/product they're attached to is actually performing. Are there really people out there who still reach for Meta-scale by default? Who start with microservices?
  • lpapez 1y ago
    > Are there really people out there who still reach for Meta-scale by default? Who start with microservices? Anecdotally, the last three greenfield projects I was a part of, the Architects (distinct people in every case) began the project along the lines of "let us define the microservices to handle our domains". Every one of those projects failed, in my opinion not primarily owing to bad technical decisions - but they surely didn't help either by making things harder to pivot, extend and change. Clean Code ruined a generation of engineers IMO.
    • robertlagrant 1y ago
      I think this sounds more like Domain Driven Design than Clean Code.
      • ahoka 1y ago
        It kinda started with Clean Code. I remember some old colleagues walking around with the book in their hand and deleting ten year old comments in every commit they made: "You see, we don't need that anymore, because the code describes itself". It made a generation (generations?) of software developers think that all the architectural patterns were found now, we can finally do real engineering and just have to find the one that fits for the problem at hand! Everyone asked the SOLID principles during interviews, because that's how real engineers design! I think "cargo cult" was getting used at that time too to describe this phenomenon.
        • 1y ago
• jwr 1y ago
  I don't get this scalability craze either. Computers are stupid fast these days and unless you are doing something silly, it's difficult to run into CPU speed limitations. I've been running a SaaS for 10 years now. Initially on a single server, after a couple of years moved to a distributed database (RethinkDB) and a 3-server setup, not for "scalability" but to get redundancy and prevent data loss. Haven't felt a need for more servers yet. No microservices, no Kubernetes, no AWS, just plain bare-metal servers managed through ansible. I guess things look different if you're using somebody else's money.
  • floating-io 1y ago
    For how many users, and at what transaction rate? Not disagreeing that you can do a lot on a lot less than in the old days, but your story would be much more impactful with that information. :)
  • drob518 1y ago
    One of the silliest things you can do to cripple your performance is build something that is artificially over distributed, injecting lots of network delays between components, all of which have to be transited to fulfill a single user request. Monoliths are fast. Yes, sometimes you absolutely have to break something into a standalone service, but that’s rare.
    • hedora 1y ago
      I've notice a strong correlation between artificially over-distributing, and not understanding things like the CAP theorem. So, you end up with a slow system that's added a bunch of unsolvable distributed systems problems on its fast path. (Most distributed systems problems are solvable, but only if the person that architected the system knows what they're doing. If they know what they're doing, they won't over-distribute stuff.)
      • drob518 1y ago
        Yes, that too. If you look at the commits for Heisenbugs associated with the system, you have a good chance of seeing artificial waits injected to “fix” things.
• _ZeD_ 1y ago
  Honestly, in my experience, the only good reason to have microservices in a "software solution" is to be able to match 1 service -> 1 mantainer/team and have a big (read "nested", with multiple level of middle-managers) group of teams, each that may have different goals. In this way it's very easy to "map" a manager/team to a "place" in the solution map, with very explicit and documented interactions between them
  • nycdotnet 1y ago
    This is Conway’s Law: You ship your org chart.
  • hsn915 1y ago
    Nice story but the places I've seen that make use of services, there's never a "1 server -> 1 team". It's more like 20 services distributed among 3 teams, and some services are "shared" by all teams
    • rvitorper 1y ago
      I can relate to this
• smokel 1y ago
  ThoughtWorks gathers this phenomenon under the term "envy": Web Scale envy [1] or Big Data envy [2] are two relevant blips on their technology radar. It is typically better to keep things simple. [1] https://www.thoughtworks.com/radar/techniques/high-performance-envy-web-scale-envy https://www.thoughtworks.com/radar/techniques/high-performan... [2] https://www.thoughtworks.com/radar/techniques/big-data-envy https://www.thoughtworks.com/radar/techniques/big-data-envy
• abujazar 1y ago
  I've seen my share of insanely over-engineered Azure locked-in applications that could easily have been run on an open source stack on a $20 VM.
  • hedora 1y ago
    But what if payroll grows to 100M internal users?
    • abujazar 1y ago
      Yea, and what if my country suddenly grows by 10000%?
    • WJW 1y ago
      Seems like a great problem to have. Surely one of those millions of employees can be used to change the current system at that point. Until then, no reason to overengineer it.
    • roncesvalles 1y ago
      I sense sarcasm but just to add on, most software problems actually have very predictable natural upper bounds. There are only X number of people in the world growing at a well-defined rate. There are only X number of people in your country, or in your TAM (e.g. if you're a restaurant) etc. This is especially true for B2B. The need to accommodate runaway scale (unbounded N and unbounded rate of growth of N) is actually quite rare.
• yobbo 1y ago
  Many startup business models have no chance of becoming profitable unless they reach a certain scale, but they might have less than 1% probability of reaching that scale. Making it scalable is easy work since it is deterministic, but growing customers is not. Another perspective is that the defacto purpose of startups (and projects at random companies) may actually be work experience and rehearsal for the day the founders and friends get to interview at an actual FAANG. I think the author's “dress for the job you want, not the job you have” nails it.
  • nicoburns 1y ago
    I guess the work is deterministic, but it often (unintentionally) makes the systems being developed non-deterministic!
    • potatolicious 1y ago
      Ah yes. I once worked at a startup that insisted on Mongo despite not having anywhere near the data volume for it to make any sense at all. Like, we're talking 5 orders of magnitude off of what one would reasonably expect to need a Mongo deployment. I was but a baby engineer then, and the leads would not countenance anything as pedestrian as MySQL/Postgres. Anyway, fast forward a bit and we were tasked with building an in-house messaging service. And at that point Mongo's eventual consistency became a roaring problem. Users would get notifications that they had a new message, and then when they tried to read it it was... well... not yet consistent. We ended up implementing all kinds of ugly UX hacks to work around this, but really we could've run the entire thing off of sqlite on a single box and users would've been able to read messages instantaneously, so...
      • nicoburns 1y ago
        I've seen similar with Firebase. Luckily I took over as tech lead at this company, so I was able to migrate us to Postgres. Amusingly, as well as being more reliable, the Postgres version (on a single small database instance) was also much faster than the previous Firebase-based version (due to it enabling JOINs in the database rather than in application code).
• hsn915 1y ago
  I think it was around 2015 when everything was basically AWS and Kubernetes The turning point might have been Heroku? Prior to Heroku, I think people just assumed you deploy to a VPS. Heroku taught people to stop thinking about the production environment so much. I think people were so inspired by it and wanted to mimic it for other languages. It got more people curios about AWS. Ironically, while the point of Heroku was to make deployment easy and done with a single command, the modern deployment story on cloud infrastructure is so complicated most teams need to hold a one hour meeting with several developers "hands on deck" and going through a very manual process. So it might seem counter intuitive to suggest that the trend was started by Heroku, because the result is the exact opposite of the inspiration.
  • deleted 1y ago
    [deleted]
• esher 1y ago
  I can relate - running a small hosting business. People come up with too complex solutions. They solve problems that they'd wish to have. For instance: HA setups are complex. If not done correctly, like in most cases, people don't gain the additional '9' from the SLA.
• llm_nerd 1y ago
  This piece is written with a pretty cliche dismissive tone that assumes that everything everyone else does is driven by cargo-culting if not outright ignorance. That people make these choices because they're just rushing to chase the latest trend. They're just trying to be cool, you see. Here's the thing, though: Almost every choice that leads to scalability also leads to reliability. These two patterns are effectively interchangeable. Having your infra costs be "$100 per month" (a claim that usually comes with a massive disclaimer, as an aside) but then falling over for a day because your DB server crashed is a really, really bad place to be.
  • blueflow 1y ago
    > Here's the thing, though: Almost every choice that leads to scalability also leads to reliability. How is that supposed to happen. Without k8 involved somehow?
    • 97nomad 1y ago
      There is a lot of instruments, that don't need k8s to be scalable and reliable. Starting from stateless services and simple load balancers and ending with actor systems like in Erlang or Akka.
  • crazygringo 1y ago
    > Almost every choice that leads to scalability also leads to reliability. Empirically, that does not seem to be the case. Large scalable systems also go offline for hours at a time. There are so many more potential points of failure due to the complexity. And even with a single regular server, it's very easy to keep a live replica backup of the database and point to that if the main one goes down. Which is a common practice. That's not scaling, just redundancy.
    • llm_nerd 1y ago
      >Empirically, that does not seem to be the case. Failures are astonishingly, vanishingly rare. Like it's amazing at this point how reliable almost every system is. There are a tiny number of failures at enormous scale operations (almost always due to network misconfigurations, FWIW), but in the grand scheme of things we've architected an outrageously reliable set of platforms. >That's not scaling, just redundancy. In practice it almost always is scaling. No one wants to pay for a whole n server just to apply shipped logs to. I mean, the whole premise of this article is that you should get the most out of your spend, so in that case much better is two hot servers. And once you have two hot...why not four, distributed across data centers. And so on.
• TickleSteve 1y ago
  scale vertically before horizontally... - scaling vertically is cheaper to develop - scaling horizontally gets you further. What is correct for your situation depends on your human, financial and time resources.
• lambdaone 1y ago
  I had a client with a system just like this. EBS, S3, RDS, Cognito, the lot. It cost $00s per month under almost no load, and was a maintenance nightmare - which was the real problem, not the cost, as it stopped working altogether eventually. A bit of hacking later, it all fits on a single VM that costs ~$10/month to run and is far easier to build, deploy and maintain.
• drob518 1y ago
  > The first problem every startup solves is scalability. The first problem every startup should solve is “how do we have enough money to not go bust in two months”, but that’s a hard problem, whereas scalability is trivially solvable by reading a few engineering blogs, and anyway it’s not like anyone will ever call you out on it, since you’ll go bust in two months. I laughed. I cried. Having a back full of microservices scars, I can attest that everything said here is true. Just build an effin monolith and get it done.
• DrScientist 1y ago
  Isn't it simple as the following? Break your code into modules/components that have a defined interface between them. That interface only passes data - not code with behaviour - and signal the method calls may fail to complete ( ie throw exceptions ). ie the interface could be a network call in the future. Allow easy swapping of interface implementations by passing them into constructors/ using factories or dependency injection frameworks if you must. That's it - you can then start with everything in-process and the rapid development that allows, but if you need to you can add splitting into networked microservices - any complexity that arises from the network aspect is hidden behind the proxy, with the ultimate escape hatch of the exception. Have I missed something?
  • crazygringo 1y ago
    You're not missing much, but I don't understand why you're just basically repeating what the article already says. Except the article also says to use a monorepo.
    • DrScientist 1y ago
      I think I've added a couple of elements to make it possible to scale your auth service if you need to. Easily swappable implementations and making sure the interfaces advertise that calls may simply fail. Even so it's still very simple. To scale your auth service you just write a proxy to a remote implementation and pass that in - any load balancing etc is hidden behind that same interface and none of the rest of the code cares.
      • crazygringo 1y ago
        Good point! Sorry if I was being ungenerous. I like the idea of the remote implementation being proxied -- not sure I've come across that pattern before.
    • stavros 1y ago
      No, I'm saying you don't need to use a monorepo! The repo discussion is a bit orthogonal, and up to you to decide whether you want a single repo or multiple repos with modules/libraries that get deployed together.
• BirAdam 1y ago
  Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several PHP servers in front of a MySQL database that had one primary r/w with one read only replica. Storage (at that time) was usually done with glusterfs. This was scalable enough at the time for hundreds of thousands of concurrent users, though the video quality was quite a bit lower than what people expect today. Today at AWS, it is easily possible for people to spend a multiple of the cost of that hardware setup every month for far less compute power and storage.
  • aeyes 1y ago
    The architecture you describe is ok because in the end it is a fairly simple website. Little user interaction, limited amount of content (at most a few million records), few content changes per day. The most complex part is probably to have some kind of search engine but even with 10 million videos an ElasticSearch index is probably no larger than 1GB. The only problem is that there is a lot of video data.
    • ben_w 1y ago
      This is probably also true for 98% of startups. I think most people don't realise that "10 million" records is small, for a computer. (That said, I have had to deal with code that included an O(n^2) de-duplication where the test data had n ~= 20,000, causing app startup to take 20 minutes; the other developer insisted there was no possible way to speed this up, later that day I found the problem, asked the CTO if there was a business reason for that de-duplication, removed the de-duplication, and the following morning's stand-up was "you know that 20 minute startup you said couldn't possibly be sped up? Yeah, well, I sped it up and now it takes 200ms")
      • phkahler 1y ago
        I thought you were going to say to reduced O(n^2) to O(n*log(n)), but you just deleted the operation. Normally I'd say that's great, but just how much duplicate data is being left around now? Is that OK?
• gampleman 1y ago
  Hilariously written but also too true. One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time. Another place I worked at we were paying for 2 dev ops engineers (and those guys don't come cheap) to maintain our deployment cluster for 3 apps which each had a single customer (with a handful of users). This whole operation had like 20 people and an engineering team of 8.
  • andoando 1y ago
    We have the same shit and its super annoying too cause in addition I cant do shit without going through the dev ops team even though were 5 engineers.
  • radiator 1y ago
    This sounds just about right: I have read that Kubernetes is the greek term for "more containers than customers".
  • Thiez 1y ago
    What were these dev ops engineers doing all day? Surely you can only polish a cluster so much before it's done and there is nothing left to do?
    • gampleman 1y ago
      You should have seen the architecture they came up with... it had ALL the bells and whistles you could possibly imagine and cost an absolute fortune. Of course they eventually got bored and quit. And then it became really annoying since no one else understood anything about it.
    • jeffrallen 1y ago
      It takes approximately 3 months to get it "just right". Luckily, k8s releases (and CNI and auth sidecars, and...) release every 2.8 months.
• tesdinger 1y ago
  > you usually only have one database What if I use the cloud? I don't even know how many servers my database runs on. Nor do I care. It's liberating not having to think about it at all.
  • sgarland 1y ago
    And your cloud providers thank you for giving their executives a third yacht.
• JohnMakin 1y ago
  If you’ve ever been in a situation where you do suddenly face scale and have to rip apart a legacy monolith that was built without scale in mind, you’ll chuckle at this article. It’s extremely painful.
  • sgarland 1y ago
    Legitimately asking, how? The only bottleneck should be the DB, and if you can saturate a 128-core DB, I want to see your queries and working set size. Not saying it can’t happen, but it’s rare that someone has actually maxed out MySQL or Postgres without there being some serious schema and query flaws, or just poor / absent tuning.
    • JohnMakin 1y ago
      You’re thinking purely in terms of app performance. have you ever seen a terrible db schema? Having to suddenly iterate fast with a brittle codebase that doesnt really allow that ive seen bring teams to their knees for a year+. I’ve seen monoliths because of their sheer size and how much crap and debt is packed into them, build and deploy processes taking several hours if not an entire day for some fix that could be ci/cd’d in seconds if it wasn’t such a ball of mud. Then, what tends to happen, is the infrastructure around it tends to compensate heavily for it, which turns into its own ball of mud. Nothing wrong with properly scaled monoliths but it’s a bit naive, in my personal experience, to just scoff at scale when your business succeeding relies on scale at some point. Don’t prematurely optimize, but don’t be oblivious to future scenarios, because they can happen quicker than you think
      • whstl 1y ago
        That was the reality of the fintech I worked at. The schema wasn't really a problem, but the sheer amount of queries per request. Often a user opening a page or clicking a button would cause 100-200 database queries, including updates. This would prevent strategies such as "just replicating the data somewhere". It was so badly architected that every morning the app would stop responding due to users doing their morning routine operations. And they only had around 300 employees. And this was just an internal app, the B2C part was already isolated because we couldn't afford to be offline. The solution I started working on was doing similar to the strangler fig pattern and replacing parts of the API with new code that talked directly to the ORM. Naturally this didn't made the people who wrote the legacy code happy, but at least the outages stopped.
• ben_w 1y ago
  Some related blog posts of mine with some thematic overlap: • https://benwheatley.github.io/blog/2025/02/26-14.04.07.html https://benwheatley.github.io/blog/2025/02/26-14.04.07.html • https://benwheatley.github.io/blog/2024/04/07-21.31.19.html https://benwheatley.github.io/blog/2024/04/07-21.31.19.html
• pdhborges 1y ago
  Scale articles are too focused on architecture. What about business problems that come with scale. At a certain scale rare events are common many cases cease to be fixable by some random process that involves humans you have to handle a lot more business scenarios with your code.
• reactordev 1y ago
  I read this, and have the opposite experience. Your monolith will fester as developers step on each others toes. You aren’t solving for scalability, you’re solving for sovereignty. Giving other teams the ability to develop their own service without needing to conform to your archaic grey beard architecture restrictions and your lack of understanding what a pod is or how to get your logs from your cloud. No, this whole article reads like someone who is crying that they no longer have their AS/200. Bye. The reason people use AWS and all those 3rd party is so they don’t have to reinvent the wheel which this author seems hell bent on. Why are we using TCP when a Unix file is fine… why are we using databases when a directory and files is fine? Why are we scaling when we aren’t Google when my single machine can serve a webpage? Why am I getting paid to be an engineer while eschewing all the things that we have advanced over the last two decades? Yeah, these are not the right questions. The real question should be: “Now that we have scale what are we gonna do with it?”
  • sgarland 1y ago
    > Giving other teams the ability to develop their own service without needing to conform to your archaic grey beard architecture restrictions IME at many different SaaS companies, the only one that had serious reliability was the one that had “archaic grey beard architecture restrictions.” Devs want to use New Shiny X? Put a formal request before the architectural review committee; they’ll read it, then explain how what the team wants already exists in a different form. I don’t know why so many developers - notably, not system design experts, nor having any background in infrastructure - think that they know better than the gray beards. They’ve seen some shit. > and your lack of understanding what a pod is or how to get your logs from your cloud. No one said the gray beards don’t know this. At the aforementioned company, we ran hybrid on-prem and AWS, and our product was hybrid K8s and traditional Linux services. Re: cloud logs, every time I’ve needed logs, it has consistently been faster for me to ssh onto the instance (assuming it wasn’t ephemeral) and use ripgrep. If I don’t know where the logs were emitted from, I’ll find that first, then ssh. The only LaaS I’ve used that was worth a damn was Sumologic, but I have no idea how they are now, as that was years ago.
    • fragmede 1y ago
• Havoc 1y ago
  Comes down to knowing when to stop. You don’t really want to DIY your own orchestrator etc. So better off just using kubernetes. But then not going too far down that rabbit hole. ie yes kubernetes but the simplest vanilla version of it you can manage
  • sgarland 1y ago
    I wouldn’t start with K8s, and I’ve administered it at multiple companies. Unless every one of your initial hires is a SWE turned SRE, you’re in for a bad time (and you don’t need it). I’d personally start with Linux services on some VMs, but Docker Compose is also valid. There are plenty of wrappers around Compose to add features if you’d like.
  • apf6 1y ago
    Simplest version of Kubernetes is zero Kubernetes. You can instead run your service using a process manager like PM2 or similar. I think even using Docker is overkill for a lot of small teams.
• sreekanth850 1y ago
  Frontend folks weren’t happy with how simple things were, so after seeing microservices, they invented microfrontends, and balance in complexity restored.
• mjr00 1y ago
  > The first problem every startup should solve is “how do we have enough money to not go bust in two months”, but that’s a hard problem, whereas scalability is trivially solvable by reading a few engineering blogs [...] Do you know what the difference between Google and your startup is? It’s definitely not scalability, you’ve solved that problem. It’s that Google has billions upon billions with which to pay for that scalability, which is really good because scalability is expensive. Too true. Now that I've stepped into an "engineering leadership" role and spend as much time looking at finances as I do at code, I've formed the opinion that in 99.999% of cases, engineering problems are really business problems. If you could throw infinite time and money at the technical challenges, they'd no longer be challenging. But businesses, especially startups, don't have infinite (or even "some") money and time, so the challenge is doing the best engineering work you can, given time and budget constraints. > The downsides [of the monolith approach] I like the article's suggestion of using explicitly defined API boundaries between modules, and that's a good approach for a monolith. However one massive downside that cannot be ignored -- by having a single monolith you now have an implicit dependency on the same runtime working on all parts of your code. What I mean by this is, all your code is going to share the same Python version and same libraries (particularly true in Python, where it's not a common/well-supported use case to have multiple versions of library dependencies). This means that if you're working on Module A, and you realize you need a new feature from Pandas 2.x, but the rest of the code is on Pandas 1.x... well, you can't upgrade unless you go and fix Modules B, C, D ... Z to work with Pandas 2. This won't be an issue at the start, but it's worth pointing out. Being forced to upgrade a core library or language runtime and finding out it's a multi-month disruptive project can be brutal.
  • mejutoco 1y ago
    > by having a single monolith you now have an implicit dependency on the same runtime working on all parts of your code I think many people would not consider what I am going to suggest a monolith, but docker compose all hosted in the same server acts a lot like a monolith. > particularly true in Python, where it's not a common/well-supported use case to have multiple versions of library dependencies You can use virtual environments each with its python version and packages.
• okaleniuk 1y ago
  Most of the startups actually plan to survive for more than 2 months. And it makes total sense to think about scalability, reliability, and performance while it's still possible to change your whole stack every other week. Not forgetting about other things such as securing your cash flow, growing your talent pool, protecting your IP, etc. Finding a good balance between multiple focii is exactly the job for a founder. Of course, it's a hard job, that's why we don't see many successful startups to begin with.
• zokier 1y ago
  In lot of contexts scaling down is far more important than scaling up. In that sense scalability is cost-optimization; instead of provisioning fixed capacity that is enough for (predicted) peak loads, you can scale based on actual demand and save money or have higher utilization.
• tobyhinloopen 1y ago
  It isn't hard to make something a bit scalable, but it is very hard to make it scalable _later_.
• pigcat 1y ago
  My friend is the first dev hire at a startup where they prematurely overengineered for scalability. The technical founders had recently exited a previous startup and their rationale was that it makes a future acquisition easier, since a potential acquirer will weigh scalability in their evaluation of the code (and maybe even conflate it with quality). In fact it was a regret from their first startup that they hadn't baked in scalability earlier. I remain skeptical of the decision, but curious if there's any truth to the fact that acquirers weigh scalability in their scorecard?
  • danielmarkbruce 1y ago
    Sure, if the acquirer thinks the product is going to sell a lot. A relatively common plan (it doesn't always work) for large enterprise software companies is to buy a product and then use their very large sales force to sell it into all their existing customers. If thats the plan, you have to make sure the product will work with all the increased usage. I'd still suggest it's far better to optimize for building the right product - the "is this going to scale" problem is one of the nicest problems you can face.
• tptacek 1y ago
  Related: Crawshaw's "one process programming": https://crawshaw.io/blog/one-process-programming-notes https://crawshaw.io/blog/one-process-programming-notes
• alpine01 1y ago
  There's a now famous Harvard lecture video on YouTube of Zuckerberg earlier in the Facebook days, where he walks through the issues they hit early on. https://www.youtube.com/watch?v=xFFs9UgOAlE https://www.youtube.com/watch?v=xFFs9UgOAlE I watched it ages ago, but I seem to remember one thing that I liked was that each time they changed the architecture, it was to solve a problem they had, or were beginning to have. They seemed to be staying away from pre-optimization and instead took the approach of tackling problems as they had as they appeared, rather than imagining problems long before/if they occurred. It's a bit like the "perfect is the enemy of done" concept - you could spend 2-3x the time making it much more scalable, but that might have an opportunity cost which weakens you somewhere else or makes it harder/more expensive to maintain and support. Take it with a pinch of salt, but I thought it seemed like quite a good level-headed approach to choosing how to spend time/money early on, when there's a lot of financial/time constraints.
• charlimangy 1y ago
  Working in modern architectures that can scale is pretty important for developers that want to have attractive resumes. Given that your startup has a 9 out of 10 chance of failing you're going to need another job. If you want people to stay you have to give them the security of keeping up with at least some of the latest fashions.
• gamerDude 1y ago
  When I'm working with new developers I always have to convince them to simplify their setup. Why are we on autoscaled, pay by the query infra when we are serving a few people. Then they complain how expensive it is. I had someone tell me that their costs were $1500/mon when they were in demo stages. I asked them why they aren't hosting on a single small server for $20. And they responded that it didn't matter because they were using free credits. Except that those free credits will go away and you'll find yourself not wanting to do all the work to move it over when it would've been easier to do so when you just had that first monolith server up. I think free credits and hyped up technology is to blame. So, basically a gamed onboarding process that gets people to over-engineer and spend more.
  • Capricorn2481 1y ago
    I'm not totally following the cost analysis from some of these comments. I agree that there's no reason to make your architecture overly complicated, especially if it will cost dev time. I'm not saying this as someone that does cloud very much. We normally do a single server. More as an outsider looking in. If you load balance 4 smaller ec2 instances vs a larger one, it's significantly cheaper. There is overhead when you run the same app on 4 machines, but not as much as people think. An idle system uses 300-600mb of ram, leaving roughly 3.5gb vram usable, per machine. 4 t3.medium instances are about $119/mo, vs. an equivalent t3.2xlarge which is $238/mo. Also people will often say Digital Ocean is cheaper, but they compare it to non-equivalent AWS services. A Digital Ocean VPS compared to an equivalent EC2 is about the same. So I understand when people argue about complicating architecture or optimizing prematurely, but I don't think the math on actual server cost really checks out. I would prefer to do dedicated hosting, but Hetzner doesn't have that in the U.S.
• ReptileMan 1y ago
  That is helluva verbose way to quote Knuth ...
• treve 1y ago
  A bit of an alternative take on this, but I talk to a lot of folks at small start-ups (in Toronto, if that matters), but it seems like most people actually get this right and understand not to bring in complexity until later. Things like microservices seems like they are mostly understood as a tool that's not really meant to solve a real scalibility problem and is massive liability early on. The exceptions are usually just inexperienced people at the helm. My feeling is, hire someone with adequate experience and this is likely not an issue. I do think architecture astronauts tend to talk a lot more about their houses of cards, which makes it seem like these set ups are more popular than they are.
• jcarrano 1y ago
  I think part of the problem is (some) programmers being unable to draw clear encapsulation boundaries when writing a monolith. I'm not even referring to imposing a discipline for a whole team, but the ability to design a clean internal API and stick to it oneself.
• sagyam 1y ago
  I have read and watched these articles and videos where people seem to have a problem with Microservice, Kubernetes, cloud providers, or anything that's not a PHP server sitting behind an nginx running on a $5 VPS. I have also seen the front-end analogy of these types of posts, where anything that is not written using HTML, CSS, and jQuery is unnecessary bloat. I will soon write a blog, which I think will cover more points and nuances of both sides. For now, here are some of my scattered thoughts. - If deploying your MVP to EKS is overengineering, then signing a year-long lease for bare metal is hubris. Both think one day they will need it, but only one of them can undo that decision. - Don't compare your JBOD to a multi-region replicated, CDN-enabled object store that can shrug off a DDoS attack. One protects you from those egress fees, and the other protects you from a disaster. They are not comparable. - A year from now, the startup you work for may not exist. Being able to write that you have experience with that trendy technology on your resume sure sounds nice. Given the layoffs we are seeing right now, putting our interest above the company's may be a good idea. - Yes, everyone knows modern CPUs are very fast, and paying $300/mo for an 8-core machine feels like a ripoff, but unless you are business of renting GPUs and selling tokens. Compute was never your cost center; it was always humans. For some companies, not being able to meet your SLA due to talent attrition is scarier than the cloud bill. I know these are one-sided arguments, and I said I would cover both sides with more nuance. I need some time to think through all the arguments, especially on the frontend side. I will soon write a blog.
  • rvitorper 1y ago
    I thought capitalism was about adding value, not conflict of interest
• vbezhenar 1y ago
  Everything is scalable, because it became very easy to write scalable software. I guess that's the reason.
• mannyv 1y ago
  Actually, scalability is cheap. Our AWS bill until recently was around $160-$200 a month. To get the level of HA and performance would require at least 20 boxes in two data centers. Dev/test/prod with an HA db and a backend that never dies. I’ve built those on bare iron and they’re expensive. If you’re going for saas and customers that don’t care about your infrastructure then a hetzner box is fine. But really, creating resilient infrastructure is super cheap now. Agree with “make what your customers want,” but many customers actually want a service that doesn’t barf.
  • n_u 1y ago
    Could you explain this some more? How are your costs so low in comparison? Are you using serverless?
    • bigbuppo 1y ago
      That's how the cloudy platforms get you. They're very cheap on the low end, until they're not.
      • mannyv 1y ago
        No, it's because we changed the way we process our metrics. Previously we processed our metrics by consolidating them into multidimensional entries on a minute basis. We moved to single metric second-based collection, because it was getting too complicated to process and because we wanted second-by-second measurement to measure engagement more granularly. That increased our data retention tremendously. We're still under the cost for the other timestream products, but we'll be adjusting how we do that in a quarter or two.
      • etothepii 1y ago
        I've heard this a few times. Can you explain a bit more why you think that's a problem. I've always made the assumption that once they become "not cheap" you now have the cost to offset investment against.
    • mannyv 1y ago
• deleted 1y ago
  [deleted]
• bogwog 1y ago
  TL;DR: premature optimization is the root of all evil
• aubanel 1y ago
  Great article! It seems like the whole industry has overfitted to the System designs interviews of FAANG, thus focused on extreme scaling need that few companies actually have.
• incorrecthorse 1y ago
  Plot twist: It's not actually scalable because no amount of tools and buzzwords can compensate for the lack of experience in proper architecture for scaling.
• dzonga 1y ago
  There's also the 'myth' of the modular monolith.
• dt3ft 1y ago
  Because Microsoft, Google and Amazon, to name a few, sell the story of the cloud to decision makers who can sign a subscription contract. Developers go with the flow, not daring to question the setup. Meanwhile, they host their own servers on a r-pi and ship sideprojects. Devs are not at fault here. It’s the management.
  • stavros 1y ago
    Having been on both sides, nope, it's 100% the devs pushing for this. I've had innumerable discussions trying to convince developers to simplify things, but their reaction was basically "how can we go against best practice?! Unthinkable!". It was a huge uphill battle. Meanwhile, management didn't have an opinion, it was up to the teams to architect things how they liked, and they liked sprawling microservice everything, because they wanted to own 100% of their code, any other concern be damned.
• shenenee 1y ago
  Man is 100% right; the insanity we put ourselves through for all kinds of hypotheticals is beyond mental
• xg15 1y ago
  > The first problem every startup solves is scalability. The first problem every startup should solve is “how do we have enough money to not go bust in two months” Why is the second question the devs' responsibility? Shouldn't it be the founders'?
  • bcrosby95 1y ago
    Because how the devs decide to build the system influences whether you will have enough money to not go bust in two months.
    • xg15 1y ago
      Good point, but unless the only dev is also a founder, I'd think they aren't the ones with the responsibility to come up with a business model. They definitely have the responsibility to write software so it most efficiently serves that business model.
• afiodorov 1y ago
  I've found that building my side projects to be "scalable" is a practical side effect of choosing the most cost-effective hosting. When a project has little to no traffic, the on-demand pricing of serverless is unbeatable. A static site on S3 or a backend on Lambda with DynamoDB will cost nothing under the AWS free tier. A dedicated server, even a cheap one, is an immediate and fixed $8-10/month liability. The cost to run a monolith on a VPS only becomes competitive once you have enough users to burn through the very generous free tiers, which for many side projects is a long way off. The primary driver here is minimizing cost and operational overhead from day one.
  • mejutoco 1y ago
    > A dedicated server, even a cheap one, is an immediate and fixed $8-10/month liability. Personally, I am more worried about the infinitely-scalable service potentially (liability) sending a huge bill after the fact. This "liability" of $8-10 is predictable, like a Netflix subscription.
• arthurofbabylon 1y ago
  I agree, of course. Choose an architecture for the pragmatic reality, not some fantastical non-reality. However, I appreciate the craft. Some of these unnecessary optimizations (rather, “introduced complexities”) are vestigial accoutrements that come alongside generally good software design. Not all, but some. So I tolerate a fair amount of fanciness in myself and others when it coincides with solid intent and healthy output. That said, we should absolutely not tolerate the presence of appurtenances of complexity at the architectural layer – that is a place reserved for pure 100% pragmatism.
• deleted 1y ago
  [deleted]
• dev_l1x_be 1y ago
  Scalability is solved problem as of 2025, this is why. Do we need it? Probably not, in most cases anyways. Most of the time people do it because they like complex systems.
• koito17 1y ago
  In 2015, a single server with 40 cores and 128 GB of RAM was able to handle 2 million WebSocket connections running an Elixir program[0]. One of my hot takes is that a gaming PC has fast enough hardware to serve thousands of clients with a static Rust binary and SQLite. Pair with Litestream, and you have easy-to-test, continuous backups. It's nice being able to test backups by just running `litestream restore` and then running a single binary on my development machine. Addtionally, when the backend is a single static binary, you gain the opportunity to test the entire system in CI without maintaining an ad-hoc cloud environment or heirarchy of mock services. The points of contention, for me personally, would be managing deployments and observability. Of course, at my workplace, I wouldn't dare to suggest this kind of architecture, but as others have mentioned, a single machine can go a long way, and I doubt most my projects will ever need to scale beyond 40 cores and 128 GB of RAM. [0] https://x.com/chris_mccord/status/659430661942550528 https://x.com/chris_mccord/status/659430661942550528
  • rvitorper 1y ago
    Can’t argue with that. Machines are pretty capable, and elixir is awesome as well
• deadbabe 1y ago
  Unfortunately software engineering suffers from the opposite problem of many other engineering disciplines. Any idiot could build a bridge by just overbuilding everything, an engineer helps you build the minimum viable bridge. In software, it’s the opposite. Idiots can easily roll out products and services with crude and basic code. You only need true engineers for the high volume high performance stuff. And if that’s not what you’re doing – you don’t need engineers. The logical conclusion is then to fire them.
• andersmurphy 1y ago
  A VPS, caddy, sqlite and livestream is really all you need in my experience. Help's if your language has decent support for real and green threads (java/go/clojure).
• nate_martin 1y ago
  >The first problem every startup solves is scalability. I don't think this is true at all. The first problem they solve is typically finding product market fit, which startups will do by sacrificing scalability and quality for speed of execution.
• closeparen 1y ago
  >Modules cannot call each other, except through specific interfaces (for our Python monolith, we put those in a file called some_module/api.py, so other modules can do from some_module.api import some_function, SomeClass and call things that way. This is a solution to a large chunk of what people want out of microservices. There are just two problems, both of which feel tractable to a language/runtime that really wanted to solve them: 1. If the code implementing the module API is private, it must all be colocated in one package. If it is public, then anyone can import it, breaking the module boundary. You need a visibility system that can say "this subtree of packages can cooperate with each other, but code outside the subtree can only use the top-level package." 2. If a change module A has a problem, you must roll back the entire monolith, preventing a good change in module B from reaching users. You need a way to change the deployed version of different modules independently. Short of microservices, they could be separate processes doing some kind of IPC, or you need a runtime with hot reloading (and a reasonably careful backwards compatibility story).
  • aiono 1y ago
    > 1. If the code implementing the module API is private, it must all be colocated in one package. If it is public, then anyone can import it, breaking the module boundary. You need a visibility system that can say "this subtree of packages can cooperate with each other, but code outside the subtree can only use the top-level package." This is easily achieved in Scala with a particular use of package paths. You are allowed to make some symbols only visible under a package path.
    • closeparen 1y ago
      Unfortunately the thought pattern “modern software development is way too complex, we need something radically simpler… I know! I’ll use Scala!” is not a thing.
  • eximius 1y ago
    Visibility systems are great! > If a change module A has a problem, you must roll back the entire monolith, preventing a good change in module B from reaching users. eh. In these setups you really want to be fixing forward for the reason you describe - so you revert the commit for feature A or turn off the feature flag for it or something. You don't really want to be reverting deployments. If you have to, well, then it's probably worth the small cost of feature B being delayed. But there are good solutions to shipping multiple features at the same time without conflicting.
• jrochkind1 1y ago
  > No, no you don’t, you can deploy your modules to their own repos and work on them that way, though then you lose the nice property of being able to atomically deploy changes across your entire codebase when an API changes. That's not all you lose. You also lose being able to have a single git SHA to describe the state of the entire system at any time. And, lose the naturalness of running CI on the entire system at a given state (and knowing what CI ran on what state), although you can rig that up unnaturally of course.
  • cestith 1y ago
    I hear you about rigging those repos together artificially. I used to work somewhere that had several different systems written as monoliths that eventually needed to interact more closely with each other. There was an in-house ticket system (since made a support service wrapped around ZenDesk, but ZD wouldn’t replicate all its functionality). There was an in-house employee management system. There was a first-party CRM. There was a homegrown e-commerce store. They’d built their own licensing servers for their software. Eventually the CRM was managing large customer licenses, the ticket system linked to the store to sell priority support, and the store was selling licenses to smaller customers. So instead of making these things all support APIs that supported the other applications, people started copying libraries around from service to service. Then to make sure those libraries didn’t fall out of date, the (at the time rsync) deployment process for each of those apps was changed to require a pull from every one of those repos, then a push from that staging server to the production servers. Then security did a PCI-DSS internal audit, and the developers couldn’t just get onto a staging server and make direct changes to production. So I, as the lead SRE at the time, wrote a builder web app that takes a config file per project. It holds the data on the repos and the default tag to pull from each. The web app allows the developer to update to a different commit or tag for any particular repo involved. Then a single button pulls everything and serially takes production servers out of rotation, updates them, and puts them back into production. It’s something that could have been avoided many different ways including using a monorepo for those systems.
• liampulles 1y ago
  Early in my career I was given the opportunity (in between consulting jobs) to make an MVP for a revamp of our internal employee management system. I seized the opportunity to deploy my own Kubernetes cluster, and create a sidecar to help with user authentication (because of course we'd need a common way to do this for the multi-language suite of microservices we'd be building). I used up pretty much the entirety of my time designing and architecting how this colossus was going to work, and by the end I realized how foolish the whole endeavor was. That was really an instructive failure - at my next job, I got everyone behind turning our team's microservices back into a modular monolith, and it worked very well.
• mmcnl 1y ago
  Scalability is not purely technical. It's also organizational. For all its drawbacks, the microservices architecture is easier to scale from an organizational perspective.
  • whstl 1y ago
    Only when the service boundaries and interfaces are built with this in mind. A service that is isolated enough it could be another company? Sure, this scales. But do company hierarchies and organization practices help this happen? I haven't seen it outside of places like Amazon where there was a mandate for it to be that way. What companies end up with in practice are services so tightly coupled with the rest of the company that they requiring a mishmash of API requests in both directions and endless coordination. Aka a distributed monolith. All the problems with zero the advantages.
  • dagss 1y ago
    My experience is it adds a multiplier 10x to the work that needs to be done, which is then a good fit for the 9x extra developers upper management decided to hire. Sure, it keeps everyone busy with work they feel is meaningful -- but it really is work that was not really needed in the first place with a monolith. Much simpler to try to stay fewer developers as long as possible. Also in my experience it is impossible to get people along with organizing the company along software architecture division lines, thus creating a lot of political problems.
• rednafi 1y ago
  CV padding is a real thing. Too many engineers love joining moonshot programs at large companies. The reason - being able to play with cool toys with the least amount of accountability. Google has Google problems. So unless you are operating at that scale, blindly adopting their tech won’t solve your problems. But it might bring you a raise.
• oooyay 1y ago
  The architecture the author is describing is called SOA and to me is much more optimal. There's variations of SOA that can occur in a monolith or as separate services but at the end of the day it stresses separation at interfaces that most people like microservices for. Microservices are really only architecturally necessary if certain parts of your application have outlier performance characteristics where it wouldn't make sense to scale the whole thing for any number of reasons (eg: database connection pooling or some other metric). As for why microservices got so popular I think the answer lies in writing. The more a pattern is written on the more likely it is to be repeated and modeled. Finally, the author also goes in after cloud usage in general. We're a decade and a half away from the first of these convergences. I was a systems engineer in those days and I remember how terrible they were. Software engineers requested a virtual machine, a pool of network resources, firewall rules, etc and eventually they got what they needed. The primary motivator of the cloud wasn't scale per se, to me it was that now a competent developer has an API to request those things on. They're operating on machine time instead of human time. Some people extrapolate the same argument above and replace cloud with virtual machine and containers, "Why do I need containers when I can simply operate this load balancer, some VMs on an autoscaling group, and a managed database?!" Again, we quickly forgot that for many software engineer immersing themselves in image pipelines, operating systems maintenance, and networking details bogs them down in releasing the thing they really care about - the thing that logically gets them paid. Containers, again, traded that complexity to another team that solves it once for many people and lets software engineers live a relatively less complex life from their perspective. There's an old adage I used to hear in the Marines that goes something like, "If you're not in the infantry then you're serving it. If you aren't doing that then you should question exactly what it is that you do here." The same can be said for software and those outside of product - we end up living to serve those that are building more closely on the product itself. That's how I contextualize this history/evolution anyway.
• j45 1y ago
  Because when the cloud initially came out 2000-2010, linux had a few things to improve including networking, virtualization. The public cloud provided a way to avoid all of this headache. Once it got figured out, the cloud wasn't the only way to scale anymore. Except, more people than not might not know that today about the cloud. Linux has gotten orders of magnitude more efficient and effective, so has the public cloud, and by extension, so has scaling (and self hosting)..
• Animats 1y ago
  You can get an awful lot done with Go, MySQL or MariaDB, FCGI, Apache or Ngnix. You start with one server on shared hosting for a few dollars a month. Then scale up to a dedicated server, if the load appears. Then scale up to multiple servers with a load balancer. Replicate the database. On the load side, if you're accumulating "statistics" about user behavior, do you really need them for every user? Maybe only one user in a hundred. Or a thousand. When you exceed the limits of that, you're a big company and can afford AWS. A few years ago, we had those guys with the liquid meal (not Jucero, the Soylent guys) boasting about their "infrastructure". Not for making the product, for their web site. From their financials, you could calculate that they were doing about four sales a minute. Their "infrastructure" could run on an Raspberry Pi.
  • roncesvalles 1y ago
    The problem with a VPS/baremetal solution is always high availability. Once you introduce HA, you may as well go into the cloud since HA'ing stateful stuff using VPS is a major pain. Also, you can get a lot done with a serverless (FaaS/PaaS) solution and a simple DB like DynamoDB.
    • foldr 1y ago
      I don't disagree with this, but I think people sometimes miss the fact that even cheap commodity hardware is way more reliable than most software. Your site may go down more often due to bugs in your HA configuration than it would have gone down because of your VPS dying once a year or so. If you are not a huge operation, the sweet spot for HA is probably something very simple (e.g. two servers behind a load balancer, or something like that).
      • Animats 1y ago
        Two servers, load balancer, replicated Postgres database on the back end, offsite backups?
• right2copy 1y ago
  if copyright didn't exist, i'm not so sure that our current data systems would be considered "scalable" - scalability is a relative term.
• ecshafer 1y ago
  The topic of this blog post is the same thing that DHH has been railing on for the last few years. Run your own infrastructure, keep things simple.
• deleted 1y ago
  [deleted]
• scuff3d 1y ago
  A problem not addressed by the article is customer expectations. If you work for a company that does contracting, you have to deal with customers who have enough knowledge to know all the buzz words but not enough knowledge to actually know what they're asking for. If you don't give them Kubernetes and micro services they don't want to pay you.
• shwaj 1y ago
  I recall hearing, more than a decade ago IIRC, that facebook.com ran as a monolith for much longer than you would have expected. Perhaps someone with direct experience can comment?
  • andrewmcwatters 1y ago
    Stack Overflow also famously ran on very little hardware.
• mattbillenstein 1y ago
  I think more engineers would benefit from "you can just build stuff" thinking. Like you don't have to use all the complicated whizz-bang tech that everyone else is using. You can build a boring backend on Linux VMs without containers using open-source software - it's simpler or at least a different level of complexity compared to the big clouds and orchestration systems like k8s, and honestly, it's just more fun to work on - I almost never write yaml - it's a joy. I wrote my own deployment system using this idea - machines, roles, software and services that map to those roles, idempotent operations, and a constantly-connected async rpc system to orchestrate it all. Written from scratch in a language I like with a config language I like. My deploys are often < 10s (if I'm not waiting on webpack to build the UI) and all connect up to a chatops channel in Slack. I understand it because I wrote it all. Will it scale to infinity? Definitely not, but it's good enough for my uses. So, I guess - just build stuff using simple primitives. Write simple software - modules and functions and a lot of stateless code. Use postgres for persistence - it's really that good. Use nginx and dns load balancing - tried and true simple architecture.
• thelastgallon 1y ago
  Everything is scalable because for an enterprising executive/manager/employee the most important thing they are trying to solve is scaling their empire. Pay is based on how many reports you have and how big is your budget. Instead of 2 x 20K beefy servers, you get to spend tens or hundreds of million on cloud spend, then dozens of $500K/year SRE/DevOps/Cloud 'experts' running multi-cloud hybrid running kubernetes thingies. There will be SO many internal automation tools and big bonuses for developing those tools (wins all around), yet more and more hiring! Once other executives understand that you can scale your team massively, they will hire you to scale even greater heights! More #winning! There is a human element of everything and perverse incentives. Understanding these explains most things that seem baffling at first.
  • cindyllm 1y ago
    [dead]
  • roncesvalles 1y ago
    Exactly. Once you become a people manager, the only metric that matters in your career is "number of people below you".
• maxamillion 1y ago
  I love this blog post so much. <3 Stop solving problems you don't have. You have no users, you don't need to support a million of them. You make no money, you don't need to burn through all your cloud credits so fast. Just chill, start with a couple Linodes, Droplets, or Lightsail instances and see how far you can get with a simple app, api, load balancer, and a database. You'd be shocked how far that stretches when you're talking about servicing real paying customers.
• zem 1y ago
  as an aside to the last bit about monorepos, from a dev point of view monorepos are pretty awesome. you never need to deal with complicated merge commits and you always have a consistent state of the world, enforced by ci. i would imagine maintaining the tooling and infrastructure to support said monorepo is pretty hard and painful, but as a user I love working in one.
• rvitorper 1y ago
  BEAM for the win
• perrygeo 1y ago
  The problem I never see addressed: scaling is not proportional to popularity. There is some sunk cost or "baseload" scale required to run a service. Think google maps purchasing and processing satellite imagery; if you want to compete, you also need to regularly update petabytes of data. That's table stakes, even if you have one user.
• throw-10-13 1y ago
  Vertically scaling hetzner/do machines with some basic networking gets you farther than most startups survive.
• john_the_writer 1y ago
  The only bit that I didn't agree with was the downsides. You can scale individual modules. You can set your load balancer to make all calls to specific endpoints go to specific servers. These can scale independently. If you have background workers (ActiveJobs/Oban for example), these can be on different queues, that you can scale. It's actually really easy to build out a mono-repo system allow for scale. If you organise your workers into folders based on their purpose (reporting, exporting, ... ), and you're careful about feature flags, you can drastically reduce git-conflicts and CICD issues.
• brokegrammer 1y ago
  I agree with this post, but how do we convince decision makers to use simpler solutions? Here's what happened to me not long ago: A person contacted me to build a tool that will have a single user. After some questioning, the person said they would likely have up to 3 users for this tool at some point, and no more. So I suggested we use Python and a SQlite database. When I reached out after not having heard from them for 2 weeks, they said that they went with a more experienced programmer because the latter suggested MongoDB, a separate ReactJS front-end, and micro services with Golang. All deployed to AWS, of course. In the end, I lost the gig because I wanted to save the client time and money.
• cadamsdotcom 1y ago
  Author talks about enforcing internal code boundaries within their monolith. There is a very simple solution. Ask your AI to write you a little python script that checks which modules are imported by which other ones, and validates against a set of rules. Prompts go something like: “we should prevent code in web route handlers from directly importing the database, make it print an error saying only repository classes can use the database directly”. It will write code that visits the python AST or maybe that uses wild regexes. The result is the same: your prompt, codified, turned into a repeatable deterministic tool you can use any time for zero tokens and that runs in under a second. Put the script in pre-commit and husky. Keep adding rules over time. If it gets too slow, ask your AI to optimise it for you. These tools codify the tacit knowledge needed to work in your codebase. Which is a boon for onboarding (new colleagues get blocked by it during development instead of code review.) And because it runs just before everyone commits, you have one less thing to think about when reviewing.
• cadamsdotcom 1y ago
  Tach, mentioned in the article, seems to be in maintained. But I dug around and found this comment: https://github.com/gauge-sh/tach/issues/791#issuecomment-3386224007 https://github.com/gauge-sh/tach/issues/791#issuecomment-338... > since nobody else has done it yet, i've forked tach and updated it to work on python 3.14 - https://github.com/DetachHead/dtach https://github.com/DetachHead/dtach