5 ms·

I am one of the biggest proponents of microservices. I helped build the platform at Netflix, I've literally traveled around the world extolling the virtues of

I am one of the biggest proponents of microservices. I helped build the platform at Netflix, I've literally traveled around the world extolling the virtues of microservices. But I also advise a lot of startups, and you know what I tell them nearly every time? Build a monolith. It's so much easier to start out with one codebase and one database, and that will scale for a while. Especially if you use a key/value store like DynamoDB (although you will lose relational functionality that can be helpful at the start). And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services? And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). The one that may need to scale independently, or that you want to be able to deploy separately. Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.

crooked-v 3y ago
"A while" is underselling it. As long as you have people who are half-decent with SQL, "just put Postgres on a big db server" will get you to 50 million row tables before you have to start thinking about even hiring a real DBA.
- jedberg 3y ago
  True. "While" for some large value of time. And if you set up the auto-vacuumer correctly from the start, you can go even further!
  - turtles3 3y ago
    Unpopular opinion: if you're skimping on ops/DBA resources (as you may need to do in a startup), then MySQL is a better default. By all means use postgres if your use case demands it, but personally I find the ops story for MySQL takes less engineering overhead.
    - lenkite 3y ago
      I agree with this "unpopular opinion". Worked with both MySQL and postgres based mid-scale apps of several thousands of users. Postgres is so deeply lauded here at HN yet requires two more magnitudes of operations work to keep it up and running. Vacuuming sucks hard.
    - avereveard 3y ago
      That's not wrong, but postgres allows for cramming a lot of functionality in the db, and it's fast at following storage trend, i.e. with pgvector
      - turtles3 3y ago
        That is a very specific use case, and might only be a small subset of your actual data. If you don't have these specific requirements (eg. CRUD apps), you can save yourself a lot of unnecessary headaches by defaulting to MySQL. My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.
    - savrajsingh 3y ago
      Yes! Vanilla MySQL and a good ORM (peewee, anyone?) gets you extremely far.
    - PedroBatista 3y ago
      Yes, and most successful companies who started in the last ~20 years started ( and many continue ) with a monolith and a MySQL database. Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one. Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.
- baq 3y ago
  50M is something that can easily be done with just a dev who understands that sql is more than select, insert and update with the manual, google and chatgpt. You can get really damn far with a fat postgres box.
  - hnpxr 3y ago
    It's not even funny how many software engineers just don't know that, SQL is crazy fast and performant, if you have basic understanding about it - I was once refactoring (or, rather, getting rid of) a microservice that was just a JSON blob storage on top of Postgres, without having any schema for blobs, with 100s of 1000s of them, no indices, and main complaint was - it's slow.
    - renegade-otter 3y ago
      Ignorance of database basics is something I allude to in the article, but it's a larger topic that I might vent about later.
  - dalbasal 3y ago
    The problem with these discussions is that you can always get a lot out of any given architecture, structure, db approach or whatnot. Can. There's just a lot of daylight between "can" and "likely will." Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals. There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.
    - baq 3y ago
      You're right of course. My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.
- jiggawatts 3y ago
  It’s so cute that you think 50M rows is big. Your phone can handle many times that, and update it tens of thousands of times per second.
  - dagw 3y ago
    The problem isn't storing or inserting 50M rows, it querying 50M rows in non trivial ways. And the difference in performance between doing that 'right' and 'wrong' is orders of magnitude.
    - omegabravo 3y ago
      doing it on your phone only has one person querying it. Scale that to several thousand and it might appear different.
      - jiggawatts 3y ago
        I've scaled 300M rows in just one of many similarly sized tables to 1M users... in 2007... on a single box with spinning rust in it. Heck, my laptop could handle 100x the production load. It amazes me that my comment (while admittedly flippant) got voted down. It really is true that your phone can update a 50M row table about 10K times per second! That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do. Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.
        sethammons 3y ago
        I believe you are downvoted due to the "how cute" part, not the scale part. You could have delivered useful info without a put down
- n0us 3y ago
  > As long as you have people who are half-decent with SQL :/
elliotec 3y ago
Ah, it’s you! You’ve got a hell of a resume. And been accidentally incredibly convincing in getting many people into many early messes. Jokes aside, thanks for at least coming around to advise startups sanely. I’d personally never advocate for microservices until at the scale of Netflix or Amazon or Reddit, and even then only with in-house expertise at your level. Otherwise it’s a nightmare. Thanks for everything, especially your contributions of sanity.
- jedberg 3y ago
  Aww, thanks for the kind words. I apologize if I caused you any harm with my talks. I did in fact start out saying everyone should use microservices, but I pulled back as I saw how damaging that can be to a small startup, or even a large enterprise that doesn't actually need it. We all make mistakes!
  - mattmanser 3y ago
    If you want a sobering look at the damage you've helped do, go look at job listings on a generic job site. At least here in the UK almost all of them have microservices listed as a requirement. Given that at best like 5% of companies would benefit from microservices, that's a terrible spot our industry is in.
    - hobs 3y ago
      And the more sober of us would say 5% is a huge exaggeration (unless you mean by headcount)
    - deleted 3y ago
      [deleted]
aledalgrande 3y ago
> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing. Do not do microservices when you're growing (or ever, most of the time). And by the love of god, don't split your data. Data is much more complex to manage than code.
- Aeolun 3y ago
  > And by the love of god, don't split your data. I think people feel that if they introduce a new data store, they won’t have to deal with the existing nearly unusable massive data store. Of course that just exacerbates the problem.
- ngc248 3y ago
  There is a way to splitting data. The service which owns the data always does the writes and others who need reads on that data can store replicas. ofc the complication then will be in replicating data, but this will enable services to massively scale and eliminate SPOFs
  - code_biologist 3y ago
    The other part of this pattern you mention (replicating data) seems solved in many cases by the data warehouse patterns of the last few years. Stuff it all in Snowflake or BigQuery then readers can query as they see fit. Query engines like Trino can paper over data storage heterogeneity / breakdowns in centralization. I'm not a fan of the "lakehouse" terminology, but it is the thing. There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.
  - amne 3y ago
    As long as you're not replicating the data model also which means you're now replicating the business logic that go along with the data model. An ETL approach from the start will save you a lot of headache.
- 3y ago
hliyan 3y ago
I wrote this 8 years ago: Microservices vs. "air-gapped" modules https://www.linkedin.com/pulse/maintainable-software-architectures-microservices-vs-air-liyanage/ https://www.linkedin.com/pulse/maintainable-software-archite.... You can achieve the same end as a microservice using a module, by simply having lint rules that prevent you from importing other application level modules. This way the only possible comms interface is to pass events with a payload of primitive typed parameters.
- jedberg 3y ago
  An interesting idea. Sort of a good half way. But one issue I see is that you can still have a shared data store. That means you can accidentally (or intentionally) use the database to pass back-channel messages —- have one module store data and another read it. That could lead to hard to find bugs. Did this ever come up for you?
  - foobarian 3y ago
    I think systems like this are prone to buckling under product pressure. If it takes a few minutes to override a lint rule it's bound to happen sooner or later. With a microservice system the amount of effort is genuinely higher to add cross-dependencies.
  - hliyan 3y ago
    The best "microservice" based system I've ever worked on was full ten years before the term had been invented. It had multiple services that read and wrote to the same database, but each used a common dynamically linked library that was the common interface to access that database (no naked SQL was allowed -- you had to #include <foo> and then do Foo* foo = Foo::findById("xxx"); etc. Worked remarkably well. The above common library was very carefully maintained and any addition had to be reviewed and approved. The approach seemed to work well.
    - jedberg 3y ago
      That's the services model that Facebook uses and parts of Amazon use. A single data store surrounded by a bunch of small services with a prescribed data access layer. That's really more of a monolith though. One of the key tenants of microservices is data separation -- each service has its own data store.
ddalex 3y ago
My go to page about this is http://widgetsandshit.com/teddziuba/2008/04/im-going-to-scale-my-foot-up-y.html http://widgetsandshit.com/teddziuba/2008/04/im-going-to-scal... which perfectly summarizes the problem AND solution
Copenjin 3y ago
I wish you good luck with your redemption arc, every monolith counts.
lifeisstillgood 3y ago
How would you split the total overhead between monolith and monorepo? (sorry this ran away with me) A dumb example is that if I start with my single codebase running on a single server, I am likely to have a single git repo (foo). Then I have a genius idea and put all the email handling code into foo.mail and soon I have foo.web and foo.payments. All is fine as long as I am just checking out HEAD each time. The code running in the runtime is still one big set of code. If I get creative and put in load balancers it's still a monolith. But if I split out the web servers from the email servers, then I kind of start to see microservices appear. I am trying not to be pedantic, but I am truly interested in experienced views on where the pain really starts to appear. At this point (server to server comms), I should look at mTLS, and centralised logging and all the good stuff to manage microservices. But how much pain was there before? What if I was a large company and so hired a dev or two per repo (you know to get that 9 women one month effect). Co-ordinating multiple devs over different repos, even with one monolithic runtime, is painful (experience tells me). So I am interested in where the break points are, and whether there are easier paths up the mountain?
- afiori 3y ago
  Splitting out the webserver (assuming it is the main entrypoint for users) seems more like an infrastructure choice than application architecture and having an independent email-sending-service looks more like replacing a third-party offer (like turboSMTP) with an in-house service. I do not think that this is what people mean by microservices.
  - lifeisstillgood 3y ago
    I get that. I was trying to describe a small path from monolith to microservice without inventing yet another student and teacher database. But the even the step you mention matters. going to a third party service is very similar to reaching out to your own internal service. Make it a "customer" service. Is the pain in microservices (or APIs and mTLS) or is the pain in managing a team that now has to do customer stuff only ? or email stuff only?
    - 3y ago
usrbinbash 3y ago
Wise words. They boil down to a very simple truth that was as accurate half a century ago as it is today: Make things as simple as possible, and as complex as necessary. I can always make something more complex than it is now. As you say, we can take out things from a monolith and make them into a service. It can be hard to do so, sure. But nowhere near as hard, as trying to get complexity OUT of a system once it's in. Everyone who ever tried to revert a bunch of microservices back into a Monolith knows exactly what I am talking about. It usually amounts to the same work as a ground-up rebuild.
Shinchy 3y ago
Absolutely agree, I've seen far too many companies spend far too much time working out the infrastructural relationships of microservices. Spending very little time working on the actual applications needs itself.
johnboy123 3y ago
Just chipping in with thoughts on DynamoDB, (although I have worked on much smaller scale systems) I am a long term dev, done lots of SQL, but for the past few years I have been using DynamoDB, and I am using it for my new startup (So I rate it). Cons - You have to be very aware of your query patterns, and not having ad-hoc queries is a pain. Plus sides - With on demand billing, its free if you aren't using it - Built correctly, it will scale - No Schema upgrades (This one is massive for me) On the last point, I really do appreciate not having to worry about keeping schemas upto date across all devs and environments. We use quite a simple pattern of a table per entity, as opposed to single table design, because it allows us to just use the API at the highest level of abstraction, where you just write your objects to dynamoDB. (You can still do lower level requests to poke values and such like)
- jwestbury 3y ago
  > No Schema upgrades (This one is massive for me) At Amazon, relational databases are banned unless you get an explicit exemption from senior leadership. This is the primary reason why. Too many cases of schema upgrades causing outages. The problem with DDB or other NoSQL applications, like you say, is how much you need to consider your query patterns. The last major project I worked on using DDB, we spent a couple of days just thinking through our query patterns so we could come up with the right database design and data structures. (We still believe it was the right choice, though.)
  - deleted 3y ago
    [deleted]
Gud 3y ago
Wise. Every time I’ve started a project trying to make an amazing platform with everything segregated with micro services etc. I’ve spent so much more time on the _platform_ than on the actual product. Now I go with SQLite and some basic python script and pivot from there. Ironically that’s how I used to do it before the micro service fad.
danielovichdk 3y ago
I honestly believe that doing small services is long term better approach than a monolith. But as you I strongly believe it's not where you want to start. But I do not agree on is more time consuming than the other. It's just time spent on different matters. For the sake of money spent, I have not seen any data on how a monolith outperforms small services. For maintaining the platform, I think 25% sounds a bit high. Money spent should be measured in many different aspects. One is definitely productivity. And I have seen more stale monoliths than I have seen stale small services. Whether it's better or not is not up to me to judge. I just know what I prefer. Having been exposed to small service architecture where it has been working really well and very poor, there are few things that stand out. - Conquer and divide (your monolith over time). - Responsibility boundaries are easier to cope with for developers in a small service since they often don't have generic and yagni abstractions. - Less code to comprehend for a small service and the cognitive load decreases. - Build time, test time, deployment time. - Should lean up against a direct business measurement and value. - Group chatting services into one. I have yet to see a monolith that over time is not really deteriorating, but I haven't worked with SO or Shopify. We could also argue that Cobol and Fortran is still of good use but time has also changed leaving that style of system development exposed as old and dusty. But like any software development occurrence, it takes responsibility, mandate and proper leadership to get things done in a decent manner. So if you start by making 3 services that has to chat to find a user profile you probably don't know what you're doing. And 25 people (the Threads image) is not a small team IMO. Good luck
amelius 3y ago
Of course it is even better to build a distributed monolith...
HerculePoirot 3y ago
The problem is not with the devs but with the investors I believe. They're looking for unicorns, unicorns attract million of customers, so they need to work at scale, hence microservices.
- hobs 3y ago
  Well, in the previous cycle it was clear that the investors were looking for buzzwords and marketing hype because they often don't understand and frankly dont care - they just wanted to make you look good enough to sell to the next guy, as they were all assuming you were unprofitable in and unprofitable out. They all just want a story that you are going to be the next google so they can sell that story to the next guy. Will that hold in the non zero interest rate world? The AI hype doesn't make me think the world has changed that much.
- gremlinunderway 3y ago
  make up a new word for monolith that sounds sexy instead of big, bulky and scary. Something like "uniform-services" or "harmonized-services". Vertically-distributed harmonized services.
glimshe 3y ago
Most people don't understand the point of microservices. They look at the idea and are attracted by the power of modular interfaces, customizable scalability and independent deployment, without really thinking of the additional engineering overhead that are required to unleash these capabilities. I've seen ex-Netflix software engineers taking jobs elsewhere and proposing microservices for systems that would receive little to no benefit from them. In practice, the implementation of microservices in these contexts become a costly solution looking for a problem.
arp242 3y ago
> It's so much easier to start out with one codebase and one database, and that will scale for a while. Lots of businesses don't even need to scale anyway; Netflix is a "high customer, low revenue per customer" type of business, but there's lots of "low customer, high revenue per customer" businesses too, perhaps even more than the first one. These are often the type of products where you could quite literally run production on your laptop if you wanted to. At the last place I worked they built all this microservice bonanza for ... 300 customers... If they ever got a few thousand customers it would be quite successful, and tens of thousands would be hugely successful. What usually happens is that they don't spend that "25% of your engineering time just maintaining the platform", so the platform was impossible to run locally, and production was hanging together with duct tape. (Aside: in a recent interview I was asked to design a system for "500 million concurrent users" – this is for a business that's mostly in the low-customer/high-revenue type. I still don't know if the test was to point out that 500M is an utterly bonkers number – about a 10% of the internet connected people on the planet – or that they really thought this was somehow a vaguely realistic number for a startup. I answered with "build the simplest thing possible, a monolith, and focus on features and making sure you're building something people want". I didn't get hired, so I guess they were somehow under the misapprehension that you need to design things for 500M users right from the start?)
shortlived 3y ago
> And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services I did not know. Does anyone have pointers or examples on this?
- jedberg 3y ago
  As an example, you'd write your entire API as a flask app, and then deploy that app to Lambda. Then send all requests to that one Lambda. As long as your startup time is quick (and your datastore is elsewhere, like in DynamoDB) then it will work great for quite a while. Lambda will basically run your app enough times to handle all the requests. You have to be careful when you design it such that you don't rely on subsequent requests coming to the same machine, but you can also design it so that if they do come to the same machine it will work, using a tiered cache.
rewmie 3y ago
> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). Wasn't this the rule of thumb for microservices from the very start? That the main driving force is to ensure separate teams work independently on stand-alone projects they own exclusively? Scaling tends to only become an issue when the project grows considerably.
- jedberg 3y ago
  Yes. But that idea got lost in the fray. Even I tried to build a system from scratch with microservices at one point.
  - rewmie 3y ago
    > Yes. But that idea got lost in the fray. I don't think so. One of the very first things that people learn when onboarding into microservices is the microservices tax, whose corollary is don't onboard onto microservices if there isn't a clear significant win that justifies all the tradeoffs. Moreover, the main selling point of microservices is organizational. That's also right in the first lesson on microservices. If you're not working in a large team who owns a large project and wants to lower managerial overhead by splitting up the team, you need to try very hard to find a justification to peel off specific responsibilities from a service into an independent microservice. That's microservices 101. Those who jump onto the microservicss bandwagon without having a concrete justification have only themselves to blame, and not any particular architectural style. They are making the piss poor decisions and blaming abstract concepts for their own mistakes.
thinkmassive 3y ago
> break out an independently scalable part of the system into a microservice … The one that may need to scale independently, or that you want to be able to deploy separately. How often would you estimate that security concerns are a valid reason to isolate functionality into a microservice? Separation of concerns, reduced attack surface and blast radius, etc
nijave 3y ago
>Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency. My take is, microservices multiply your operational problems so you need a really solid platform/infrastructure/developer experience team otherwise you'll introduce a bunch of new headaches. These include things like CI/CD (building, storing artifacts, testing artifacts, promoting artifacts between environments), observability (metrics, logs, distributed tracing, error monitoring), distributed systems quirks like cascading failures, release/change management and communication, service discovery/routing, automated infra provisioning for things like datastores. Unless you have a really good handle on all these pieces (which I think most startups don't), you end up in an operational nightmare spending tons of time trying to keep the system going. Of course, the next step is throwing on 3rd party products to try to solve this which adds even more complexity. Throw in Datadog for observability, Istio service mesh for networking/traffic, ArgoCD/kustomize/Helm/Kubernetes to manage all this infra, etc etc
MattRogish 3y ago
I agree - microservices are a technical solution to a people problem. Stevey's seminal "Platforms Rant" (https://gist.github.com/chitchcock/1281611 https://gist.github.com/chitchcock/1281611) touches on this - the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way. Splitting into services siloed each team and allowed them to move independently and by side effect, faster. Not due to inherent properties of (micro) services but because one goes faster by removing the things that slow you down. As a startup, you do not have this problem. You likely will _never_ have this problem as your default future state is 'dead'. Do the simplest thing that can possibly work - build a monolith using tools you know that give you the ability to predictably, rapidly iterate on the product. After you hit some semblance of product/market fit and need to scale, you can do that. Scaling is a solved problem. Premature scaling is not. [1. This is a joke. Kubernetes wasn't even a thought in Google's eye at this point in history]*
- rewmie 3y ago
  > the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way. Not quite. Amazon's problem was that they were experiencing too much impedance between teams, and some teams were even siloing themselves to the extent they were creating problems for everyone around them. Bezos' diktat was intended to break through a bunch of petty office politics bullshit that was dragging down the company and preventing teams from delivering their work. The diktat boiled down to basically ordering everyone to grant access to the service they owned and provided to external teams that need it, no exception, and would be held liable if they failed to provide it, intentionally or not
1letterunixname 3y ago
The issue isn't blindly adopting microservices or a monolith as a religion or bikeshedding thereabouts. Architectural and ops concerns lead to dividing an overall service into sufficient and necessary units of abstraction. To efficiently run more than 1 service leads to standardization of automation of the concerns of application platform infrastructure: - stateful data backup, protection recovery - configuration management - OS security - authentication, authorization, audit, encryption, and identity management - monitoring - analytics - rate limiting - A/B, etc. feature tests - sharded deployment Chopping up 1 service into many more services doesn't make the above concerns go away. Neither does collapsing many into 1. The internal pieces (and often interfaces between business units) need to be broken down into decoupled units of abstraction. If people want to call that "microservices" or a "monolith", it's kind of irrelevant. Containers of abstraction should serve a purpose rather than hinder it.
wutwutwat 3y ago
Obligatory "PostgreSQL when it's not your job" share. It'll get many shops pretty damn far imo https://www.pgcon.org/2016/schedule/attachments/437_not-your-job-pgcon-2016.pdf https://www.pgcon.org/2016/schedule/attachments/437_not-your...
spixy 3y ago
I saved url of your comment into my favourites, if I need to convince people in a future.
dysoco 3y ago
> And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services Could I get more info on this? All I've found so far is that it's an anti-pattern. Isn't the startup time a killer?
- jedberg 3y ago
  > Isn't the startup time a killer? It depends on your programming language and how many libraries you are importing. If you use Python you can get startup times less than 200ms. Also, if your app is active, most startups won't be cold (Lambda reuses the VM a bunch of times before recycling it if it's staying active). You can also add a health check API and just poke that once a minute to keep it warm if you want (at the cost of increased invocations). But yes, if you're doing something that requires a lot of setup, then this pattern isn't for you. But it turns out most web apps are simple enough it can be done this way, especially if you're already using a pattern where the frontend is doing a lot of the heavy lifting and your API is mostly just marshaling data from the client to the database with some security and small logic in between.

crooked-v 3y ago

"A while" is underselling it. As long as you have people who are half-decent with SQL, "just put Postgres on a big db server" will get you to 50 million row tables before you have to start thinking about even hiring a real DBA.

jedberg 3y ago
True. "While" for some large value of time. And if you set up the auto-vacuumer correctly from the start, you can go even further!
- turtles3 3y ago
  Unpopular opinion: if you're skimping on ops/DBA resources (as you may need to do in a startup), then MySQL is a better default. By all means use postgres if your use case demands it, but personally I find the ops story for MySQL takes less engineering overhead.
  - lenkite 3y ago
    I agree with this "unpopular opinion". Worked with both MySQL and postgres based mid-scale apps of several thousands of users. Postgres is so deeply lauded here at HN yet requires two more magnitudes of operations work to keep it up and running. Vacuuming sucks hard.
  - avereveard 3y ago
    That's not wrong, but postgres allows for cramming a lot of functionality in the db, and it's fast at following storage trend, i.e. with pgvector
    - turtles3 3y ago
      That is a very specific use case, and might only be a small subset of your actual data. If you don't have these specific requirements (eg. CRUD apps), you can save yourself a lot of unnecessary headaches by defaulting to MySQL. My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.
  - savrajsingh 3y ago
    Yes! Vanilla MySQL and a good ORM (peewee, anyone?) gets you extremely far.
  - PedroBatista 3y ago
    Yes, and most successful companies who started in the last ~20 years started ( and many continue ) with a monolith and a MySQL database. Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one. Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.
baq 3y ago
50M is something that can easily be done with just a dev who understands that sql is more than select, insert and update with the manual, google and chatgpt. You can get really damn far with a fat postgres box.
- hnpxr 3y ago
  It's not even funny how many software engineers just don't know that, SQL is crazy fast and performant, if you have basic understanding about it - I was once refactoring (or, rather, getting rid of) a microservice that was just a JSON blob storage on top of Postgres, without having any schema for blobs, with 100s of 1000s of them, no indices, and main complaint was - it's slow.
  - renegade-otter 3y ago
    Ignorance of database basics is something I allude to in the article, but it's a larger topic that I might vent about later.
- dalbasal 3y ago
  The problem with these discussions is that you can always get a lot out of any given architecture, structure, db approach or whatnot. Can. There's just a lot of daylight between "can" and "likely will." Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals. There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.
  - baq 3y ago
    You're right of course. My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.
jiggawatts 3y ago
It’s so cute that you think 50M rows is big. Your phone can handle many times that, and update it tens of thousands of times per second.
- dagw 3y ago
  The problem isn't storing or inserting 50M rows, it querying 50M rows in non trivial ways. And the difference in performance between doing that 'right' and 'wrong' is orders of magnitude.
  - omegabravo 3y ago
    doing it on your phone only has one person querying it. Scale that to several thousand and it might appear different.
    - jiggawatts 3y ago
      I've scaled 300M rows in just one of many similarly sized tables to 1M users... in 2007... on a single box with spinning rust in it. Heck, my laptop could handle 100x the production load. It amazes me that my comment (while admittedly flippant) got voted down. It really is true that your phone can update a 50M row table about 10K times per second! That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do. Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.
      - sethammons 3y ago
        I believe you are downvoted due to the "how cute" part, not the scale part. You could have delivered useful info without a put down
n0us 3y ago
> As long as you have people who are half-decent with SQL :/

elliotec 3y ago

Ah, it’s you! You’ve got a hell of a resume. And been accidentally incredibly convincing in getting many people into many early messes. Jokes aside, thanks for at least coming around to advise startups sanely. I’d personally never advocate for microservices until at the scale of Netflix or Amazon or Reddit, and even then only with in-house expertise at your level. Otherwise it’s a nightmare. Thanks for everything, especially your contributions of sanity.

jedberg 3y ago
Aww, thanks for the kind words. I apologize if I caused you any harm with my talks. I did in fact start out saying everyone should use microservices, but I pulled back as I saw how damaging that can be to a small startup, or even a large enterprise that doesn't actually need it. We all make mistakes!
- mattmanser 3y ago
  If you want a sobering look at the damage you've helped do, go look at job listings on a generic job site. At least here in the UK almost all of them have microservices listed as a requirement. Given that at best like 5% of companies would benefit from microservices, that's a terrible spot our industry is in.
  - hobs 3y ago
    And the more sober of us would say 5% is a huge exaggeration (unless you mean by headcount)
  - deleted 3y ago
    [deleted]

aledalgrande 3y ago

> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing. Do not do microservices when you're growing (or ever, most of the time). And by the love of god, don't split your data. Data is much more complex to manage than code.

Aeolun 3y ago
> And by the love of god, don't split your data. I think people feel that if they introduce a new data store, they won’t have to deal with the existing nearly unusable massive data store. Of course that just exacerbates the problem.
ngc248 3y ago
There is a way to splitting data. The service which owns the data always does the writes and others who need reads on that data can store replicas. ofc the complication then will be in replicating data, but this will enable services to massively scale and eliminate SPOFs
- code_biologist 3y ago
  The other part of this pattern you mention (replicating data) seems solved in many cases by the data warehouse patterns of the last few years. Stuff it all in Snowflake or BigQuery then readers can query as they see fit. Query engines like Trino can paper over data storage heterogeneity / breakdowns in centralization. I'm not a fan of the "lakehouse" terminology, but it is the thing. There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.
- amne 3y ago
  As long as you're not replicating the data model also which means you're now replicating the business logic that go along with the data model. An ETL approach from the start will save you a lot of headache.
3y ago

hliyan 3y ago

I wrote this 8 years ago: Microservices vs. "air-gapped" modules https://www.linkedin.com/pulse/maintainable-software-architectures-microservices-vs-air-liyanage/ https://www.linkedin.com/pulse/maintainable-software-archite.... You can achieve the same end as a microservice using a module, by simply having lint rules that prevent you from importing other application level modules. This way the only possible comms interface is to pass events with a payload of primitive typed parameters.

jedberg 3y ago
An interesting idea. Sort of a good half way. But one issue I see is that you can still have a shared data store. That means you can accidentally (or intentionally) use the database to pass back-channel messages —- have one module store data and another read it. That could lead to hard to find bugs. Did this ever come up for you?
- foobarian 3y ago
  I think systems like this are prone to buckling under product pressure. If it takes a few minutes to override a lint rule it's bound to happen sooner or later. With a microservice system the amount of effort is genuinely higher to add cross-dependencies.
- hliyan 3y ago
  The best "microservice" based system I've ever worked on was full ten years before the term had been invented. It had multiple services that read and wrote to the same database, but each used a common dynamically linked library that was the common interface to access that database (no naked SQL was allowed -- you had to #include <foo> and then do Foo* foo = Foo::findById("xxx"); etc. Worked remarkably well. The above common library was very carefully maintained and any addition had to be reviewed and approved. The approach seemed to work well.
  - jedberg 3y ago
    That's the services model that Facebook uses and parts of Amazon use. A single data store surrounded by a bunch of small services with a prescribed data access layer. That's really more of a monolith though. One of the key tenants of microservices is data separation -- each service has its own data store.

ddalex 3y ago

My go to page about this is http://widgetsandshit.com/teddziuba/2008/04/im-going-to-scale-my-foot-up-y.html http://widgetsandshit.com/teddziuba/2008/04/im-going-to-scal... which perfectly summarizes the problem AND solution

Copenjin 3y ago

I wish you good luck with your redemption arc, every monolith counts.

lifeisstillgood 3y ago

How would you split the total overhead between monolith and monorepo? (sorry this ran away with me) A dumb example is that if I start with my single codebase running on a single server, I am likely to have a single git repo (foo). Then I have a genius idea and put all the email handling code into foo.mail and soon I have foo.web and foo.payments. All is fine as long as I am just checking out HEAD each time. The code running in the runtime is still one big set of code. If I get creative and put in load balancers it's still a monolith. But if I split out the web servers from the email servers, then I kind of start to see microservices appear. I am trying not to be pedantic, but I am truly interested in experienced views on where the pain really starts to appear. At this point (server to server comms), I should look at mTLS, and centralised logging and all the good stuff to manage microservices. But how much pain was there before? What if I was a large company and so hired a dev or two per repo (you know to get that 9 women one month effect). Co-ordinating multiple devs over different repos, even with one monolithic runtime, is painful (experience tells me). So I am interested in where the break points are, and whether there are easier paths up the mountain?

afiori 3y ago
Splitting out the webserver (assuming it is the main entrypoint for users) seems more like an infrastructure choice than application architecture and having an independent email-sending-service looks more like replacing a third-party offer (like turboSMTP) with an in-house service. I do not think that this is what people mean by microservices.
- lifeisstillgood 3y ago
  I get that. I was trying to describe a small path from monolith to microservice without inventing yet another student and teacher database. But the even the step you mention matters. going to a third party service is very similar to reaching out to your own internal service. Make it a "customer" service. Is the pain in microservices (or APIs and mTLS) or is the pain in managing a team that now has to do customer stuff only ? or email stuff only?
  - 3y ago

usrbinbash 3y ago

Wise words. They boil down to a very simple truth that was as accurate half a century ago as it is today: Make things as simple as possible, and as complex as necessary. I can always make something more complex than it is now. As you say, we can take out things from a monolith and make them into a service. It can be hard to do so, sure. But nowhere near as hard, as trying to get complexity OUT of a system once it's in. Everyone who ever tried to revert a bunch of microservices back into a Monolith knows exactly what I am talking about. It usually amounts to the same work as a ground-up rebuild.

Shinchy 3y ago

Absolutely agree, I've seen far too many companies spend far too much time working out the infrastructural relationships of microservices. Spending very little time working on the actual applications needs itself.

johnboy123 3y ago

Just chipping in with thoughts on DynamoDB, (although I have worked on much smaller scale systems) I am a long term dev, done lots of SQL, but for the past few years I have been using DynamoDB, and I am using it for my new startup (So I rate it). Cons - You have to be very aware of your query patterns, and not having ad-hoc queries is a pain. Plus sides - With on demand billing, its free if you aren't using it - Built correctly, it will scale - No Schema upgrades (This one is massive for me) On the last point, I really do appreciate not having to worry about keeping schemas upto date across all devs and environments. We use quite a simple pattern of a table per entity, as opposed to single table design, because it allows us to just use the API at the highest level of abstraction, where you just write your objects to dynamoDB. (You can still do lower level requests to poke values and such like)

jwestbury 3y ago
> No Schema upgrades (This one is massive for me) At Amazon, relational databases are banned unless you get an explicit exemption from senior leadership. This is the primary reason why. Too many cases of schema upgrades causing outages. The problem with DDB or other NoSQL applications, like you say, is how much you need to consider your query patterns. The last major project I worked on using DDB, we spent a couple of days just thinking through our query patterns so we could come up with the right database design and data structures. (We still believe it was the right choice, though.)
- deleted 3y ago
  [deleted]

Gud 3y ago

Wise. Every time I’ve started a project trying to make an amazing platform with everything segregated with micro services etc. I’ve spent so much more time on the _platform_ than on the actual product. Now I go with SQLite and some basic python script and pivot from there. Ironically that’s how I used to do it before the micro service fad.

danielovichdk 3y ago

I honestly believe that doing small services is long term better approach than a monolith. But as you I strongly believe it's not where you want to start. But I do not agree on is more time consuming than the other. It's just time spent on different matters. For the sake of money spent, I have not seen any data on how a monolith outperforms small services. For maintaining the platform, I think 25% sounds a bit high. Money spent should be measured in many different aspects. One is definitely productivity. And I have seen more stale monoliths than I have seen stale small services. Whether it's better or not is not up to me to judge. I just know what I prefer. Having been exposed to small service architecture where it has been working really well and very poor, there are few things that stand out. - Conquer and divide (your monolith over time). - Responsibility boundaries are easier to cope with for developers in a small service since they often don't have generic and yagni abstractions. - Less code to comprehend for a small service and the cognitive load decreases. - Build time, test time, deployment time. - Should lean up against a direct business measurement and value. - Group chatting services into one. I have yet to see a monolith that over time is not really deteriorating, but I haven't worked with SO or Shopify. We could also argue that Cobol and Fortran is still of good use but time has also changed leaving that style of system development exposed as old and dusty. But like any software development occurrence, it takes responsibility, mandate and proper leadership to get things done in a decent manner. So if you start by making 3 services that has to chat to find a user profile you probably don't know what you're doing. And 25 people (the Threads image) is not a small team IMO. Good luck

amelius 3y ago

Of course it is even better to build a distributed monolith...

HerculePoirot 3y ago

The problem is not with the devs but with the investors I believe. They're looking for unicorns, unicorns attract million of customers, so they need to work at scale, hence microservices.

hobs 3y ago
Well, in the previous cycle it was clear that the investors were looking for buzzwords and marketing hype because they often don't understand and frankly dont care - they just wanted to make you look good enough to sell to the next guy, as they were all assuming you were unprofitable in and unprofitable out. They all just want a story that you are going to be the next google so they can sell that story to the next guy. Will that hold in the non zero interest rate world? The AI hype doesn't make me think the world has changed that much.
gremlinunderway 3y ago
make up a new word for monolith that sounds sexy instead of big, bulky and scary. Something like "uniform-services" or "harmonized-services". Vertically-distributed harmonized services.

glimshe 3y ago

Most people don't understand the point of microservices. They look at the idea and are attracted by the power of modular interfaces, customizable scalability and independent deployment, without really thinking of the additional engineering overhead that are required to unleash these capabilities. I've seen ex-Netflix software engineers taking jobs elsewhere and proposing microservices for systems that would receive little to no benefit from them. In practice, the implementation of microservices in these contexts become a costly solution looking for a problem.

arp242 3y ago

> It's so much easier to start out with one codebase and one database, and that will scale for a while. Lots of businesses don't even need to scale anyway; Netflix is a "high customer, low revenue per customer" type of business, but there's lots of "low customer, high revenue per customer" businesses too, perhaps even more than the first one. These are often the type of products where you could quite literally run production on your laptop if you wanted to. At the last place I worked they built all this microservice bonanza for ... 300 customers... If they ever got a few thousand customers it would be quite successful, and tens of thousands would be hugely successful. What usually happens is that they don't spend that "25% of your engineering time just maintaining the platform", so the platform was impossible to run locally, and production was hanging together with duct tape. (Aside: in a recent interview I was asked to design a system for "500 million concurrent users" – this is for a business that's mostly in the low-customer/high-revenue type. I still don't know if the test was to point out that 500M is an utterly bonkers number – about a 10% of the internet connected people on the planet – or that they really thought this was somehow a vaguely realistic number for a startup. I answered with "build the simplest thing possible, a monolith, and focus on features and making sure you're building something people want". I didn't get hired, so I guess they were somehow under the misapprehension that you need to design things for 500M users right from the start?)

shortlived 3y ago

> And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services I did not know. Does anyone have pointers or examples on this?

jedberg 3y ago
As an example, you'd write your entire API as a flask app, and then deploy that app to Lambda. Then send all requests to that one Lambda. As long as your startup time is quick (and your datastore is elsewhere, like in DynamoDB) then it will work great for quite a while. Lambda will basically run your app enough times to handle all the requests. You have to be careful when you design it such that you don't rely on subsequent requests coming to the same machine, but you can also design it so that if they do come to the same machine it will work, using a tiered cache.

rewmie 3y ago

> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). Wasn't this the rule of thumb for microservices from the very start? That the main driving force is to ensure separate teams work independently on stand-alone projects they own exclusively? Scaling tends to only become an issue when the project grows considerably.

jedberg 3y ago
Yes. But that idea got lost in the fray. Even I tried to build a system from scratch with microservices at one point.
- rewmie 3y ago
  > Yes. But that idea got lost in the fray. I don't think so. One of the very first things that people learn when onboarding into microservices is the microservices tax, whose corollary is don't onboard onto microservices if there isn't a clear significant win that justifies all the tradeoffs. Moreover, the main selling point of microservices is organizational. That's also right in the first lesson on microservices. If you're not working in a large team who owns a large project and wants to lower managerial overhead by splitting up the team, you need to try very hard to find a justification to peel off specific responsibilities from a service into an independent microservice. That's microservices 101. Those who jump onto the microservicss bandwagon without having a concrete justification have only themselves to blame, and not any particular architectural style. They are making the piss poor decisions and blaming abstract concepts for their own mistakes.

thinkmassive 3y ago

> break out an independently scalable part of the system into a microservice … The one that may need to scale independently, or that you want to be able to deploy separately. How often would you estimate that security concerns are a valid reason to isolate functionality into a microservice? Separation of concerns, reduced attack surface and blast radius, etc

nijave 3y ago

>Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency. My take is, microservices multiply your operational problems so you need a really solid platform/infrastructure/developer experience team otherwise you'll introduce a bunch of new headaches. These include things like CI/CD (building, storing artifacts, testing artifacts, promoting artifacts between environments), observability (metrics, logs, distributed tracing, error monitoring), distributed systems quirks like cascading failures, release/change management and communication, service discovery/routing, automated infra provisioning for things like datastores. Unless you have a really good handle on all these pieces (which I think most startups don't), you end up in an operational nightmare spending tons of time trying to keep the system going. Of course, the next step is throwing on 3rd party products to try to solve this which adds even more complexity. Throw in Datadog for observability, Istio service mesh for networking/traffic, ArgoCD/kustomize/Helm/Kubernetes to manage all this infra, etc etc

MattRogish 3y ago

I agree - microservices are a technical solution to a people problem. Stevey's seminal "Platforms Rant" (https://gist.github.com/chitchcock/1281611 https://gist.github.com/chitchcock/1281611) touches on this - the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way. Splitting into services siloed each team and allowed them to move independently and by side effect, faster. Not due to inherent properties of (micro) services but because one goes faster by removing the things that slow you down. As a startup, you do not have this problem. You likely will _never_ have this problem as your default future state is 'dead'. Do the simplest thing that can possibly work - build a monolith using tools you know that give you the ability to predictably, rapidly iterate on the product. After you hit some semblance of product/market fit and need to scale, you can do that. Scaling is a solved problem. Premature scaling is not. [1. This is a joke. Kubernetes wasn't even a thought in Google's eye at this point in history]*

rewmie 3y ago
> the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way. Not quite. Amazon's problem was that they were experiencing too much impedance between teams, and some teams were even siloing themselves to the extent they were creating problems for everyone around them. Bezos' diktat was intended to break through a bunch of petty office politics bullshit that was dragging down the company and preventing teams from delivering their work. The diktat boiled down to basically ordering everyone to grant access to the service they owned and provided to external teams that need it, no exception, and would be held liable if they failed to provide it, intentionally or not

1letterunixname 3y ago

The issue isn't blindly adopting microservices or a monolith as a religion or bikeshedding thereabouts. Architectural and ops concerns lead to dividing an overall service into sufficient and necessary units of abstraction. To efficiently run more than 1 service leads to standardization of automation of the concerns of application platform infrastructure: - stateful data backup, protection recovery - configuration management - OS security - authentication, authorization, audit, encryption, and identity management - monitoring - analytics - rate limiting - A/B, etc. feature tests - sharded deployment Chopping up 1 service into many more services doesn't make the above concerns go away. Neither does collapsing many into 1. The internal pieces (and often interfaces between business units) need to be broken down into decoupled units of abstraction. If people want to call that "microservices" or a "monolith", it's kind of irrelevant. Containers of abstraction should serve a purpose rather than hinder it.

wutwutwat 3y ago

Obligatory "PostgreSQL when it's not your job" share. It'll get many shops pretty damn far imo https://www.pgcon.org/2016/schedule/attachments/437_not-your-job-pgcon-2016.pdf https://www.pgcon.org/2016/schedule/attachments/437_not-your...

spixy 3y ago

I saved url of your comment into my favourites, if I need to convince people in a future.

dysoco 3y ago

> And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services Could I get more info on this? All I've found so far is that it's an anti-pattern. Isn't the startup time a killer?

jedberg 3y ago
> Isn't the startup time a killer? It depends on your programming language and how many libraries you are importing. If you use Python you can get startup times less than 200ms. Also, if your app is active, most startups won't be cold (Lambda reuses the VM a bunch of times before recycling it if it's staying active). You can also add a health check API and just poke that once a minute to keep it warm if you want (at the cost of increased invocations). But yes, if you're doing something that requires a lot of setup, then this pattern isn't for you. But it turns out most web apps are simple enough it can be done this way, especially if you're already using a pattern where the frontend is doing a lot of the heavy lifting and your API is mostly just marshaling data from the client to the database with some security and small logic in between.