20 ms·

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic AP

by cfors 4y ago

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic API server, but your database. Use One Big Database. Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). It is so much easier to do these joins efficiently in a single database than fanning out RPC calls to multiple different databases, not to mention dealing with inconsistencies, lack of atomicity, etc. etc. Spin up a specific reader of that database if there needs to be OLAP queries, or use a message bus. But keep your OLTP data within one database for as long as possible. You can break apart a stateless microservice, but there are few things as stagnant in the world of software than data. It will keep you nimble for new product features. The boxes that they offer on cloud vendors today for managed databases are giant!

hoosieree 4y ago
This is the macro version of the von Neumann bottleneck.
Closi 4y ago
Breaking apart a stateless microservice and then basing it around a giant single monolithic database is pretty pointless - at that stage you might as well just build a monolith and get on with it as every microservice is tightly coupled to the db.
- cfors 4y ago
  No disagreement here. I love a good monolith.
- roflyear 4y ago
  Absolutely. I know someone who considers "different domains" (as in web domains) to count as a microservice! What is the point of that? it doesn't add anything. Just more shit to remember and get right (and get wrong!)
- radu_floricica 4y ago
  To note that quite a bit of the performance problems come when writing stuff. You can get away with A LOT if you accept 1. the current service doesn't do (much) writing and 2. it can live with slightly old data. Which I think covers 90% of use cases. So you can end up with those services living on separate machines and connecting to read only db replicas, for virtually limitless scalability. And when it realizes it needs to do an update, it either switches the db connection to a master, or it forwards the whole request to another instance connected to a master db.
- adrianmsmith 4y ago
  That's true, unless you need (1) Different programming languages e.g. you're written your app in Java but now you need to do something for which the perfect Python library is available. (2) Different parts of your software need different types of hardware. Maybe one part needs a huge amount of RAM for a cache, but other parts are just a web server. It'd be a shame to have to buy huge amounts of RAM for every server. Splitting the software up and deploying the different parts on different machines can be a win here. I reckon the average startup doesn't need any of that, not suggesting that monoliths aren't the way to go 90% of the time. But if you do need these things, you can still go the microservices route, but it still makes sense to stick to a single database if at all possible, for consistency and easier JOINs for ad-hoc queries, etc.
  - Closi 4y ago
    These are both true - but neither requires service-oriented-architecture. You can split up your applicaiton into chunks that are deployed on seperate hardware, and use different languages, without composing your whole architecture into microservices. A monolith can still have a seperate database server and a web server, or even many different functions split across different servers which are horizontally scalable, and be written in both java and python. Monoliths have had seperate database servers since the 80s (and probably before that!). In fact, part of these applications defining characteristics at the enterprise level is that they often shared one big central database, as often they were composed of lots of small applications that would all make changes to the central database, which would often end up in a right mess of software that was incredibly hard to de-pick! (And all the software writing to that database would, as you described, be written in lots of different languages). People would then come along and cake these central databases full of stored procedures to make magic changes to implement functionality that wasn't available in the legacy applications that they can't change because of the risk and then you have even more of a mess!
- noduerme 4y ago
  I disagree. Suppose you have an enormous DB that's mainly written to by workers inside a company, but has to be widely read by the public outside. You want your internal services on machines with extra layers of security, perhaps only accessible by VPN. Your external facing microservices have other things like e.g. user authentication (which may be tied to a different monolithic database), and you want to put them closer to users, spread out in various data centers or on the edge. Even if they're all bound to one database, there's a lot to recommend keeping them on separate, light cheap servers that are built for http traffic and occasional DB reads. And even more so if those services do a lot of processing on the data that's accessed, such as building up reports, etc.
  - Closi 4y ago
    You've not really built microservices then in the purest sense though - i.e. all the microservices aren't independently deployable components. I'm not saying what you are proposing isn't a perfectly valid architectural approach - it's just usually considered an anti-pattern with microservices (because if all the services depend on a single monolith, and a change to a microservice functionality also mandates a change to the shared monolith which then can impact/break the other services, we have lost the 'independence' benefit that microservices supposedly gives us where changes to one microservice does not impact another). Monoliths can still have layers to support business logic that are seperate to the database anyway.
- manigandham 4y ago
  Why would you break apart a microservice? Any why do you need to use/split into microservices anyway? 99% of apps are best fit as monolithic apps and databases and should focus on business value rather than scale they'll never see.
  - Closi 4y ago
    Totally agree. I guess I just don't see the value in having a monolith made up of microservices - you might as well just build a monolith if you are going down that route. And if your application fits the microservices pattern better, then you might as well go down the microservices pattern properly and not give them a big central DB.
    - adgjlsfhk1 4y ago
      The one advantage of microservice on a single database model is that it lets you test the independent components much more easily while avoiding the complexity of database sharding.
  - Gigachad 4y ago
    Where I work we are looking at it because we are starting to exceed the capabilities of one big database. Several tables are reaching the billions of rows mark and just plain inserts are starting to become too much.
    - nicoburns 4y ago
      Yeah, the at the billions of rows mark it definitely makes sense to start looking at splitting things up. On the other hand, the company I worked for split things up from the start, and when I joined - 4 years down the line - their biggest table had something like 50k rows, but their query performance was awful (tens of seconds in cases) because the data was so spread out.
      - collyw 4y ago
        Am I missing something? 2 ^ 30 is over 1 billion. So a properly indexed table with 1 billion rows will take a similar effort to search as a 30 row unindexed table. Or are there other factors coming into play that I haven't thought of.
- AtNightWeCode 4y ago
  Agree. Nothing worse than having different programs changing data in the same database. The database should not be an integration point between services.
  - jethro_tell 4y ago
    if you have multiple micro services updating the database you need to have a database access layer service as well. there's some real value with abstraction and microservices but you can try to run them against a monolithic database service
    - bergkvist 4y ago
      No amount of abstraction is going to save you from the problem of 2 processes manipulating the same state machine.
      - pc86 4y ago
        In this example, it's the job of the "database access layer service" to manage those processes and prevent issues. But, terrible service name aside, this is a big reason why two services accessing the same database is a capital-H Huge anti-pattern, and really screams "using this project to learn how to do microservices."
  - deleted 4y ago
    [deleted]
- deleted 4y ago
  [deleted]
belak 4y ago
This is absolutely true - when I was at Bitbucket (ages ago at this point) and we were having issues with our DB server (mostly due to scaling), almost everyone we talked to said "buy a bigger box until you can't any more" because of how complex (and indirectly expensive) the alternatives are - sharding and microservices both have a ton more failure points than a single large box. I'm sure they eventually moved off that single primary box, but for many years Bitbucket was run off 1 primary in each datacenter (with a failover), and a few read-only copies. If you're getting to the point where one database isn't enough, you're either doing something pretty weird, are working on a specific problem which needs a more complicated setup, or have grown to the point where investing in a microservice architecture starts to make sense.
- altdataseller 4y ago
  What if your product simply stores a lot of data (ie a search engine) How is that weird?
  - rmbyrro 4y ago
    a search engine won't need joins, but other things (ie text indexing) that can be split in a relatively easier way.
  - zasdffaa 4y ago
    Depends what you mean by a database I guess. I take it to mean an RDBMS. RDBMSs provide guarantees that web searching doesn't need. You can afford to lose a pieces of data, provide not-quite-perfect results for web stuff. It's just wrong for an RDBMS.
    - altdataseller 4y ago
      What if you are using the database as a system of record to index into a real search engine like Elasticsearch? For a product where you have tons of data to search from (ie text from web pages)
      - 4y ago
ryanisnan 4y ago
Definitely use a big database, until you can't. My advice to anyone starting with a relational data store is to use a proxy from day 1 (or some point before adding something like that becomes scary). When you need to start sharding your database, having a proxy is like having a super power.
- _ben_ 4y ago
  Disclaimer: I am the founder of PolyScale [1]. We see both use cases: single large database vs multiple small, decoupled. I agree with the sentiment that a large database offer simplicity, until access patterns change. We focus on distributing database data to the edge using caching. Typically this eliminates read-replicas and a lot of the headache that goes with app logic rewrites or scaling "One Big Database". [1] https://www.polyscale.ai/ https://www.polyscale.ai/
- chromatin 4y ago
  Are there postgres proxies that can specifically facilitate sharding / partitioning later?
cdkmoose 4y ago
>>(they don't know how your distributed databases look, and oftentimes they really do not care) Nor should they, it's the engineer's/team's job to provide the database layer to them with high levels of service without them having to know the details
markandrewj 4y ago
Just FYI, you can have one big database, without running it on one big server. As an example, databases like Cassandra are designed to be scaled horizontally (i.e. scale out, instead of scale up). https://cassandra.apache.org/_/cassandra-basics.html https://cassandra.apache.org/_/cassandra-basics.html
- mdasen 4y ago
  There are trade-offs when you scale horizontally even if a database is designed for it. For example, DataStax's Storage Attached Indexes or Cassandra's hidden-table secondary indexing allow for indexing on columns that aren't part of the clustering/partitioning, but when you're reading you're going to have to ask all the nodes to look for something if you aren't including a clustering/partitioning criteria to narrow it down. You've now scaled out, but you now have to ask each node when searching by secondary index. If you're asking every node for your queries, you haven't really scaled horizontally. You've just increased complexity. Now, maybe 95% of your queries can be handled with a clustering key and you just need secondary indexes to handle 5% of your stuff. In that case, Cassandra does offer an easy way to handle that last 5%. However, it can be problematic if people take shortcuts too much and you end up putting too much load on the cluster. You're also putting your latency for reads at the highest latency of all the machines in your cluster. For example, if you have 100 machines in your cluster with a mean response time of 2ms and a 99th percentile response time of 150ms, you're potentially going to be providing a bad experience to users waiting on that last box on secondary index queries. This isn't to say that Cassandra isn't useful - Cassandra has been making some good decisions to balance the problems engineers face. However, it does come with trade-offs when you distribute the data. When you have a well-defined problem, it's a lot easier to design your data for efficient querying and partitioning. When you're trying to figure things out, the flexibility of a single machine and much cheaper secondary index queries can be important - and if you hit a massive scale, you figure out how you want to partition it then.
  - markandrewj 4y ago
    Cassandra was just an example, but most databases can be scaled either vertically or horizontally via sharding. You are right if misconfigured performance can be hindered, but this is also true for a database which is being scaled vertically. Generally speaking you will get better performance if you have a large dataset by growing horizontally then you would by growing vertically. https://stackoverflow.blog/2022/03/14/how-sharding-a-database-can-make-it-faster/ https://stackoverflow.blog/2022/03/14/how-sharding-a-databas...
johnbellone 4y ago
I agree with this sentiment but it is often misunderstood as a means to force everything into a single database schema. More people need to learn about logically separating schemas with their database servers!
Guid_NewGuid 4y ago
I think a strong test a lot of "let's use Google scale architecture for our MVP" advocates fail is: can your architecture support a performant paginated list with dynamic sort, filter and search where eventual consistency isn't acceptable? Pretty much every CRUD app needs this at some point and if every join needs a network call your app is going to suck to use and suck to develop.
- mhoad 4y ago
  I’ve found the following resource invaluable for designing and creating “cloud native” APIs where I can tackle that kind of thing from the very start without a huge amount of hassle https://google.aip.dev/general https://google.aip.dev/general The patterns section covers all of this and more
  - gnat 4y ago
    This is a great resource but the RFC-style documentation says what you SHOULD and MUST do, not HOW to do it ...
- skyde 4y ago
  thanks a lot for this comment. I will borrow this as an interview question :)
- SkyPuncher 4y ago
  > Pretty much every CRUD app needs this at some point and if every join needs a network call your app is going to suck to use and suck to develop. _at some point_ is the key word here. Most startups (and businesses) can likely get away with this well into Series A or Series B territory.
- threeseed 4y ago
  > if every join needs a network call your app is going to suck to use and suck to develop. And yet developers do this every single day without any issue. It is bad practice to have your authentication database be the same as your app database. Or you have data coming from SaaS products, third party APIs or a cloud service. Or even simply another service in your stack. And with complex schemas often it's far easier to do that join in your application layer. All of these require a network call and join.
ladyattis 4y ago
At my current job we have four different databases so I concur with this assessment. I think it's okay to have some data in different DBs if they're significantly different like say the user login data could be in its own database. But anything that we do which is a combination of e-commerce and testing/certification I think they should be in one big database so I can do reasonable queries for information that we need. This doesn't include two other databases we have on-prem which one is a Salesforce setup and another is an internal application system that essentially marries Salesforce to that. It's a weird wild environment to navigate when adding features.
robertlagrant 4y ago
> Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). I'm not sure how to parse this. What should "asks" be?
- cfors 4y ago
  The feature requests (asks) that product wants to build - sorry for the confusion there.
- delecti 4y ago
  The phrase "Your product asks will consistently " can be de-abbreviated to "product owners/product managers you work with will consistently request".
rbanffy 4y ago
> Use One Big Database. I emphatically disagree. I've seen this evolve into tightly coupled microservices that could be deployed independently in theory, but required exquisite coordination to work. If you want them to be on a single server, that's fine, but having multiple databases or schemas will help enforce separation. And, if you need one single place for analytics, push changes to that space asynchronously. Having said that, I've seen silly optimizations being employed that make sense when you are Twitter, and to nobody else. Slice services up to the point they still do something meaningful in terms of the solution and avoid going any further.
- danpalmer 4y ago
  To clarify the advice, at least how I believe it should be done… Use One Big Database Server… … and on it, use one software database per application. For example, one Postgres server can host many databases that are mostly* independent from each other. Each application or service should have its own database and be unaware of the others, communicating with them via the services if necessary. This makes splitting up into multiple database servers fairly straightforward if needed later. In reality most businesses will have a long tail of tiny databases that can all be on the same server, with only bigger databases needing dedicated resources. *you can have interdependencies when you’re using deep features sometimes, but in an application-first development model I’d advise against this.
  - goodoldneon 4y ago
    OP mentioned joining, so they were definitely talking about a single database
    - danpalmer 4y ago
      You can still do a ton of joining. I’d start with a monolith, that’s a single app, single database, single point of ownership of the data model, and a ton of joins. Then as services are added after the monolith they can still use the main database for ease of infra development, simpler backups and replication, etc. but those wouldn’t be able to be joined because they’re cross-service.
bartread 4y ago
Not to mention, backups, restores, and disaster recovery are so much easier with One Big Database™.
- 1500100900 4y ago
  How is backup restoration any easier if your whole PostgreSQL cluster goes back in time when you only wanted to rewind that one tenant?
  - fleddr 4y ago
    Your scenario is data recovery, not backup restoration. Wildly different things.
clairity 4y ago
> "Use One Big Database." yah, this is something i learned when designing my first server stack (using sun machines) for a real business back during the dot-com boom/bust era. our single database server was the beefiest machine by far in the stack, 5U in the rack (we also had a hot backup), while the other servers were 1U or 2U in size. most of that girth was for memory and disk space, with decent but not the fastest processors. one big db server with a hot backup was our best tradeoff for price, performance, and reliability. part of the mitigation was that the other servers could be scaled horizontally to compensate for a decent amount of growth without needing to scale the db horizontally.
marcosdumay 4y ago
> Use One Big Database Yep, with a passive replica or online (log) backup. Keeping things centralized can reduce your hardware requirement by multiple orders of magnitude. The one huge exception is a traditional web service, those scale very well, so you may not even want to get big servers for them (until you need them).
s_dev 4y ago
>Use One Big Database. It may be reasonable to have two databases e.g. a class a and class b for pci compliance. So context still deeply matters. Also having a dev DB with mock data and a live DB with real data is a common setup in many companies.
jasonwatkinspdx 4y ago
A relative worked for a hedge fund that used this idea. They were a C#/MSSQL shop, so they just bought whatever was the biggest MSSQL server at the time, updating frequently. They said it was a huge advantage, where the limit in scale was more than offset by productivity. I think it's an underrated idea. There's a lot of people out there building a lot of complexity for datasets that in the end are less than 100 TB. But it also has limits. Infamously Twitter delayed going to a sharded architecture a bit too long, making it more of an ugly migration.
- manigandham 4y ago
  Server hardware is so cheap and fast today that 99% of companies will never hit that limit in scale either.
throwaway894345 4y ago
I'm pretty happy to pay a cloud provider to deal with managing databases and hosts. It doesn't seem to cause me much grief, and maybe I could do it better but my time is worth more than our RDS bill. I can always come back and Do It Myself if I run out of more valuable things to work on. Similarly, paying for EKS or GKE or the higher-level container offerings seems like a much better place to spend my resources than figuring out how to run infrastructure on bare VMs. Every time I've seen a normal-sized firm running on VMs, they have one team who is responsible for managing the VMs, and either that team is expecting a Docker image artifact or they're expecting to manage the environment in which the application runs (making sure all of the application dependencies are installed in the environment, etc) which typically implies a lot of coordination between the ops team and the application teams (especially regarding deployment). I've never seen that work as smoothly as deploying to ECS/EKS/whatever and letting the ops team work on automating things at a higher level of abstraction (automatic certificate rotation, automatic DNS, etc). That said, I've never tried the "one big server" approach, although I wouldn't want to run fewer than 3 replicas, and I would want reproducibility so I know I can stand up the exact same thing if one of the replicas go down as well as for higher-fidelity testing in lower environments. And since we have that kind of reproducibility, there's no significant difference in operational work between running fewer larger servers and more smaller servers.
AtNightWeCode 4y ago
If you get your services right there is little or no communications between the services since a microservice should have all the data it needs in it's own store.
wefarrell 4y ago
"Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care)." This isn't a problem if state is properly divided along the proper business domain and the people who need to access the data have access to it. In fact many use cases require it - publicly traded companies can't let anyone in the organization access financial info and healthcare companies can't let anyone access patient data. And of course are performance concerns as well if anyone in the organization can arbitrarily execute queries on any of the organization's data. I would say YAGNI applies to data segregation as well and separations shouldn't be introduced until they are necessary.
- Mavvie 4y ago
  "combine these data sources" doesn't necessarily mean data analytics. Just as an example, it could be something like "show a badge if it's the user's birthday", which if you had a separate microservice for birthdays would be much harder than joining a new table.
  - wefarrell 4y ago
    Replace "people" with "features" and my comment still holds. As software, features, and organizations become more complex the core feature data becomes a smaller and smaller proportion of the overall state and that's when microservices and separate data stores become necessary.
BenoitEssiambre 4y ago
I'm glad this is becoming conventional wisdom. I used to argue this in these pages a few years ago and would get downvoted below the posts telling people to split everything into microservices separated by queues (although I suppose it's making me lose my competitive advantage when everyone else is building lean and mean infrastructure too). In my mind, reasons involve keeping transactional integrity, ACID compliance, better error propagation, avoiding the hundreds of impossible to solve roadblocks of distributed systems (https://groups.csail.mit.edu/tds/papers/Lynch/MIT-LCS-TM-394.pdf https://groups.csail.mit.edu/tds/papers/Lynch/MIT-LCS-TM-394...). But also it is about pushing the limits of what is physically possible in computing. As Admiral Grace Hopper would point out (https://www.youtube.com/watch?v=9eyFDBPk4Yw https://www.youtube.com/watch?v=9eyFDBPk4Yw ) doing distance over network wires involves hard latency constraints, not to mention dealing with congestions over these wires. Physical efficiency is about keeping data close to where it's processed. Monoliths can make much better use of L1, L2, L3, and ram caches than distributed systems for speedups often in the order of 100X to 1000X. Sure it's easier to throw more hardware at the problem with distributed systems but the downsides are significant so be sure you really need it. Now there is a corollary to using monoliths. Since you only have one db, that db should be treated as somewhat sacred, you want to avoid wasting resources inside it. This means being a bit more careful about how you are storing things, using the smallest data structures, normalizing when you can etc. This is not to save disk, disk is cheap. This is to make efficient use of L1,L2,L3 and ram. I've seen boolean true or false values saved as large JSON documents. {"usersetting1": true, "usersetting2":fasle "setting1name":"name" etc.} with 10 bits of data ending up as a 1k JSON document. Avoid this! Storing documents means, the keys, the full table schema is in every row. It has its uses but if you can predefine your schema and use the smallest types needed, you are gaining much performance mostly through much higher cache efficiency!
- tsmarsh 4y ago
  'over the wire' is less obvious than it used to be. If you're in k8s pod, those calls are really kernel calls. Sure you're serializing and process switching where you could be just making a method call, but we had to do something. I'm seeing less 'balls of mud' with microservices. Thats not zero balls of mud. But its not a given for almost every code base I wander into.
beachy 4y ago
Another area for consolidation is auth. Use one giant keycloak, with individual realms for every one of the individual apps you are running. Your keycloak is back ended by your one giant database.
z3t4 4y ago
The rule is: Keep related data together. Exceptions are: Different customers (usually don't require each others data) can be isolated. And if the database become the bottleneck you can separate unrelated services.
throwaway9870 4y ago
How do you use one big database when some of your info is stuck in an ERP system?
FpUser 4y ago
>"Use One Big Database." I do, it is running on the same big (relatively) server as my native C++ backend talking to the database. The performance smokes your standard cloudy setup big time. Serving thousand requests per second on 16 core without breaking sweat. I am all for monoliths running on real no cloudy hardware. As long as the business scale is reasonable and does not approach FAANG (like for 90% of the businesses) this solution is superior to everything else money, maintenance, development time wise.
HeavyStorm 4y ago
> they don't know how your distributed databases look, and oftentimes they really do not care Nor should they.
doctor_eval 4y ago
I agree that 1BDB is a good idea, but having one ginormous schema has its own costs. So I still think data should be logically partitioned between applications/microservices - in PG terms, one “cluster” but multiple “databases”. We solved the problem of collecting data from the various databases for end users by having a GraphQL layer which could integrate all the data sources. This turned out to be absolutely awesome. You could also do something similar using FDW. The effort was not significant relative to the size of the application. The benefits of this architecture were manifold but one of the main ones is that it reduces the complexity of each individual database, which dramatically improved performance, and we knew that if we needed more performance we could pull those individual databases out into their own machine.
lmm 4y ago
If you do this then you'll have the hardest possible migration when the time comes to split it up. It will take you literally years, perhaps even a decade. Shard your datastore from day 1, get your dataflow right so that you don't need atomicity, and it'll be painless and scale effortlessly. More importantly, you won't be able to paper over crappy dataflow. It's like using proper types in your code: yes, it takes a bit more effort up-front compared to just YOLOing everything, but it pays dividends pretty quickly.
- riku_iki 4y ago
  > Shard your datastore from day 1 what about using something like cocroach from day 1?
  - lmm 4y ago
    I don't know the characteristics of bikesheddb's upstream in detail (if there's ever a production-quality release of bikesheddb I'll take another look), but in general using something that can scale horizontally (like Cassandra or Riak, or even - for all its downsides - MongoDB) is a great approach - I guess it's a question of terminology whether you call that "sharding" or not. Personally I prefer that kind of datastore over an SQL database.
    - riku_iki 4y ago
      > over an SQL database it is actually distributed SQL Db with auto sharding, their goal is to be SQL compatible with Postgres.
- Rantenki 4y ago
  This is true IFF you get to the point where you have to split up. I know we're all hot and bothered about getting our apps to scale up to be the next unicorn, but most apps never need to scale past the limit of a single very high-performance database. For most people, this single huge DB is sufficient. Also, for many (maybe even most) applications, designated outages for maintenance are not only acceptable, but industry standard. Banks have had, and continue to have designated outages all the time, usually on weekends when the impact is reduced. Sure, what I just wrote is bad advice for mega-scale SaaS offerings with millions of concurrent users, but most of us aren't building those, as much as we would like to pretend that we are. I will say that TWO of those servers, with some form of synchronous replication, and point in time snapshots, are probably a better choice, but that's hair-splitting. (and I am a dyed in the wool microservices, scale-out Amazon WS fanboi).
wizofaus 4y ago
Surely having separate DBs all sit on the One Big Server is preferable in many cases. For cases where you really to extract large amounts of data that is derived from multiple DBs, there's no real harm in having some cross-DB joins defined in views somewhere. If there are sensible logical ways to break a monolithic service into component stand-alone services, and good business reasons to do (or it's already been designed that way), then having each talk to their own DB on a shared server should be able to scale pretty well.
cogman10 4y ago
> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position my company finds itself in where our databases can't handle the load we generate. We can't get bigger or faster hardware because we are using the biggest and fastest hardware you can buy. Distributed systems suck, sure, and they make querying cross systems a nightmare. However, by giving those aspects up, what you gain is the ability to add new services, features, etc without running into scotty yelling "She can't take much more of it!" Once you get to that point, it becomes SUPER hard to start splitting things out. All the sudden you have 10000 "just a one off" queries against several domains that are broken by trying carve out a domain into a single owner.
- winrid 4y ago
  You can get a machine with multiple terabytes of ram and hundreds of CPU cores easily. If you can afford that, you can afford a live replica to switch to during maintenance. FastComments runs on one big DB in each region, with a hot backup... no issues yet. Before you go to microservices you can also shard, as others have mentioned.
- moonchrome 4y ago
  >Once you get to that point, it becomes SUPER hard to start splitting things out. All the sudden you have 10000 "just a one off" queries against several domains that are broken by trying carve out a domain into a single owner. But that's survivorship bias and looking back at things from current problems perspective. You know what's the least future proof and scalable project ? The one that gets canceled because they failed to deliver any value in reasonable time in the early phase. Once you get to "huge project status" you can afford glacial pace. Most of the time you can't afford that early on - so even if by some miracle you knew what scaling issues you're going to have long term and invested in fixing them early on - it's rarely been a good tradeoff in my experience. I've seen more projects fail because they tangle themselves up in unnecessary complexity early on and fail to execute on core value proposition, than I've seen fail from being unable to manage the tech debt 10 years in. Developers like to complain about the second, but they get fired on the first kind. Unfortunately in todays job market they just resume pad their failures as "relevant experience" and move on to the next project - so there is not correcting feedback.
hot_gril 4y ago
I'd say, one big database per service. Often times there are natural places to separate concerns and end up with multiple databases. If you ever want to join things for offline analysis, it's not hard to make a mapreduce pipeline of some kind that reads from all of them and gives you that boundless flexibility. Then if/when it comes time for sharding, you probably only have to worry about one of those databases first, and you possibly shard it in a higher-level logical way that works for that kind of service (e.g. one smaller database per physical region of customers) instead of something at a lower level with a distributed database. Horizontally scaling DBs sound a lot nicer than they really are.
mr_toad 4y ago
> Use One Big Database. It’s never one big database. Inevitably there are are backups, replicas, testing environments, staging, development. In an ideal unchanging world where nothing ever fails and workload is predictable then the one big database is also ideal. What happens in the real world is that the one big database becomes such a roadblock to change and growth that organisations often throw away the whole thing and start from scratch.
- lukeramsden 4y ago
  > It’s never one big database. Inevitably there are are backups, replicas, testing environments, staging, development. In an ideal unchanging world where nothing ever fails and workload is predictable then the one big database is also ideal. But if you have many small databases, you need > backups, replicas, testing environments, staging, development all times `n`. Which doesn't sound like an improvement. > What happens in the real world is that the one big database becomes such a roadblock to change and growth that organisations often throw away the whole thing and start from scratch. Bad engineering orgs will clutch defeat from the jaws of victory no matter what the early architectural decisions were. The one vs many databases/services is almost moot entirely.
alecco 4y ago
Mostly agree, but you have to be very strict with the DB architecture. Have very reasonable schema. Punish long running queries. If some dev group starts hammering the DB cut them off early on, don't let them get away with it and then refuse to fix their query design. The biggest nemesis of big DB approach are dev teams who don't care about the impact of their queries. Also move all the read-only stuff that can be a few minutes behind to a separate (smaller) server with custom views updated in batches (e.g. product listings). And run analytics out of peak hours and if possible in a separate server.

by cfors 4y ago

hoosieree 4y ago

This is the macro version of the von Neumann bottleneck.

Closi 4y ago

Breaking apart a stateless microservice and then basing it around a giant single monolithic database is pretty pointless - at that stage you might as well just build a monolith and get on with it as every microservice is tightly coupled to the db.

cfors 4y ago
No disagreement here. I love a good monolith.
roflyear 4y ago
Absolutely. I know someone who considers "different domains" (as in web domains) to count as a microservice! What is the point of that? it doesn't add anything. Just more shit to remember and get right (and get wrong!)
radu_floricica 4y ago
To note that quite a bit of the performance problems come when writing stuff. You can get away with A LOT if you accept 1. the current service doesn't do (much) writing and 2. it can live with slightly old data. Which I think covers 90% of use cases. So you can end up with those services living on separate machines and connecting to read only db replicas, for virtually limitless scalability. And when it realizes it needs to do an update, it either switches the db connection to a master, or it forwards the whole request to another instance connected to a master db.
adrianmsmith 4y ago
That's true, unless you need (1) Different programming languages e.g. you're written your app in Java but now you need to do something for which the perfect Python library is available. (2) Different parts of your software need different types of hardware. Maybe one part needs a huge amount of RAM for a cache, but other parts are just a web server. It'd be a shame to have to buy huge amounts of RAM for every server. Splitting the software up and deploying the different parts on different machines can be a win here. I reckon the average startup doesn't need any of that, not suggesting that monoliths aren't the way to go 90% of the time. But if you do need these things, you can still go the microservices route, but it still makes sense to stick to a single database if at all possible, for consistency and easier JOINs for ad-hoc queries, etc.
- Closi 4y ago
  These are both true - but neither requires service-oriented-architecture. You can split up your applicaiton into chunks that are deployed on seperate hardware, and use different languages, without composing your whole architecture into microservices. A monolith can still have a seperate database server and a web server, or even many different functions split across different servers which are horizontally scalable, and be written in both java and python. Monoliths have had seperate database servers since the 80s (and probably before that!). In fact, part of these applications defining characteristics at the enterprise level is that they often shared one big central database, as often they were composed of lots of small applications that would all make changes to the central database, which would often end up in a right mess of software that was incredibly hard to de-pick! (And all the software writing to that database would, as you described, be written in lots of different languages). People would then come along and cake these central databases full of stored procedures to make magic changes to implement functionality that wasn't available in the legacy applications that they can't change because of the risk and then you have even more of a mess!
noduerme 4y ago
I disagree. Suppose you have an enormous DB that's mainly written to by workers inside a company, but has to be widely read by the public outside. You want your internal services on machines with extra layers of security, perhaps only accessible by VPN. Your external facing microservices have other things like e.g. user authentication (which may be tied to a different monolithic database), and you want to put them closer to users, spread out in various data centers or on the edge. Even if they're all bound to one database, there's a lot to recommend keeping them on separate, light cheap servers that are built for http traffic and occasional DB reads. And even more so if those services do a lot of processing on the data that's accessed, such as building up reports, etc.
- Closi 4y ago
  You've not really built microservices then in the purest sense though - i.e. all the microservices aren't independently deployable components. I'm not saying what you are proposing isn't a perfectly valid architectural approach - it's just usually considered an anti-pattern with microservices (because if all the services depend on a single monolith, and a change to a microservice functionality also mandates a change to the shared monolith which then can impact/break the other services, we have lost the 'independence' benefit that microservices supposedly gives us where changes to one microservice does not impact another). Monoliths can still have layers to support business logic that are seperate to the database anyway.
manigandham 4y ago
Why would you break apart a microservice? Any why do you need to use/split into microservices anyway? 99% of apps are best fit as monolithic apps and databases and should focus on business value rather than scale they'll never see.
- Closi 4y ago
  Totally agree. I guess I just don't see the value in having a monolith made up of microservices - you might as well just build a monolith if you are going down that route. And if your application fits the microservices pattern better, then you might as well go down the microservices pattern properly and not give them a big central DB.
  - adgjlsfhk1 4y ago
    The one advantage of microservice on a single database model is that it lets you test the independent components much more easily while avoiding the complexity of database sharding.
- Gigachad 4y ago
  Where I work we are looking at it because we are starting to exceed the capabilities of one big database. Several tables are reaching the billions of rows mark and just plain inserts are starting to become too much.
  - nicoburns 4y ago
    Yeah, the at the billions of rows mark it definitely makes sense to start looking at splitting things up. On the other hand, the company I worked for split things up from the start, and when I joined - 4 years down the line - their biggest table had something like 50k rows, but their query performance was awful (tens of seconds in cases) because the data was so spread out.
    - collyw 4y ago
      Am I missing something? 2 ^ 30 is over 1 billion. So a properly indexed table with 1 billion rows will take a similar effort to search as a 30 row unindexed table. Or are there other factors coming into play that I haven't thought of.
AtNightWeCode 4y ago
Agree. Nothing worse than having different programs changing data in the same database. The database should not be an integration point between services.
- jethro_tell 4y ago
  if you have multiple micro services updating the database you need to have a database access layer service as well. there's some real value with abstraction and microservices but you can try to run them against a monolithic database service
  - bergkvist 4y ago
    No amount of abstraction is going to save you from the problem of 2 processes manipulating the same state machine.
    - pc86 4y ago
      In this example, it's the job of the "database access layer service" to manage those processes and prevent issues. But, terrible service name aside, this is a big reason why two services accessing the same database is a capital-H Huge anti-pattern, and really screams "using this project to learn how to do microservices."
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]

belak 4y ago

This is absolutely true - when I was at Bitbucket (ages ago at this point) and we were having issues with our DB server (mostly due to scaling), almost everyone we talked to said "buy a bigger box until you can't any more" because of how complex (and indirectly expensive) the alternatives are - sharding and microservices both have a ton more failure points than a single large box. I'm sure they eventually moved off that single primary box, but for many years Bitbucket was run off 1 primary in each datacenter (with a failover), and a few read-only copies. If you're getting to the point where one database isn't enough, you're either doing something pretty weird, are working on a specific problem which needs a more complicated setup, or have grown to the point where investing in a microservice architecture starts to make sense.

altdataseller 4y ago
What if your product simply stores a lot of data (ie a search engine) How is that weird?
- rmbyrro 4y ago
  a search engine won't need joins, but other things (ie text indexing) that can be split in a relatively easier way.
- zasdffaa 4y ago
  Depends what you mean by a database I guess. I take it to mean an RDBMS. RDBMSs provide guarantees that web searching doesn't need. You can afford to lose a pieces of data, provide not-quite-perfect results for web stuff. It's just wrong for an RDBMS.
  - altdataseller 4y ago
    What if you are using the database as a system of record to index into a real search engine like Elasticsearch? For a product where you have tons of data to search from (ie text from web pages)
    - 4y ago

ryanisnan 4y ago

Definitely use a big database, until you can't. My advice to anyone starting with a relational data store is to use a proxy from day 1 (or some point before adding something like that becomes scary). When you need to start sharding your database, having a proxy is like having a super power.

_ben_ 4y ago
Disclaimer: I am the founder of PolyScale [1]. We see both use cases: single large database vs multiple small, decoupled. I agree with the sentiment that a large database offer simplicity, until access patterns change. We focus on distributing database data to the edge using caching. Typically this eliminates read-replicas and a lot of the headache that goes with app logic rewrites or scaling "One Big Database". [1] https://www.polyscale.ai/ https://www.polyscale.ai/
chromatin 4y ago
Are there postgres proxies that can specifically facilitate sharding / partitioning later?

cdkmoose 4y ago

>>(they don't know how your distributed databases look, and oftentimes they really do not care) Nor should they, it's the engineer's/team's job to provide the database layer to them with high levels of service without them having to know the details

markandrewj 4y ago

Just FYI, you can have one big database, without running it on one big server. As an example, databases like Cassandra are designed to be scaled horizontally (i.e. scale out, instead of scale up). https://cassandra.apache.org/_/cassandra-basics.html https://cassandra.apache.org/_/cassandra-basics.html

mdasen 4y ago
There are trade-offs when you scale horizontally even if a database is designed for it. For example, DataStax's Storage Attached Indexes or Cassandra's hidden-table secondary indexing allow for indexing on columns that aren't part of the clustering/partitioning, but when you're reading you're going to have to ask all the nodes to look for something if you aren't including a clustering/partitioning criteria to narrow it down. You've now scaled out, but you now have to ask each node when searching by secondary index. If you're asking every node for your queries, you haven't really scaled horizontally. You've just increased complexity. Now, maybe 95% of your queries can be handled with a clustering key and you just need secondary indexes to handle 5% of your stuff. In that case, Cassandra does offer an easy way to handle that last 5%. However, it can be problematic if people take shortcuts too much and you end up putting too much load on the cluster. You're also putting your latency for reads at the highest latency of all the machines in your cluster. For example, if you have 100 machines in your cluster with a mean response time of 2ms and a 99th percentile response time of 150ms, you're potentially going to be providing a bad experience to users waiting on that last box on secondary index queries. This isn't to say that Cassandra isn't useful - Cassandra has been making some good decisions to balance the problems engineers face. However, it does come with trade-offs when you distribute the data. When you have a well-defined problem, it's a lot easier to design your data for efficient querying and partitioning. When you're trying to figure things out, the flexibility of a single machine and much cheaper secondary index queries can be important - and if you hit a massive scale, you figure out how you want to partition it then.
- markandrewj 4y ago
  Cassandra was just an example, but most databases can be scaled either vertically or horizontally via sharding. You are right if misconfigured performance can be hindered, but this is also true for a database which is being scaled vertically. Generally speaking you will get better performance if you have a large dataset by growing horizontally then you would by growing vertically. https://stackoverflow.blog/2022/03/14/how-sharding-a-database-can-make-it-faster/ https://stackoverflow.blog/2022/03/14/how-sharding-a-databas...

johnbellone 4y ago

I agree with this sentiment but it is often misunderstood as a means to force everything into a single database schema. More people need to learn about logically separating schemas with their database servers!

Guid_NewGuid 4y ago

I think a strong test a lot of "let's use Google scale architecture for our MVP" advocates fail is: can your architecture support a performant paginated list with dynamic sort, filter and search where eventual consistency isn't acceptable? Pretty much every CRUD app needs this at some point and if every join needs a network call your app is going to suck to use and suck to develop.

mhoad 4y ago
I’ve found the following resource invaluable for designing and creating “cloud native” APIs where I can tackle that kind of thing from the very start without a huge amount of hassle https://google.aip.dev/general https://google.aip.dev/general The patterns section covers all of this and more
- gnat 4y ago
  This is a great resource but the RFC-style documentation says what you SHOULD and MUST do, not HOW to do it ...
skyde 4y ago
thanks a lot for this comment. I will borrow this as an interview question :)
SkyPuncher 4y ago
> Pretty much every CRUD app needs this at some point and if every join needs a network call your app is going to suck to use and suck to develop. _at some point_ is the key word here. Most startups (and businesses) can likely get away with this well into Series A or Series B territory.
threeseed 4y ago
> if every join needs a network call your app is going to suck to use and suck to develop. And yet developers do this every single day without any issue. It is bad practice to have your authentication database be the same as your app database. Or you have data coming from SaaS products, third party APIs or a cloud service. Or even simply another service in your stack. And with complex schemas often it's far easier to do that join in your application layer. All of these require a network call and join.

ladyattis 4y ago

At my current job we have four different databases so I concur with this assessment. I think it's okay to have some data in different DBs if they're significantly different like say the user login data could be in its own database. But anything that we do which is a combination of e-commerce and testing/certification I think they should be in one big database so I can do reasonable queries for information that we need. This doesn't include two other databases we have on-prem which one is a Salesforce setup and another is an internal application system that essentially marries Salesforce to that. It's a weird wild environment to navigate when adding features.

robertlagrant 4y ago

> Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). I'm not sure how to parse this. What should "asks" be?

cfors 4y ago
The feature requests (asks) that product wants to build - sorry for the confusion there.
delecti 4y ago
The phrase "Your product asks will consistently " can be de-abbreviated to "product owners/product managers you work with will consistently request".

rbanffy 4y ago

> Use One Big Database. I emphatically disagree. I've seen this evolve into tightly coupled microservices that could be deployed independently in theory, but required exquisite coordination to work. If you want them to be on a single server, that's fine, but having multiple databases or schemas will help enforce separation. And, if you need one single place for analytics, push changes to that space asynchronously. Having said that, I've seen silly optimizations being employed that make sense when you are Twitter, and to nobody else. Slice services up to the point they still do something meaningful in terms of the solution and avoid going any further.

danpalmer 4y ago
To clarify the advice, at least how I believe it should be done… Use One Big Database Server… … and on it, use one software database per application. For example, one Postgres server can host many databases that are mostly* independent from each other. Each application or service should have its own database and be unaware of the others, communicating with them via the services if necessary. This makes splitting up into multiple database servers fairly straightforward if needed later. In reality most businesses will have a long tail of tiny databases that can all be on the same server, with only bigger databases needing dedicated resources. *you can have interdependencies when you’re using deep features sometimes, but in an application-first development model I’d advise against this.
- goodoldneon 4y ago
  OP mentioned joining, so they were definitely talking about a single database
  - danpalmer 4y ago
    You can still do a ton of joining. I’d start with a monolith, that’s a single app, single database, single point of ownership of the data model, and a ton of joins. Then as services are added after the monolith they can still use the main database for ease of infra development, simpler backups and replication, etc. but those wouldn’t be able to be joined because they’re cross-service.

bartread 4y ago

Not to mention, backups, restores, and disaster recovery are so much easier with One Big Database™.

1500100900 4y ago
How is backup restoration any easier if your whole PostgreSQL cluster goes back in time when you only wanted to rewind that one tenant?
- fleddr 4y ago
  Your scenario is data recovery, not backup restoration. Wildly different things.

clairity 4y ago

> "Use One Big Database." yah, this is something i learned when designing my first server stack (using sun machines) for a real business back during the dot-com boom/bust era. our single database server was the beefiest machine by far in the stack, 5U in the rack (we also had a hot backup), while the other servers were 1U or 2U in size. most of that girth was for memory and disk space, with decent but not the fastest processors. one big db server with a hot backup was our best tradeoff for price, performance, and reliability. part of the mitigation was that the other servers could be scaled horizontally to compensate for a decent amount of growth without needing to scale the db horizontally.

marcosdumay 4y ago

> Use One Big Database Yep, with a passive replica or online (log) backup. Keeping things centralized can reduce your hardware requirement by multiple orders of magnitude. The one huge exception is a traditional web service, those scale very well, so you may not even want to get big servers for them (until you need them).

s_dev 4y ago

>Use One Big Database. It may be reasonable to have two databases e.g. a class a and class b for pci compliance. So context still deeply matters. Also having a dev DB with mock data and a live DB with real data is a common setup in many companies.

jasonwatkinspdx 4y ago

A relative worked for a hedge fund that used this idea. They were a C#/MSSQL shop, so they just bought whatever was the biggest MSSQL server at the time, updating frequently. They said it was a huge advantage, where the limit in scale was more than offset by productivity. I think it's an underrated idea. There's a lot of people out there building a lot of complexity for datasets that in the end are less than 100 TB. But it also has limits. Infamously Twitter delayed going to a sharded architecture a bit too long, making it more of an ugly migration.

manigandham 4y ago
Server hardware is so cheap and fast today that 99% of companies will never hit that limit in scale either.

throwaway894345 4y ago

I'm pretty happy to pay a cloud provider to deal with managing databases and hosts. It doesn't seem to cause me much grief, and maybe I could do it better but my time is worth more than our RDS bill. I can always come back and Do It Myself if I run out of more valuable things to work on. Similarly, paying for EKS or GKE or the higher-level container offerings seems like a much better place to spend my resources than figuring out how to run infrastructure on bare VMs. Every time I've seen a normal-sized firm running on VMs, they have one team who is responsible for managing the VMs, and either that team is expecting a Docker image artifact or they're expecting to manage the environment in which the application runs (making sure all of the application dependencies are installed in the environment, etc) which typically implies a lot of coordination between the ops team and the application teams (especially regarding deployment). I've never seen that work as smoothly as deploying to ECS/EKS/whatever and letting the ops team work on automating things at a higher level of abstraction (automatic certificate rotation, automatic DNS, etc). That said, I've never tried the "one big server" approach, although I wouldn't want to run fewer than 3 replicas, and I would want reproducibility so I know I can stand up the exact same thing if one of the replicas go down as well as for higher-fidelity testing in lower environments. And since we have that kind of reproducibility, there's no significant difference in operational work between running fewer larger servers and more smaller servers.

AtNightWeCode 4y ago

If you get your services right there is little or no communications between the services since a microservice should have all the data it needs in it's own store.

wefarrell 4y ago

"Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care)." This isn't a problem if state is properly divided along the proper business domain and the people who need to access the data have access to it. In fact many use cases require it - publicly traded companies can't let anyone in the organization access financial info and healthcare companies can't let anyone access patient data. And of course are performance concerns as well if anyone in the organization can arbitrarily execute queries on any of the organization's data. I would say YAGNI applies to data segregation as well and separations shouldn't be introduced until they are necessary.

Mavvie 4y ago
"combine these data sources" doesn't necessarily mean data analytics. Just as an example, it could be something like "show a badge if it's the user's birthday", which if you had a separate microservice for birthdays would be much harder than joining a new table.
- wefarrell 4y ago
  Replace "people" with "features" and my comment still holds. As software, features, and organizations become more complex the core feature data becomes a smaller and smaller proportion of the overall state and that's when microservices and separate data stores become necessary.

BenoitEssiambre 4y ago

I'm glad this is becoming conventional wisdom. I used to argue this in these pages a few years ago and would get downvoted below the posts telling people to split everything into microservices separated by queues (although I suppose it's making me lose my competitive advantage when everyone else is building lean and mean infrastructure too). In my mind, reasons involve keeping transactional integrity, ACID compliance, better error propagation, avoiding the hundreds of impossible to solve roadblocks of distributed systems (https://groups.csail.mit.edu/tds/papers/Lynch/MIT-LCS-TM-394.pdf https://groups.csail.mit.edu/tds/papers/Lynch/MIT-LCS-TM-394...). But also it is about pushing the limits of what is physically possible in computing. As Admiral Grace Hopper would point out (https://www.youtube.com/watch?v=9eyFDBPk4Yw https://www.youtube.com/watch?v=9eyFDBPk4Yw ) doing distance over network wires involves hard latency constraints, not to mention dealing with congestions over these wires. Physical efficiency is about keeping data close to where it's processed. Monoliths can make much better use of L1, L2, L3, and ram caches than distributed systems for speedups often in the order of 100X to 1000X. Sure it's easier to throw more hardware at the problem with distributed systems but the downsides are significant so be sure you really need it. Now there is a corollary to using monoliths. Since you only have one db, that db should be treated as somewhat sacred, you want to avoid wasting resources inside it. This means being a bit more careful about how you are storing things, using the smallest data structures, normalizing when you can etc. This is not to save disk, disk is cheap. This is to make efficient use of L1,L2,L3 and ram. I've seen boolean true or false values saved as large JSON documents. {"usersetting1": true, "usersetting2":fasle "setting1name":"name" etc.} with 10 bits of data ending up as a 1k JSON document. Avoid this! Storing documents means, the keys, the full table schema is in every row. It has its uses but if you can predefine your schema and use the smallest types needed, you are gaining much performance mostly through much higher cache efficiency!

tsmarsh 4y ago
'over the wire' is less obvious than it used to be. If you're in k8s pod, those calls are really kernel calls. Sure you're serializing and process switching where you could be just making a method call, but we had to do something. I'm seeing less 'balls of mud' with microservices. Thats not zero balls of mud. But its not a given for almost every code base I wander into.

beachy 4y ago

Another area for consolidation is auth. Use one giant keycloak, with individual realms for every one of the individual apps you are running. Your keycloak is back ended by your one giant database.

z3t4 4y ago

The rule is: Keep related data together. Exceptions are: Different customers (usually don't require each others data) can be isolated. And if the database become the bottleneck you can separate unrelated services.

throwaway9870 4y ago

How do you use one big database when some of your info is stuck in an ERP system?

FpUser 4y ago

>"Use One Big Database." I do, it is running on the same big (relatively) server as my native C++ backend talking to the database. The performance smokes your standard cloudy setup big time. Serving thousand requests per second on 16 core without breaking sweat. I am all for monoliths running on real no cloudy hardware. As long as the business scale is reasonable and does not approach FAANG (like for 90% of the businesses) this solution is superior to everything else money, maintenance, development time wise.

HeavyStorm 4y ago

> they don't know how your distributed databases look, and oftentimes they really do not care Nor should they.

doctor_eval 4y ago

I agree that 1BDB is a good idea, but having one ginormous schema has its own costs. So I still think data should be logically partitioned between applications/microservices - in PG terms, one “cluster” but multiple “databases”. We solved the problem of collecting data from the various databases for end users by having a GraphQL layer which could integrate all the data sources. This turned out to be absolutely awesome. You could also do something similar using FDW. The effort was not significant relative to the size of the application. The benefits of this architecture were manifold but one of the main ones is that it reduces the complexity of each individual database, which dramatically improved performance, and we knew that if we needed more performance we could pull those individual databases out into their own machine.

lmm 4y ago

If you do this then you'll have the hardest possible migration when the time comes to split it up. It will take you literally years, perhaps even a decade. Shard your datastore from day 1, get your dataflow right so that you don't need atomicity, and it'll be painless and scale effortlessly. More importantly, you won't be able to paper over crappy dataflow. It's like using proper types in your code: yes, it takes a bit more effort up-front compared to just YOLOing everything, but it pays dividends pretty quickly.

riku_iki 4y ago
> Shard your datastore from day 1 what about using something like cocroach from day 1?
- lmm 4y ago
  I don't know the characteristics of bikesheddb's upstream in detail (if there's ever a production-quality release of bikesheddb I'll take another look), but in general using something that can scale horizontally (like Cassandra or Riak, or even - for all its downsides - MongoDB) is a great approach - I guess it's a question of terminology whether you call that "sharding" or not. Personally I prefer that kind of datastore over an SQL database.
  - riku_iki 4y ago
    > over an SQL database it is actually distributed SQL Db with auto sharding, their goal is to be SQL compatible with Postgres.
Rantenki 4y ago
This is true IFF you get to the point where you have to split up. I know we're all hot and bothered about getting our apps to scale up to be the next unicorn, but most apps never need to scale past the limit of a single very high-performance database. For most people, this single huge DB is sufficient. Also, for many (maybe even most) applications, designated outages for maintenance are not only acceptable, but industry standard. Banks have had, and continue to have designated outages all the time, usually on weekends when the impact is reduced. Sure, what I just wrote is bad advice for mega-scale SaaS offerings with millions of concurrent users, but most of us aren't building those, as much as we would like to pretend that we are. I will say that TWO of those servers, with some form of synchronous replication, and point in time snapshots, are probably a better choice, but that's hair-splitting. (and I am a dyed in the wool microservices, scale-out Amazon WS fanboi).

wizofaus 4y ago

Surely having separate DBs all sit on the One Big Server is preferable in many cases. For cases where you really to extract large amounts of data that is derived from multiple DBs, there's no real harm in having some cross-DB joins defined in views somewhere. If there are sensible logical ways to break a monolithic service into component stand-alone services, and good business reasons to do (or it's already been designed that way), then having each talk to their own DB on a shared server should be able to scale pretty well.

cogman10 4y ago

> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position my company finds itself in where our databases can't handle the load we generate. We can't get bigger or faster hardware because we are using the biggest and fastest hardware you can buy. Distributed systems suck, sure, and they make querying cross systems a nightmare. However, by giving those aspects up, what you gain is the ability to add new services, features, etc without running into scotty yelling "She can't take much more of it!" Once you get to that point, it becomes SUPER hard to start splitting things out. All the sudden you have 10000 "just a one off" queries against several domains that are broken by trying carve out a domain into a single owner.

winrid 4y ago
You can get a machine with multiple terabytes of ram and hundreds of CPU cores easily. If you can afford that, you can afford a live replica to switch to during maintenance. FastComments runs on one big DB in each region, with a hot backup... no issues yet. Before you go to microservices you can also shard, as others have mentioned.
moonchrome 4y ago
>Once you get to that point, it becomes SUPER hard to start splitting things out. All the sudden you have 10000 "just a one off" queries against several domains that are broken by trying carve out a domain into a single owner. But that's survivorship bias and looking back at things from current problems perspective. You know what's the least future proof and scalable project ? The one that gets canceled because they failed to deliver any value in reasonable time in the early phase. Once you get to "huge project status" you can afford glacial pace. Most of the time you can't afford that early on - so even if by some miracle you knew what scaling issues you're going to have long term and invested in fixing them early on - it's rarely been a good tradeoff in my experience. I've seen more projects fail because they tangle themselves up in unnecessary complexity early on and fail to execute on core value proposition, than I've seen fail from being unable to manage the tech debt 10 years in. Developers like to complain about the second, but they get fired on the first kind. Unfortunately in todays job market they just resume pad their failures as "relevant experience" and move on to the next project - so there is not correcting feedback.

hot_gril 4y ago

I'd say, one big database per service. Often times there are natural places to separate concerns and end up with multiple databases. If you ever want to join things for offline analysis, it's not hard to make a mapreduce pipeline of some kind that reads from all of them and gives you that boundless flexibility. Then if/when it comes time for sharding, you probably only have to worry about one of those databases first, and you possibly shard it in a higher-level logical way that works for that kind of service (e.g. one smaller database per physical region of customers) instead of something at a lower level with a distributed database. Horizontally scaling DBs sound a lot nicer than they really are.

mr_toad 4y ago

> Use One Big Database. It’s never one big database. Inevitably there are are backups, replicas, testing environments, staging, development. In an ideal unchanging world where nothing ever fails and workload is predictable then the one big database is also ideal. What happens in the real world is that the one big database becomes such a roadblock to change and growth that organisations often throw away the whole thing and start from scratch.

lukeramsden 4y ago
> It’s never one big database. Inevitably there are are backups, replicas, testing environments, staging, development. In an ideal unchanging world where nothing ever fails and workload is predictable then the one big database is also ideal. But if you have many small databases, you need > backups, replicas, testing environments, staging, development all times `n`. Which doesn't sound like an improvement. > What happens in the real world is that the one big database becomes such a roadblock to change and growth that organisations often throw away the whole thing and start from scratch. Bad engineering orgs will clutch defeat from the jaws of victory no matter what the early architectural decisions were. The one vs many databases/services is almost moot entirely.

alecco 4y ago

Mostly agree, but you have to be very strict with the DB architecture. Have very reasonable schema. Punish long running queries. If some dev group starts hammering the DB cut them off early on, don't let them get away with it and then refuse to fix their query design. The biggest nemesis of big DB approach are dev teams who don't care about the impact of their queries. Also move all the read-only stuff that can be a few minutes behind to a separate (smaller) server with custom views updated in batches (e.g. product listings). And run analytics out of peak hours and if possible in a separate server.