16 ms·

The database ruins all good ideas

251 points by kristianp 5y ago

yowlingcat 5y ago
Am I just dense, or was the idea the CEO had just not a good idea and simply garden variety premature optimization? Also, it seems unfair to say that the database ruins all good ideas when there continues to be significant impressive innovation in the space in cloud services. Like Amazon Aurora for one. If you really want to treat your RDBMS like an elastically scalable (with significant caveats) resource priced by the request, you can.
- justsomeuser 5y ago
  I think the title implicitly meant “good”, as in “you think it’s a good idea until you press it against the grindstone that is reality”. Also I think these issues arise at the database because that’s were the write concurrency is pushed.
thrill 5y ago
Use CockroachDB.
- AdrianB1 5y ago
  To solve what problem?
  - rad_gruchalski 5y ago
    To get replication and horizontal scalability for free (well, within the license fee). But I’d go for YugabyteDB instead of CockroachDB.
- bobthebuilders 5y ago
  CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC which contirbutes to GC pauses. The dev team refuses to listen to feedback to port their code to C ;(.
  - Guvante 5y ago
    Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument when it comes to any production system that it isn't even worth discussing. The reality is there are limited resources to work on any given project and "rewrite" is generally not the correct way to fix a given problem. Especially since the first thing you are going to need to do is show that a network system isn't resilient to GC pauses or that GC pauses are frequent enough to impact throughput (very very few applications are actually materially impacted by sub second spikes in latency).
    - pkolaczk 5y ago
      GC pauses are not the only GC issue. Thrashing the caches and blowing up memory usage by an order of magnitude can be also very bad for performance. In a database system memory is very precious - the more of it you can use for caching / buffering users data, the better the performance. As for the subsecond spikes in latency, these tend to multiply in a distributed system. If serving a client request takes N internal requests, the likelihood of hitting a GC pause somewhere is much larger than if you did only one local request. Not sure about Go, but none of the "free" Java's GC guarantees low pauses. There is STW fallback even in the most advanced ones like ZGC. So you never know when it stops for more than 1s.
      - ngrilly 5y ago
        Go GC pauses are bounded at 0.5 ms: https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
        pkolaczk 5y ago
        Is it a hard guarantee or just a soft goal like in G1, which tries to stay within the target pause milliseconds, but there are many ways it can fail to? Can it handle 100+ GB large heaps? Even if it is a hard guarantee, then, from the link you posted, it is not even generational, so it will scan the whole heap quite frequently, and that is going to influence average performance quite visibly - you definitely dont want a database system to access all its cached memory once in a while. Database systems are really all about memory and I/O management. You shouldn't outsource those core features to a universal algorithm, unless you wish to forgo any competitive advantage (at least in performance department). So this pushes the devs into the off-heap manual memory management territory, where dragons live (at least in Java, again - maybe Go is better in that regard). I've been there, and I don't recommend.
hughrr 5y ago
No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider. The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. If you spent the last decade writing shitty cross database SQL then you’ll have to silo by customer. That’s the only real constraint.
- tremon 5y ago
  I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good". Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without understanding the problem space is a disaster waiting to happen (but luckily, it may not happen on your watch so you can continue hacking).
  - hughrr 5y ago
    Very good points there. I have directly experienced the problems associated with blindly assuming that you can write your own consistency guarantees in Redis. I didn't do it but I had to write some tooling to rewrite the AOF to get some data back...
  - HWR_14 5y ago
    I'm sorry, are you saying that a good idea should survive the loss of referential integrity or data consistency? I don't feel that's what you mean, and it's probably my fault for misreading your comment.
    - erik_seaberg 5y ago
      I read it the other way: if abandoning correctness is the only way to make your idea feasible, your idea kinda sucks.
jarym 5y ago
The mistake is thinking half the system is doing nothing. It’s a bit like saying I’m paying to store my backups offsite but most of the time it’s not really used so it’s a waste of money. It just doesn’t hold. Most SQL problems that most of us have to deal with stem from inadequate indexes and/or poorly written queries. No fancy active-active setup will ever solve those issues.
- jayd16 5y ago
  It might be a waste. Think RAID 1 vs RAID 5. There might be plenty of ways were leaving performance on the table.
  - jeltz 5y ago
    Which one of those leaves performance on the table? One is optimized for performance and the other for disk usage.
    - jayd16 5y ago
      Maybe its a bit tortured of a metaphor, but perhaps with continued research we can find a way to use more of the hardware just like how RAID 5 provides more storage over RAID 1 while still keeping parity. I don't think its foolish to keep asking for more. We still might find it.
renewiltord 5y ago
Can’t be bothered. Aurora in Postgres/MySQL and then just scale that vertically till it stops.
rrdharan 5y ago
If you really want or need to go active-active but also don’t want to think about it much and can live with MySQL my preference would probably be AWS Aurora multi-master if you can live with its limitations: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-multi-master.html#aurora-multi-master-limitations https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide... As noted there and elsewhere though often you are better off just scaling up. If you do want to really think about it then NewSQL (Cloud Spanner or others) could be the way to go but the cost, performance, compatibility/portability and other concerns make this a much more complex decision that isn’t easy to capture in a blog post (or HN comment ;).
trollied 5y ago
The article is talking about Oracle RAC, and the "crossover" he is talking about was usually really quick Infiniband, not a crossover cable (so, a separate NIC for good reason!). The person that wrote the article doesn't seem to understand RAC enough to actually comment properly. A properly tuned RAC instance will scale horizontally very very well.
- don-code 5y ago
  Former (recovering?) Exadata user here. I'll go as far as to say that configuring a best-practices RAC cluster is easier than configuring a best-practices MySQL cluster. RAC solves some additional problems, like discovery and load balancing, that MySQL kicks out to the user (e.g. you have to manage DNS and run a tool like ProxySQL separately).
- inopinatus 5y ago
  If you look back far enough, it wasn’t infiniband but a small separate fibrechannel fabric, and further back again I recall buying specialist dual-ported SCSI drives that were marketed primarily for cluster i.e. quorum-disk purposes. More than one goat was required to make these configurations work. (I seem to recall that many SAS drives now are dual-ported but designed as such to fulfil a desire for multi-path resiliency within storage arrays; this may have different characteristics to those needed of a quorum device)
- hestefisk 5y ago
  I’ve seen Oracle RAC in use in many banks in particular where uptime and resilience are critical requirements. It scales well horizontally, but at a very high cost.
- cratermoon 5y ago
  How much redundancy do you really have if both (or all N) systems are within 10km of each other?
- Swannie 5y ago
  Alternatively a private L2 network for a N > 2 RAC cluster.
edelweis 5y ago
Databases scale just fine. Its called sharding. Azure has HyperScale (with is Citus), AWS has Aurora. The actual problem with databases is upgrading to major versions without downtime. All the other problems are just noise
- sonthonax 5y ago
  Agreed. But one impediment to sharding is bad DB design, which is rife right now because everyone starts with Django/Rails/Laravel and starts treating the DB like a document store that happens to have some relational features.
robertlagrant 5y ago
I think that being able to have separate databases for different domains is pretty good, although it does rely partly on the application layer to keep (UU)IDs consistent. I'd be curious to know if there's a way to have databases talk to each other to just sync up primary keys for referential integrity. That could maximise the benefit of decoupled databases while still having good referential integrity. And a network disconnection would still mean existing PKs would be in place, it just wouldn't be aware of new ones. Not perfect, but not bad, perhaps.
- cratermoon 5y ago
  https://lamport.azurewebsites.net/pubs/paxos-simple.pdf https://lamport.azurewebsites.net/pubs/paxos-simple.pdf
AdrianB1 5y ago
A dockerized database server is something I cannot understand; I understand bare metal, I can accept virtualized, but I cannot find a good used case for a mid sized or large server (dozens of gigabytes to dozens of terabytes) dockerized and I don't know why a smaller server is a problem.
- kevml 5y ago
  A docker image is simply packaging. There’s a small, almost always negligible, performance hit. But the value comes in being able to ship a consistent application and configuration pairing. This becomes really valuable when shipping multiple databases for multiple services.
  - AdrianB1 5y ago
    I know what docker does, I never saw a database server deployed in mass. One of my teams manage over 250 database servers, but nobody ever supported the idea of using docker for that (all the servers run on Windows). Shipping multiple databases for multiple services does not necessarily mean multiple database servers. In the database world I saw large servers with many databases more often than multiple servers of one database each; in the second case it was the vendor laziness (cannot give the name), not than a reasonable business or technical reason. When I asked about it the answer was "we'll consolidate in the next release".
    - kevml 5y ago
      Sure there’s a lot of variables here. If you’re a Windows shop I would not expect a push for docker since it is Linux based. As far as multi-tenant logical databases, that is also highly dependent on workload and failure domains (and significant cost if you’re talking Oracle).
- majewsky 5y ago
  For our team, the reason is standardization. We have our entire baremetal infrastructure running Kubernetes. There is no process for running anything in not-a-Docker-container. This does come with some caveats for stateful databases, but it's still easier to manage than a special-snowflake environment for database deployments.
l0k3ndr 5y ago
I have been working on rewriting a monolith into individual services during past year at work - and what we finally implemented for consistency across different databases of services was that we keep a signal queue - where whenever we encounter an inconsistency, a doc is pushed containing the info and we have a corrective service always reading from that queue and doing the necessary updates to tables. We made a heirarchy of source-of-truthness and we use that to decide what to do with inconsistent data. Though, users sometimes have to see a error messagge "We had an error and we are fixing it in few XXX time" based on current queue load - but it has been working fine mostly for us.
- danielheath 5y ago
  … is that less operational overhead than running a single process on a server?
  - deleted 5y ago
    [deleted]
- aflag 5y ago
  Why was that change made? From my understanding users are constantly seeing outages now where they see an error message and have to wait. Were there more outages before? How was the previous system?
zzzeek 5y ago
just because it's the thing I know best at the moment, it's worth checking out Galera which actually does do multi-master replication. Of course you have to live in MySQL/MariaDB world which is a non-starter for a lot of peeps.
samatman 5y ago
SQLite is a remarkably good solution to most of these problems, if deployed correctly. For your main line-of-business database? Of course not. But a deployment of rqlite[0] for your service workers in a read-heavy workload? cuts out a round-trip out of the VM, mocking is trivial, there's a lot to like there. [0]: https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite
- eloff 5y ago
  You're missing the whole point of the article. If you set this up in an active-active configuration (if that's even supported) you have the exact same distributed systems problems as the author.
- cortesoft 5y ago
  The whole point of having a database is that you need persistent data that is consistent for all clients. You can’t have a bunch of SQLite dbs and get that.
- otoolep 5y ago
  rqlite author here. Happy to answer any questions about it.
simonw 5y ago
The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed a write have their read traffic sent to the leader for the next 10 seconds or so. Not trivial, but also not impossibly difficult - plenty of places implement this without too much trouble. Scaling writes is a lot harder - but a well specc'd relational database server will handle tens of thousands of writes per second, so the vast majority of projects will never have to solve this. When you do need to solve this, patterns for horizontally sharding your data exist. They're not at all easy to implement, but it's not an impossible problem either. The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box). Overall, my experience is that the database /enables/ all good ideas. Building stateful applications without a relational database in the mix is usually the wrong choice.
- tremon 5y ago
  horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas There's one feature that I'd like to see in that area: partitioned writable replicas. In the same vein that you can partition the table storage across an index, I'd like it to be possible to assign different writers to different parts of a table/database. Of course, you'd still need a single primary replica to handle the transactions that transcend the configured partitions, but we already have routing engines that can transparently redirect an incoming query to any available replica, so it's partly there. There's probably corner cases lurking that I can't even think of, but in my mind it's the only thing missing from building a truly web-scale (multiple zones, multiple datacenters) ACID-preserving relational database.
  - taffer 5y ago
    This is essentially the idea of VoltDB [1]: All tables are partitioned and each partition is assigned to a single thread. As long as each query only touches a single partition, you get linear horizontal scalability and serializable isolation. The drawback is that if a query touches multiple partitions, only that one query can be executed on the entire cluster, so this architecture only works for very specific OLTP workloads. [1] https://www.voltdb.com/ https://www.voltdb.com/
benlivengood 5y ago
And this is why Google wrote Spanner. I think cockroachdb tries to solve the same problems. If you need ACID compliance and you need a lot of it, everywhere, all the time, now there are better options than giant Sun/IBM boxes. Databases are not the problem.
- tyingq 5y ago
  It's still a fairly new thing for people outside of Google. It's been a few years, so I imagine it's better now...but when I tried CockroachDB, it was pretty easy to trigger horrible write latency. Bad enough to be a non-starter.
  - namibj 5y ago
    Which version, and what networking did you have between the cluster nodes?
    - tyingq 5y ago
      I don't recall other than it was before 2.x, but I don't think it's terribly important. The cockroachdb folks were very transparent about it, including a couple of major re-works to address performance. Here's a blog post about one of them: https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ I was just trying to say that a stable, high performance SQL database with distributed writes is a fairly new thing.
- CRConrad 5y ago
  > If you need ACID compliance and you need a lot of it, everywhere, all the time, now there are better options than giant Sun/IBM boxes. Oh I dunno, I'd still say most claims to be "better" at data integrity with performance, and a lot of it than a good old zOS mainframe with a soundly designed DB2 database on it are [Citation needed]-class... But yeah, databases certainly aren't the problem here.
da39a3ee 5y ago
I’m not convinced the author knows what they’re talking about. The answer to the question they posed is basically “ACID”, so unclear what all the verbiage is. Also unclear why they expect all readers to have an architecture featuring multiple application servers all sharing a single database, since that is neither classic monolith nor classic microservices.
- slumdev 5y ago
  I get the same impression. Relational with ACID guarantees can't be implemented in a distributed system. Depending on which guarantees you want to give up, you might be able to get close. A lot of newer platforms give up guarantees to achieve something like this. But if you have multiple nodes that have to sync every transaction over a dedicated NIC, well, that's not a distributed system. It's just multiprocessing with extra latency.
  - rad_gruchalski 5y ago
    > Relational with ACID guarantees can't be implemented in a distributed system. CockroachDB and YugabyteDB would like to have a word.
    - slumdev 5y ago
      It's a causality problem, not a technology problem. Information can't be changed in two places simultaneously. That latency, no matter how small, requires us to either eat the latency, give up guarantees, or alter our system's behavior in some other small way.
      - rad_gruchalski 5y ago
        Of course. But thanks to raft per table, the queries are coordinated in such a way that to the observer it looks like acid.
k__ 5y ago
Aren't there quite some solutions today? Fauna, Upstash, DynamoDB, CosmoDB, Firestore, Planetscale.
icakir 5y ago
weird, nobody mention nuodb.
- erik_seaberg 5y ago
  At a glance, they don’t publish a price list. There might be a 500 GB trial version behind “You are not authorized to access this page.”
han_solo 5y ago
vitess on k8s gives u some nice options.. sure a little extra application logic but so what. you can easily break out of vitess sharding whereever its needed and you still get the benefit of k8s schuduling.
AtlasBarfed 5y ago
The issue with scaling the database is that SQL joins (generally) do not scale under CAP. So, alas, you either go SQL in the early stages and then need to do considerable engineering to down-convert to say, Cassandra or DynamoDB. Or you accept reduced database language sugar and complexity up-front (no joins, limited index/views, or architect with explicit sharding) with a more scalable database approach. There's basically no magic sauce for scalable SQL. Frankly most of the people telling you otherwise are selling varying degrees of snake oil. As people will point out, SQL databases on modern hardware scale pretty freaking large. So you can get a lot of mileage putting off the "true scalability refactor". The good news, by then you should know your queries and data that need full scaling, you aren't guessing ahead of time.
- haileys 5y ago
  SQL is just a language. Any techniques which can be used scalably (client-side joins for instance) could also be used by an implementation supporting the SQL language. Perhaps your argument holds for some of the more well-known RDBMSes out there, but I don't think SQL necessarily has to be unscalable in the general case.
  - AtlasBarfed 5y ago
    Let's take the general case of a join: two tables that are keyed/indexed by fundamentally different data. To distribute data, you have a data distribution scheme. Cassandra and various distributed hash maps (which is the typical approach) it is a consistent hash function. But even if you do distribution using natural ordering, the same problem exists: The data you are joining is going to be on different nodes on a row-by-row case. Hashing will produce this by the design of the hash function. Natural ordering will do this because different key datatypes will order differently. In the case of large scale distribution across a LOT of machines (which is what you invariably have to go to once you expend the options in big iron), that means a huge amount of network traffic, with each retrieval needing to be resolved for consistency due to if you want AP. If you rely on CP, then your join is dependent on SO MANY nodes correctly communicating that you become extremely exposed to network partitions, retries, etc. Thus you either shard your data so all data is on the same machine (but your joins are necessarily subsets of the overall data: only the shard), or you don't and prepare for extremely bad performance, unreliable performance, or approximations of correctness.
ksec 5y ago
>Each tier is either easy to reason about scaling out horizontally except for the database. Vitess [1], A database clustering system for horizontal scaling of MySQL, or Planetscale [2] which is the SaaS version. Of course everything is good on paper until you run into edge cases. But I am convinced within this decade scaling problem or hassle will be a thing of the past for 95% of us. [1] https://vitess.io https://vitess.io [2] https://www.planetscale.com https://www.planetscale.com
madhadron 5y ago
I feel like this misses the point. The reason all the other stuff besides the database is so straightforward is because you can localize all the state in the database so you can deal with the hard parts in one place and with maximum support. The database doesn't ruin all good ideas. It makes all those ideas for other tiers possible.
rad_gruchalski 5y ago
More people should know about YugabyteDB. Granted though, article talks about about „traditional RDBMS”.
xupybd 5y ago
To me this boils down to the hard part of concurrency is shared memory.
ofrzeta 5y ago
The article is confusing (confused?), mixing up different concepts like containerisation and active/active database configuration while not even going into detail but only mentioning concepts in passing by. I don't get the point.
coderintherye 5y ago
"I learned a ton about splitbrain mostly through trauma" The most typical way to learn about splitbrain it seems. I don't think I've experienced more work trauma than split brain trauma.
gregjor 5y ago
Article would benefit from an example or two of ideas ruined by databases. The examples given all have to do with implementation details, not ideas. Databases actually enable a lot of good ideas, and reliable well-understood implementations. Pointing out that database servers don’t scale horizontally like web servers had the same level of insight as pointing out cars don’t scale like kittens.
zaptheimpaler 5y ago
Databases are the reason all that other shit is so easy. Databases are the heart of computing but somehow this clown industry still doesn't get that at a large scale. I’ll replicate your clown ass todo list app with a few DB tables, a few queries and 1 HTML table dont make me fucking do it :D
- sonthonax 5y ago
  Also, developers never design their own database schemas anymore, then when the some crappy ORM (like Django’s) generates a tediously slow schema they blame the database. It’s weird how developers get asked about OO design patterns in interviews, but not once have I ever been asked about database design (beyond some useless stuff about 2nd normal form).
  - cornel_io 5y ago
    Huh, almost every lead backend job I've interviewed for had a lot of DB design questions, more than normal coding stuff because it's so critical to scaling. It's core to the role even for lower level positions, so it really should be more common.
    - nesarkvechnep 5y ago
      Clearly you are not a Node.is dev.
      - sonthonax 5y ago
        Or a Django dev, or a Rails dev, or any other ‘framework developer’.
        cornel_io 5y ago
        As a matter of fact, I have done both of those a lot, and I'd argue that knowing how model design interacts with the database is arguably more important because if what footguns the ORMs can be. But you are correct, interviews for those types of jobs for whatever reason don't tend to focus as much on DB stuff, though they really really should given that a quarter of your job will end up being unfucking the system away from the poor choices that your predecessors made (if you're guaranteed to be serving 1mm DAU and everyone will be generating thousands of records per month, maaaybe don't use normal ints for uids...).
rognjen 5y ago
I liked the article idea but I feel like it is two sort of related things in one. I'd have liked to learn more about why it's only one db. For instance because they're often insanely efficient and one server can handle many queries. I was also hoping that it'd suggest specific alternatives.
A1kmm 5y ago
Even without using NewSQL databases, it is often very easy to structure your application for application-level sharding. For example, for https://corridorchat.com/ https://corridorchat.com/, we have a relatively small number of business accounts (tenants), but with a many users per tenant. And new tenants are created relatively infrequently in the scheme of things. So I have an architecture with a central 'corridorchat central' PostgreSQL database and a scalable number of shard clusters, all managed with Patroni + Consul, and fall-backs that are read-only until they need to be promoted. Consul DNS allows the application to look up either a read-only replica or a write one. To know what shard a tenant is hosted on, it is necessary to read from the central database. This requires one of the read replicas - and I can create as many of these as I need. Many transactions then require writing to the shard database for that tenant - but since I balance tenants between shards (and have several shard databases per cluster to allow for future scaling if a shard becomes too hot) I can add more shard clusters as needed. Writing to the central database is constrained, but it is a very rare operation, so there is no expected scaling problem there.
- sa46 5y ago
  This is great until you have a tenant too large for a single shard. It’s pretty difficult to perfectly shard relational data. At some point you’ll run into a need for cross shards queries. I think for most workloads, this approach should work for a good long while.
ahachete 5y ago
Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertical scalability [1]. Do you need millions of transactions per second? Then go ahead. Don't? Then ou can scale very very far with a properly tuned and administered relational database like Postgres. [1]: https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/ https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/
- diminish 5y ago
  If the number of users, number of active users and all major tables grow 100x, would postgres be able to support the application, without major re-designs?
  - quonn 5y ago
    Presumably they would then have 100x the cashflow? Enough for a redesign?
  - jmchuster 5y ago
    At 100x, gitlab would be generating $15 billion in revenue each year. Having to redesign their database structure wouldn't be the thing that keeps them up at night. I'll also add, that the point is that a devtools unicorn is able to run with a single postgres cluster, so maybe it isn't the rdbms that's going to be the limiting factor on your startup with a hundred users.
- KronisLV 5y ago
  But what about when there will be problems with your single point of failure? Sometimes you'll run into situations where an instance of any software has degraded performance for reasons beyond either your control or understanding. That's one of the situations where horizontal scaling let's you deal with such circumstances better, instead of outright impacting everything and everyone connected to it. A front end web server container crumbles under the load? No worries, just redirect the traffic to others. A back end API container has problems with the JVM reserved memory and/or the thread pool for processing connections has been filled and all new incoming requests just get queued up? Just have more instances to cope with the load and don't let one instance under load affect all others. But what are you supposed to do if long running SQL exhausts the resources of your DBMS? What if a background process needs to run some really complicated writes, but your application needs write capability without slowing down? That's not to say that bad configurations or bad code shouldn't be addressed, but any means of fighting it and minimizing the impact is worthwhile in my eyes.
satyanash 5y ago
Isn't that like saying reality ruins all good ideas. :)
cobertos 5y ago
> In addition, you have footguns with non-obvious behavior around [Docker] volumes. Anyone know what these are?
- aflag 5y ago
  He may be referring to overlayfs which has had its share of bugs along the years.
pkstn 5y ago
Writer clearly haven't heard about MongoDB :P
majewsky 5y ago
> Or maybe you use sqlite in dev/tests and something bigger in prod. I had that for a while. SQLite for the test suite and Postgres for actual runs. It was a shitshow, having to support two separate SQL dialects with separate feature sets. Then I realized that you can actually bootstrap and start a Postgres in tmpfs in about one second, so that's what my tests do nowadays.
- sebazzz 5y ago
  Nice, does MSSQL have something similar? I know there is localDB, but that has its quirks and databases need to be cleaned up manually.

251 points by kristianp 5y ago

Am I just dense, or was the idea the CEO had just not a good idea and simply garden variety premature optimization? Also, it seems unfair to say that the database ruins all good ideas when there continues to be significant impressive innovation in the space in cloud services. Like Amazon Aurora for one. If you really want to treat your RDBMS like an elastically scalable (with significant caveats) resource priced by the request, you can.

justsomeuser 5y ago
I think the title implicitly meant “good”, as in “you think it’s a good idea until you press it against the grindstone that is reality”. Also I think these issues arise at the database because that’s were the write concurrency is pushed.

thrill 5y ago

Use CockroachDB.

AdrianB1 5y ago
To solve what problem?
- rad_gruchalski 5y ago
  To get replication and horizontal scalability for free (well, within the license fee). But I’d go for YugabyteDB instead of CockroachDB.
bobthebuilders 5y ago
CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC which contirbutes to GC pauses. The dev team refuses to listen to feedback to port their code to C ;(.
- Guvante 5y ago
  Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument when it comes to any production system that it isn't even worth discussing. The reality is there are limited resources to work on any given project and "rewrite" is generally not the correct way to fix a given problem. Especially since the first thing you are going to need to do is show that a network system isn't resilient to GC pauses or that GC pauses are frequent enough to impact throughput (very very few applications are actually materially impacted by sub second spikes in latency).
  - pkolaczk 5y ago
    GC pauses are not the only GC issue. Thrashing the caches and blowing up memory usage by an order of magnitude can be also very bad for performance. In a database system memory is very precious - the more of it you can use for caching / buffering users data, the better the performance. As for the subsecond spikes in latency, these tend to multiply in a distributed system. If serving a client request takes N internal requests, the likelihood of hitting a GC pause somewhere is much larger than if you did only one local request. Not sure about Go, but none of the "free" Java's GC guarantees low pauses. There is STW fallback even in the most advanced ones like ZGC. So you never know when it stops for more than 1s.
    - ngrilly 5y ago
      Go GC pauses are bounded at 0.5 ms: https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
      - pkolaczk 5y ago
        Is it a hard guarantee or just a soft goal like in G1, which tries to stay within the target pause milliseconds, but there are many ways it can fail to? Can it handle 100+ GB large heaps? Even if it is a hard guarantee, then, from the link you posted, it is not even generational, so it will scan the whole heap quite frequently, and that is going to influence average performance quite visibly - you definitely dont want a database system to access all its cached memory once in a while. Database systems are really all about memory and I/O management. You shouldn't outsource those core features to a universal algorithm, unless you wish to forgo any competitive advantage (at least in performance department). So this pushes the devs into the off-heap manual memory management territory, where dragons live (at least in Java, again - maybe Go is better in that regard). I've been there, and I don't recommend.

hughrr 5y ago

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider. The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. If you spent the last decade writing shitty cross database SQL then you’ll have to silo by customer. That’s the only real constraint.

tremon 5y ago
I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good". Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without understanding the problem space is a disaster waiting to happen (but luckily, it may not happen on your watch so you can continue hacking).
- hughrr 5y ago
  Very good points there. I have directly experienced the problems associated with blindly assuming that you can write your own consistency guarantees in Redis. I didn't do it but I had to write some tooling to rewrite the AOF to get some data back...
- HWR_14 5y ago
  I'm sorry, are you saying that a good idea should survive the loss of referential integrity or data consistency? I don't feel that's what you mean, and it's probably my fault for misreading your comment.
  - erik_seaberg 5y ago
    I read it the other way: if abandoning correctness is the only way to make your idea feasible, your idea kinda sucks.

jarym 5y ago

The mistake is thinking half the system is doing nothing. It’s a bit like saying I’m paying to store my backups offsite but most of the time it’s not really used so it’s a waste of money. It just doesn’t hold. Most SQL problems that most of us have to deal with stem from inadequate indexes and/or poorly written queries. No fancy active-active setup will ever solve those issues.

jayd16 5y ago
It might be a waste. Think RAID 1 vs RAID 5. There might be plenty of ways were leaving performance on the table.
- jeltz 5y ago
  Which one of those leaves performance on the table? One is optimized for performance and the other for disk usage.
  - jayd16 5y ago
    Maybe its a bit tortured of a metaphor, but perhaps with continued research we can find a way to use more of the hardware just like how RAID 5 provides more storage over RAID 1 while still keeping parity. I don't think its foolish to keep asking for more. We still might find it.

renewiltord 5y ago

Can’t be bothered. Aurora in Postgres/MySQL and then just scale that vertically till it stops.

rrdharan 5y ago

If you really want or need to go active-active but also don’t want to think about it much and can live with MySQL my preference would probably be AWS Aurora multi-master if you can live with its limitations: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-multi-master.html#aurora-multi-master-limitations https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide... As noted there and elsewhere though often you are better off just scaling up. If you do want to really think about it then NewSQL (Cloud Spanner or others) could be the way to go but the cost, performance, compatibility/portability and other concerns make this a much more complex decision that isn’t easy to capture in a blog post (or HN comment ;).

trollied 5y ago

The article is talking about Oracle RAC, and the "crossover" he is talking about was usually really quick Infiniband, not a crossover cable (so, a separate NIC for good reason!). The person that wrote the article doesn't seem to understand RAC enough to actually comment properly. A properly tuned RAC instance will scale horizontally very very well.

don-code 5y ago
Former (recovering?) Exadata user here. I'll go as far as to say that configuring a best-practices RAC cluster is easier than configuring a best-practices MySQL cluster. RAC solves some additional problems, like discovery and load balancing, that MySQL kicks out to the user (e.g. you have to manage DNS and run a tool like ProxySQL separately).
inopinatus 5y ago
If you look back far enough, it wasn’t infiniband but a small separate fibrechannel fabric, and further back again I recall buying specialist dual-ported SCSI drives that were marketed primarily for cluster i.e. quorum-disk purposes. More than one goat was required to make these configurations work. (I seem to recall that many SAS drives now are dual-ported but designed as such to fulfil a desire for multi-path resiliency within storage arrays; this may have different characteristics to those needed of a quorum device)
hestefisk 5y ago
I’ve seen Oracle RAC in use in many banks in particular where uptime and resilience are critical requirements. It scales well horizontally, but at a very high cost.
cratermoon 5y ago
How much redundancy do you really have if both (or all N) systems are within 10km of each other?
Swannie 5y ago
Alternatively a private L2 network for a N > 2 RAC cluster.

edelweis 5y ago

Databases scale just fine. Its called sharding. Azure has HyperScale (with is Citus), AWS has Aurora. The actual problem with databases is upgrading to major versions without downtime. All the other problems are just noise

sonthonax 5y ago
Agreed. But one impediment to sharding is bad DB design, which is rife right now because everyone starts with Django/Rails/Laravel and starts treating the DB like a document store that happens to have some relational features.

robertlagrant 5y ago

I think that being able to have separate databases for different domains is pretty good, although it does rely partly on the application layer to keep (UU)IDs consistent. I'd be curious to know if there's a way to have databases talk to each other to just sync up primary keys for referential integrity. That could maximise the benefit of decoupled databases while still having good referential integrity. And a network disconnection would still mean existing PKs would be in place, it just wouldn't be aware of new ones. Not perfect, but not bad, perhaps.

cratermoon 5y ago
https://lamport.azurewebsites.net/pubs/paxos-simple.pdf https://lamport.azurewebsites.net/pubs/paxos-simple.pdf

AdrianB1 5y ago

A dockerized database server is something I cannot understand; I understand bare metal, I can accept virtualized, but I cannot find a good used case for a mid sized or large server (dozens of gigabytes to dozens of terabytes) dockerized and I don't know why a smaller server is a problem.

kevml 5y ago
A docker image is simply packaging. There’s a small, almost always negligible, performance hit. But the value comes in being able to ship a consistent application and configuration pairing. This becomes really valuable when shipping multiple databases for multiple services.
- AdrianB1 5y ago
  I know what docker does, I never saw a database server deployed in mass. One of my teams manage over 250 database servers, but nobody ever supported the idea of using docker for that (all the servers run on Windows). Shipping multiple databases for multiple services does not necessarily mean multiple database servers. In the database world I saw large servers with many databases more often than multiple servers of one database each; in the second case it was the vendor laziness (cannot give the name), not than a reasonable business or technical reason. When I asked about it the answer was "we'll consolidate in the next release".
  - kevml 5y ago
    Sure there’s a lot of variables here. If you’re a Windows shop I would not expect a push for docker since it is Linux based. As far as multi-tenant logical databases, that is also highly dependent on workload and failure domains (and significant cost if you’re talking Oracle).
majewsky 5y ago
For our team, the reason is standardization. We have our entire baremetal infrastructure running Kubernetes. There is no process for running anything in not-a-Docker-container. This does come with some caveats for stateful databases, but it's still easier to manage than a special-snowflake environment for database deployments.

l0k3ndr 5y ago

I have been working on rewriting a monolith into individual services during past year at work - and what we finally implemented for consistency across different databases of services was that we keep a signal queue - where whenever we encounter an inconsistency, a doc is pushed containing the info and we have a corrective service always reading from that queue and doing the necessary updates to tables. We made a heirarchy of source-of-truthness and we use that to decide what to do with inconsistent data. Though, users sometimes have to see a error messagge "We had an error and we are fixing it in few XXX time" based on current queue load - but it has been working fine mostly for us.

danielheath 5y ago
… is that less operational overhead than running a single process on a server?
- deleted 5y ago
  [deleted]
aflag 5y ago
Why was that change made? From my understanding users are constantly seeing outages now where they see an error message and have to wait. Were there more outages before? How was the previous system?

zzzeek 5y ago

just because it's the thing I know best at the moment, it's worth checking out Galera which actually does do multi-master replication. Of course you have to live in MySQL/MariaDB world which is a non-starter for a lot of peeps.

samatman 5y ago

SQLite is a remarkably good solution to most of these problems, if deployed correctly. For your main line-of-business database? Of course not. But a deployment of rqlite[0] for your service workers in a read-heavy workload? cuts out a round-trip out of the VM, mocking is trivial, there's a lot to like there. [0]: https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite

eloff 5y ago
You're missing the whole point of the article. If you set this up in an active-active configuration (if that's even supported) you have the exact same distributed systems problems as the author.
cortesoft 5y ago
The whole point of having a database is that you need persistent data that is consistent for all clients. You can’t have a bunch of SQLite dbs and get that.
otoolep 5y ago
rqlite author here. Happy to answer any questions about it.

simonw 5y ago

The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed a write have their read traffic sent to the leader for the next 10 seconds or so. Not trivial, but also not impossibly difficult - plenty of places implement this without too much trouble. Scaling writes is a lot harder - but a well specc'd relational database server will handle tens of thousands of writes per second, so the vast majority of projects will never have to solve this. When you do need to solve this, patterns for horizontally sharding your data exist. They're not at all easy to implement, but it's not an impossible problem either. The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box). Overall, my experience is that the database /enables/ all good ideas. Building stateful applications without a relational database in the mix is usually the wrong choice.

tremon 5y ago
horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas There's one feature that I'd like to see in that area: partitioned writable replicas. In the same vein that you can partition the table storage across an index, I'd like it to be possible to assign different writers to different parts of a table/database. Of course, you'd still need a single primary replica to handle the transactions that transcend the configured partitions, but we already have routing engines that can transparently redirect an incoming query to any available replica, so it's partly there. There's probably corner cases lurking that I can't even think of, but in my mind it's the only thing missing from building a truly web-scale (multiple zones, multiple datacenters) ACID-preserving relational database.
- taffer 5y ago
  This is essentially the idea of VoltDB [1]: All tables are partitioned and each partition is assigned to a single thread. As long as each query only touches a single partition, you get linear horizontal scalability and serializable isolation. The drawback is that if a query touches multiple partitions, only that one query can be executed on the entire cluster, so this architecture only works for very specific OLTP workloads. [1] https://www.voltdb.com/ https://www.voltdb.com/

benlivengood 5y ago

And this is why Google wrote Spanner. I think cockroachdb tries to solve the same problems. If you need ACID compliance and you need a lot of it, everywhere, all the time, now there are better options than giant Sun/IBM boxes. Databases are not the problem.

tyingq 5y ago
It's still a fairly new thing for people outside of Google. It's been a few years, so I imagine it's better now...but when I tried CockroachDB, it was pretty easy to trigger horrible write latency. Bad enough to be a non-starter.
- namibj 5y ago
  Which version, and what networking did you have between the cluster nodes?
  - tyingq 5y ago
    I don't recall other than it was before 2.x, but I don't think it's terribly important. The cockroachdb folks were very transparent about it, including a couple of major re-works to address performance. Here's a blog post about one of them: https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ I was just trying to say that a stable, high performance SQL database with distributed writes is a fairly new thing.
CRConrad 5y ago
> If you need ACID compliance and you need a lot of it, everywhere, all the time, now there are better options than giant Sun/IBM boxes. Oh I dunno, I'd still say most claims to be "better" at data integrity with performance, and a lot of it than a good old zOS mainframe with a soundly designed DB2 database on it are [Citation needed]-class... But yeah, databases certainly aren't the problem here.

da39a3ee 5y ago

I’m not convinced the author knows what they’re talking about. The answer to the question they posed is basically “ACID”, so unclear what all the verbiage is. Also unclear why they expect all readers to have an architecture featuring multiple application servers all sharing a single database, since that is neither classic monolith nor classic microservices.

slumdev 5y ago
I get the same impression. Relational with ACID guarantees can't be implemented in a distributed system. Depending on which guarantees you want to give up, you might be able to get close. A lot of newer platforms give up guarantees to achieve something like this. But if you have multiple nodes that have to sync every transaction over a dedicated NIC, well, that's not a distributed system. It's just multiprocessing with extra latency.
- rad_gruchalski 5y ago
  > Relational with ACID guarantees can't be implemented in a distributed system. CockroachDB and YugabyteDB would like to have a word.
  - slumdev 5y ago
    It's a causality problem, not a technology problem. Information can't be changed in two places simultaneously. That latency, no matter how small, requires us to either eat the latency, give up guarantees, or alter our system's behavior in some other small way.
    - rad_gruchalski 5y ago
      Of course. But thanks to raft per table, the queries are coordinated in such a way that to the observer it looks like acid.

k__ 5y ago

Aren't there quite some solutions today? Fauna, Upstash, DynamoDB, CosmoDB, Firestore, Planetscale.

icakir 5y ago

weird, nobody mention nuodb.

erik_seaberg 5y ago
At a glance, they don’t publish a price list. There might be a 500 GB trial version behind “You are not authorized to access this page.”

han_solo 5y ago

vitess on k8s gives u some nice options.. sure a little extra application logic but so what. you can easily break out of vitess sharding whereever its needed and you still get the benefit of k8s schuduling.

AtlasBarfed 5y ago

The issue with scaling the database is that SQL joins (generally) do not scale under CAP. So, alas, you either go SQL in the early stages and then need to do considerable engineering to down-convert to say, Cassandra or DynamoDB. Or you accept reduced database language sugar and complexity up-front (no joins, limited index/views, or architect with explicit sharding) with a more scalable database approach. There's basically no magic sauce for scalable SQL. Frankly most of the people telling you otherwise are selling varying degrees of snake oil. As people will point out, SQL databases on modern hardware scale pretty freaking large. So you can get a lot of mileage putting off the "true scalability refactor". The good news, by then you should know your queries and data that need full scaling, you aren't guessing ahead of time.

haileys 5y ago
SQL is just a language. Any techniques which can be used scalably (client-side joins for instance) could also be used by an implementation supporting the SQL language. Perhaps your argument holds for some of the more well-known RDBMSes out there, but I don't think SQL necessarily has to be unscalable in the general case.
- AtlasBarfed 5y ago
  Let's take the general case of a join: two tables that are keyed/indexed by fundamentally different data. To distribute data, you have a data distribution scheme. Cassandra and various distributed hash maps (which is the typical approach) it is a consistent hash function. But even if you do distribution using natural ordering, the same problem exists: The data you are joining is going to be on different nodes on a row-by-row case. Hashing will produce this by the design of the hash function. Natural ordering will do this because different key datatypes will order differently. In the case of large scale distribution across a LOT of machines (which is what you invariably have to go to once you expend the options in big iron), that means a huge amount of network traffic, with each retrieval needing to be resolved for consistency due to if you want AP. If you rely on CP, then your join is dependent on SO MANY nodes correctly communicating that you become extremely exposed to network partitions, retries, etc. Thus you either shard your data so all data is on the same machine (but your joins are necessarily subsets of the overall data: only the shard), or you don't and prepare for extremely bad performance, unreliable performance, or approximations of correctness.

ksec 5y ago

>Each tier is either easy to reason about scaling out horizontally except for the database. Vitess [1], A database clustering system for horizontal scaling of MySQL, or Planetscale [2] which is the SaaS version. Of course everything is good on paper until you run into edge cases. But I am convinced within this decade scaling problem or hassle will be a thing of the past for 95% of us. [1] https://vitess.io https://vitess.io [2] https://www.planetscale.com https://www.planetscale.com

madhadron 5y ago

I feel like this misses the point. The reason all the other stuff besides the database is so straightforward is because you can localize all the state in the database so you can deal with the hard parts in one place and with maximum support. The database doesn't ruin all good ideas. It makes all those ideas for other tiers possible.

rad_gruchalski 5y ago

More people should know about YugabyteDB. Granted though, article talks about about „traditional RDBMS”.

xupybd 5y ago

To me this boils down to the hard part of concurrency is shared memory.

ofrzeta 5y ago

The article is confusing (confused?), mixing up different concepts like containerisation and active/active database configuration while not even going into detail but only mentioning concepts in passing by. I don't get the point.

coderintherye 5y ago

"I learned a ton about splitbrain mostly through trauma" The most typical way to learn about splitbrain it seems. I don't think I've experienced more work trauma than split brain trauma.

gregjor 5y ago

Article would benefit from an example or two of ideas ruined by databases. The examples given all have to do with implementation details, not ideas. Databases actually enable a lot of good ideas, and reliable well-understood implementations. Pointing out that database servers don’t scale horizontally like web servers had the same level of insight as pointing out cars don’t scale like kittens.

zaptheimpaler 5y ago

Databases are the reason all that other shit is so easy. Databases are the heart of computing but somehow this clown industry still doesn't get that at a large scale. I’ll replicate your clown ass todo list app with a few DB tables, a few queries and 1 HTML table dont make me fucking do it :D

sonthonax 5y ago
Also, developers never design their own database schemas anymore, then when the some crappy ORM (like Django’s) generates a tediously slow schema they blame the database. It’s weird how developers get asked about OO design patterns in interviews, but not once have I ever been asked about database design (beyond some useless stuff about 2nd normal form).
- cornel_io 5y ago
  Huh, almost every lead backend job I've interviewed for had a lot of DB design questions, more than normal coding stuff because it's so critical to scaling. It's core to the role even for lower level positions, so it really should be more common.
  - nesarkvechnep 5y ago
    Clearly you are not a Node.is dev.
    - sonthonax 5y ago
      Or a Django dev, or a Rails dev, or any other ‘framework developer’.
      - cornel_io 5y ago
        As a matter of fact, I have done both of those a lot, and I'd argue that knowing how model design interacts with the database is arguably more important because if what footguns the ORMs can be. But you are correct, interviews for those types of jobs for whatever reason don't tend to focus as much on DB stuff, though they really really should given that a quarter of your job will end up being unfucking the system away from the poor choices that your predecessors made (if you're guaranteed to be serving 1mm DAU and everyone will be generating thousands of records per month, maaaybe don't use normal ints for uids...).

rognjen 5y ago

I liked the article idea but I feel like it is two sort of related things in one. I'd have liked to learn more about why it's only one db. For instance because they're often insanely efficient and one server can handle many queries. I was also hoping that it'd suggest specific alternatives.

A1kmm 5y ago

Even without using NewSQL databases, it is often very easy to structure your application for application-level sharding. For example, for https://corridorchat.com/ https://corridorchat.com/, we have a relatively small number of business accounts (tenants), but with a many users per tenant. And new tenants are created relatively infrequently in the scheme of things. So I have an architecture with a central 'corridorchat central' PostgreSQL database and a scalable number of shard clusters, all managed with Patroni + Consul, and fall-backs that are read-only until they need to be promoted. Consul DNS allows the application to look up either a read-only replica or a write one. To know what shard a tenant is hosted on, it is necessary to read from the central database. This requires one of the read replicas - and I can create as many of these as I need. Many transactions then require writing to the shard database for that tenant - but since I balance tenants between shards (and have several shard databases per cluster to allow for future scaling if a shard becomes too hot) I can add more shard clusters as needed. Writing to the central database is constrained, but it is a very rare operation, so there is no expected scaling problem there.

sa46 5y ago
This is great until you have a tenant too large for a single shard. It’s pretty difficult to perfectly shard relational data. At some point you’ll run into a need for cross shards queries. I think for most workloads, this approach should work for a good long while.

ahachete 5y ago

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertical scalability [1]. Do you need millions of transactions per second? Then go ahead. Don't? Then ou can scale very very far with a properly tuned and administered relational database like Postgres. [1]: https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/ https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/

diminish 5y ago
If the number of users, number of active users and all major tables grow 100x, would postgres be able to support the application, without major re-designs?
- quonn 5y ago
  Presumably they would then have 100x the cashflow? Enough for a redesign?
- jmchuster 5y ago
  At 100x, gitlab would be generating $15 billion in revenue each year. Having to redesign their database structure wouldn't be the thing that keeps them up at night. I'll also add, that the point is that a devtools unicorn is able to run with a single postgres cluster, so maybe it isn't the rdbms that's going to be the limiting factor on your startup with a hundred users.
KronisLV 5y ago
But what about when there will be problems with your single point of failure? Sometimes you'll run into situations where an instance of any software has degraded performance for reasons beyond either your control or understanding. That's one of the situations where horizontal scaling let's you deal with such circumstances better, instead of outright impacting everything and everyone connected to it. A front end web server container crumbles under the load? No worries, just redirect the traffic to others. A back end API container has problems with the JVM reserved memory and/or the thread pool for processing connections has been filled and all new incoming requests just get queued up? Just have more instances to cope with the load and don't let one instance under load affect all others. But what are you supposed to do if long running SQL exhausts the resources of your DBMS? What if a background process needs to run some really complicated writes, but your application needs write capability without slowing down? That's not to say that bad configurations or bad code shouldn't be addressed, but any means of fighting it and minimizing the impact is worthwhile in my eyes.

satyanash 5y ago

Isn't that like saying reality ruins all good ideas. :)

cobertos 5y ago

> In addition, you have footguns with non-obvious behavior around [Docker] volumes. Anyone know what these are?

aflag 5y ago
He may be referring to overlayfs which has had its share of bugs along the years.

pkstn 5y ago

Writer clearly haven't heard about MongoDB :P

majewsky 5y ago

> Or maybe you use sqlite in dev/tests and something bigger in prod. I had that for a while. SQLite for the test suite and Postgres for actual runs. It was a shitshow, having to support two separate SQL dialects with separate feature sets. Then I realized that you can actually bootstrap and start a Postgres in tmpfs in about one second, so that's what my tests do nowadays.

sebazzz 5y ago
Nice, does MSSQL have something similar? I know there is localDB, but that has its quirks and databases need to be cleaned up manually.