7 ms·
Anyone starting a tech startup will face this decision: Should I start with a database that has built-in support for sharding? Or should I just start with MySQL
by flowerlad 5y ago
Anyone starting a tech startup will face this decision: Should I start with a database that has built-in support for sharding? Or should I just start with MySQL or PostgreSQL and defer the sharding question for later?
Notion chose to do manual sharding (aka application-level sharding). That's what you end up doing if you didn't choose a database that has built-in sharding from the get go, because it is extremely hard to switch to a different database technology. (Larry Ellison compares it to a catholic marriage -- there is no divorce!)
I skimmed through the article to find the critical piece of info I was looking for: rationale for doing manual sharding. The rationale supplied in this article is "we wanted control over the distribution of our data." That's a weak explanation. It's the kind of thing you say to justify the bad choice made earlier on: failure to choose a database that supports automatic sharding from the get go.
- nosefrog 5y agoWhat open source databases support automatic sharding? I think their only other option would have been something like cockroachdb, which probably wasn't very mature when notion started.
- flowerlad 5y agoCheck out Implementations section on this page: https://en.wikipedia.org/wiki/Shard_(database_architecture) https://en.wikipedia.org/wiki/Shard_(database_architecture)
- zozbot234 5y agoPostgreSQL supports native sharding out of the box. It's "manual" in that you have to set up table partitions and foreign data access yourself, but that's arguably still preferable to hacking together the whole thing as a pure application-layer concern.
- turbocon 5y agoIt was my understanding that PostgreSQL supports table sharding out of the box on a single instance but not the ability to do distribute the processing across multiple instances.
- qaq 5y agoPostgres supports FDW. Also you can use Aurora Postgres in AWS which can take you fairly far. Another option for many SaaS products you can simply run multiple clusters and assign a given tenant to a particular cluster.
- bsder 5y agoFrom the article: > By mid-2020, it was clear that product usage would surpass the abilities of our trusty Postgres monolith, which had served us dutifully through five years and four orders of magnitude of growth. This should answer your question. Just use a standard database and get on with coding features instead. If you are successful enough to get to the point that sharding PostgreSQL becomes your bottleneck, you've won. In addition, you really can't solve scaling problems up front. Where your bottleneck actually occurs will differ from where you think it will.
- flowerlad 5y ago> Just use a standard database and get on with coding features instead. Better still, choose a "standard database" that supports sharding.
- rtpg 5y agoSerious question: Let's say I'm using Django or Rails, and I just want to write the code with the ORM. What database am I using that you think makes sense? I would see an argument that Notion's structure should have made this problem more important for them to resolve earlier in the process (like some document-based DB), but Postgres works well!
- jamesfinlayson 5y agoProbably CockroachDB - I'm not sure what other free/open source options there are. I'm sure some proprietary databases will do it too but I haven't used Oracle/MSSQL/whatever to know what they give you.
- flowerlad 5y agoAre you an IT developer, who is developing applications for internal use? If so PostgreSQL is an excellent choice! But if you're developing an app that scales to internet users consider Couchbase or some other database that matches your requirements.