15 ms·

Lessons learned from sharding Postgres at Notion

471 points by craigkerstiens 5y ago

louwrentius 5y ago
I still wonder so hard what a single bare metal server would have done in terms of performance. [0] May have saved months of planning that may never have been required. [0]: https://yourdatafitsinram.net/ https://yourdatafitsinram.net/
- 0xbadcafebee 5y ago
  This can quickly become untenable depending on the availability and contractual requirements you have. You're literally building on a single point of failure. And if it's really big and you design wrong, recovery could take weeks.
  - louwrentius 5y ago
    Put a slave next to it, that's not the point. AFAIK this is what stackoverflow still does and it works for them. Why not for notion?
    - notJim 5y ago
      Notion may have a much higher proportion of write volume than stackoverflow, given that most of stackoverflow's traffic must be from people who come in from Google, read the answers, and leave. Notion is productivity app, so I would guess most people are actively writing.
    - 0xbadcafebee 5y ago
      If StackOverflow jumped off a bridge, would you jump off one too? You can't base a huge decision like that on "well it worked for somebody else". How did it work for them? What was their platform? What was their data profile? What were their requirements? What was their acceptable level of risk? It could be that they have so many caching layers and lose so little money that they are fine with the database being down for 5 days while they recover. Or that it doesn't actually work that fine for them, because who wants to broadcast that their system is shitty? Or that they've just been lucky. It's not worth finding out the hard way. Build the best system you can with the time and money and expertise you have. Don't cheap out just because you think you can get away with it - especially on the critical stuff.
- sa46 5y ago
  You'll hit all sorts of other fun problems by vertically scaling. - Many concurrent connections to a single database which Postgres has traditionally not handled well (though improved in recent versions). - You're now on the hook for writing a database control plane. - Backup and restore is much flakier since the data volume is so much larger. Lots of weird shit starts happening when you download tens of TB. - In general, everything is a harder once you start nearing machine limits.
  - jamesfinlayson 5y ago
    Yes I thought Postgres didn't recommend more than 700 connections per server (or maybe it was 300) and that was only three or four years ago.
endisneigh 5y ago
On a related note, how far can a single Postgres instance get you? If you had some behemoth with 32TB ram and 1PB storage could all of motion fit? I’m obviously ignoring the obvious single point of failure here, but sometimes the simplicity could be worth it if you’re willing to handle that. I’d be curious to hear about a site that’s in the Alexa 1000 architected how I describe.
- aeyes 5y ago
  > If you had some behemoth with 32TB ram and 1PB storage could all of motion fit? Postgres has a bunch of low level locks and buffers (protected by locks) which are essentially single threaded. So even if you had a 500 CPU instance at some point you'd not be able to get more throughput out of it. Of course it also depends on what you are doing. Large tables with high update rate are harder to handle than large tables which are insert only. My personal opinion is that Postgres tables with >100GB data (without indexes) are a starting to be a pain to work with, no matter how much RAM, CPU or IOPS you have.
  - endisneigh 5y ago
    Is there a relational database that could efficiently leverage such a single powerful machine?
    - gfody 5y ago
      sql server scales incredibly well with the hardware but so does its price
    - haneefmubarak 5y ago
      I imagine you want NUMA aware software specifically designed for multi-socket systems; off the top of my head the ones I've heard of are IBM DB2, SAP HANA, and Microsoft SQL Server — although I've had a chance to actually use any of them.
      - jiggawatts 5y ago
alberth 5y ago
>> “Why 480 shards? … 480 is divisible by a lot of numbers”. This is an important point, doing things divisible by 12 gives you a lot of flexibility. It’s not a coincidence both time (clocks) and degrees (360) are multiples of 12.
- infogulch 5y ago
  Numberphile made a video about "highly composite numbers" / "anti-primes", which are numbers that have more factors than all smaller numbers. https://youtu.be/2JM2oImb9Qg https://youtu.be/2JM2oImb9Qg There is an oeis sequence of them that starts "1, 2, 4, 6, 12, 24, 36, 48, 60, 120, 180, 240, 360, 720, 840...", which notably does not include 480. https://oeis.org/A002182 https://oeis.org/A002182
  - moralestapia 5y ago
    This is the kind of engineering that it's almost impossible to screen in interviews (particular for HR-minded pals) and is a big part of what makes a 10x-100x engineer (which definitely DO exist). Obviously not this thing just by itself, but this sort of knowledge, applied to everyday decisions that get compound over time.
  - mikepurvis 5y ago
    But 480 is 2*240, so it's definitely a close contender.
  - alberth 5y ago
    That’s super interesting. It’s counterintuitive but 480 has the same # of factors that 360 has (which is 24 factors) https://www.calculatorsoup.com/calculators/math/factors.php https://www.calculatorsoup.com/calculators/math/factors.php
    - _v7gu 5y ago
      so it's weakly highly composite?
bamazizi 5y ago
Curious about thought on using something totally different like column oriented db, like ClickHouse, would've suited a block data oriented app like Notion better? Does anyone with experience have any thoughts in favour or against such implementation?
- akarki15 5y ago
  +1. First thought i had when i saw this was like why is notion using a postgresdb in first place- i can see it being used for customer data (which probably doesn’t require sharding) but for the content itself, a nosql or colum oriented db is better fit. the question i have is what is notion storing in these sharded postgresdb?
- dilyevsky 5y ago
  Totally different usecase with clickhouse - the point queries there still slower than citus but it kicks its ass on multi row aggregations. For auto-sharded oltp db cockroachdb or tidb seem like way better options
capableweb 5y ago
Many companies I worked with stopped working with Notion in the past because of performance problems, the application was simply too slow. Has this changed with the new sharded architecture, as I haven't used it since probably last year?
- methyl 5y ago
  I can notice the performance no longer being the problem for me. The only gripe I have with Notion is too much flexibility, it’s easy to completely mess something up even by accident.
  - sergiotapia 5y ago
    Agreed, our Notion ended up being a soup of disparate outdated notes all over the place with no rhyme or reason across a 480+ employee business. I'm founding engineer of a new startup now and I advocated for Guru. https://www.getguru.com/ https://www.getguru.com/ I like how information is marked stale/verified and the permission are much tighter. The data ladder is also better structured. I'll see how it works as our headcount increases.
  - lucis 5y ago
    You mean editing documents you're just taking a look? Notion has a lock feature [1] on a document-level. [1] - https://www.notion.so/Lock-page-content-d2b995727c0b483f9f3518c1eb4fab75 https://www.notion.so/Lock-page-content-d2b995727c0b483f9f35...
- tempest_ 5y ago
  The app is a lot more snappy, not super fast, but usable. Search is still pretty slow.
- alberth 5y ago
  Curious, what app have you seen people migrate too?
- hinkley 5y ago
  From personal experience (not with Notion) once you fall behind it’s hard to catch up, and nobody declares an emergency until it’s already happening. Doing big surgery takes time, meanwhile your workload continues to grow and grow. It’s not uncommon for a major win to only reset the clock by 6-12 months, and then you either have to run to the next one, or you have two teams working on separate angles at the same time. All the while people are learning as they go because the business settled on catch-up instead of capacity planning. Being a little successful can be tough.
craigkerstiens 5y ago
This is a great read and primer for anyone looking to shard and know they need scale. Having helped a ton of people shard their data, my advice still remains don't do it until you know you have to. The old Citus data blog is a great resource for some guidance around best practices and how to's even if not using Citus. That said some of the things you can do from day 1 make things way easier when the time comes later. Workspace ID/Tenant ID/Customer ID, having your data grouped that way makes it very shardable and saves you a lot of work later on in the case you didn't have that in your data model. I'm not sure I buy that Citus/Vitess are magic, both are reasonably clear how they work and you can dig into it. At the same time I'd weigh the downsides of Citus (can't speak to Vitess) in that the online rebalancer isn't open source so at that point it's a proprietary product.
- alberth 5y ago
  >>” Workspace ID/Tenant ID/Customer ID, having your data grouped that way makes it very shardable” Dumb question, if someone starts off by grouping their data only by Customer ID and then later needs to shard. Couldn’t all of the sharding problems go away if they simply created a new Customer_Tenant table to map customer_id into tenant_group_id
  - craigkerstiens 5y ago
    Sorry, wasn't clear enough, tenant ID = customer ID = workspace ID. There is often a benefit though to materializing that ID onto all child tables. If you take a salesforce.com example, every customer has an "orgID", from there have accounts and accounts have opportunities. It's not common for people to put the "orgID" on the opportunity from a data modeling perspective, but when it comes to sharding it gives you a big win. If you have that tenant discriminator on all tables then it's easy to route it to the appropriate physical and logical shard right away vs. having to do some DB join first in your request to figure out where it goes.
    - alberth 5y ago
      Are you suggesting creating entire database schemas specific to a customer and put the customer/orgID in all table names themselves?
theodric 5y ago
I sharded in my pance
garrettf 5y ago
Howdy all, author here! Sharding our Postgres monolith was a huge undertaking for a small team. We got a huge boost from folks that joined within weeks before the switch-over date. If you’re interested in thinking about problems like this, I’d love to chat. Plus we’re also hiring across all roles—check my profile for details. I’m happy to answer questions about the project here, feel free to reply below.
- silentsea90 5y ago
  > [2] In addition to packaged solutions, we considered a number of alternatives: switching to another database system such as DynamoDB (deemed too risky for our use case).. 1) Can you please talk more about the risks of using DynamoDB or a similar NoSQL solution? 2) Did you consider Spanner, which is the SQL DB of choice within Google and is available on Google Cloud. Thanks for the wonderful engineering blog posts!
  - garrettf 5y ago
    Oh yes—this footnote is not knocking non-relational DBs, in fact we have deployed DynamoDB in production for other use cases today. We were on a tight timeline due to impending TXID wraparound. Switching database technologies would have required us to rewrite our queries, reexamine all of our indexes, and then validate that queries were both correct and performant on the new database. Even if we had the time to derisk those concerns, we'd be moving our most critical data from a system with scaling thresholds and failure modes we understand to a system we are less familiar with. Generally, we were already familiar with the performance characteristics of Postgres, and could leverage a decent amount of monitoring and tooling we built atop it. There is nothing inherent about non-relational DBs that make them unsuitable for our workload. If we were designing our data storage architecture from scratch, we'd consider databases that are optimized for our heaviest queries (vs. the flexibility afforded by Postgres today). A large number of those are simple key-value lookups, and a plain key/value store like DynamoDB is great for that. We're considering these alternatives going forward, especially as we optimize specific user workloads. Re: Cloud Spanner: we didn't consider a cross-cloud migration at the time due to the same time constraints. Still sounds like a wonderful product, we were just not ready at the time.
flowerlad 5y ago
Anyone starting a tech startup will face this decision: Should I start with a database that has built-in support for sharding? Or should I just start with MySQL or PostgreSQL and defer the sharding question for later? Notion chose to do manual sharding (aka application-level sharding). That's what you end up doing if you didn't choose a database that has built-in sharding from the get go, because it is extremely hard to switch to a different database technology. (Larry Ellison compares it to a catholic marriage -- there is no divorce!) I skimmed through the article to find the critical piece of info I was looking for: rationale for doing manual sharding. The rationale supplied in this article is "we wanted control over the distribution of our data." That's a weak explanation. It's the kind of thing you say to justify the bad choice made earlier on: failure to choose a database that supports automatic sharding from the get go.
- nosefrog 5y ago
  What open source databases support automatic sharding? I think their only other option would have been something like cockroachdb, which probably wasn't very mature when notion started.
  - flowerlad 5y ago
    Check out Implementations section on this page: https://en.wikipedia.org/wiki/Shard_(database_architecture) https://en.wikipedia.org/wiki/Shard_(database_architecture)
- zozbot234 5y ago
  PostgreSQL supports native sharding out of the box. It's "manual" in that you have to set up table partitions and foreign data access yourself, but that's arguably still preferable to hacking together the whole thing as a pure application-layer concern.
  - turbocon 5y ago
    It was my understanding that PostgreSQL supports table sharding out of the box on a single instance but not the ability to do distribute the processing across multiple instances.
mbell 5y ago
Notion seems like an interesting data storage problem. The vast majority of the data is in the `block` entity, which are organized into a sort of nested set, written/updated individually (user edits one at a time) but read in ranged chunks (a doc). Off hand this seems like an almost worst case for PG. Since the updates to blocks could contain large data (causing them to be moved often) and there is one big table; it seems likely that the blocks for a single notion document will end up being non-continuous on disk and thus require a lot of IO/memory trashing to read them back out. PG doesn't have a way to tell it how to organize data on disk so there is no good way around this (CLUSTER doesn't count, it's unusable in most use cases). Arm chair engineering of course - but my first thought would be to find another storage system for blocks that better fits the use case and leave the rest in PG. This does introduce other problems, but it just feels like storing data like this in PG is a bad fit. Maybe storing an entire doc's worth of block entities in a jsonb column would avoid a lot of this?
- StreamBright 5y ago
  I would try to use a simple k:v system that is much easier to scale. Even S3 would be a good candidate. Maybe I am missing the point.
  - lapaz17 5y ago
    Could you explain that k:v system a little bit? Junior eng. here.
    - roguas 5y ago
      KV means key value. Essentially the suggestion is to organize your data as a big key value structure(hashmap-like). This term usually means data is not normalized/separated as in regular SQL. for instance, in SQL: User table, that has columns: id, email Article table, that has columns: id, text, user_id KV/noSQL equivalent: Article document, that has properties: id, text, user_email
- rantanplan 5y ago
  > PG doesn't have a way to tell it how to organize data on disk so there is no good way around this (CLUSTER doesn't count, it's unusable in most use cases). Aren't tablespaces (https://www.postgresql.org/docs/10/manage-ag-tablespaces.html https://www.postgresql.org/docs/10/manage-ag-tablespaces.htm...) supposed to help with that? Haven't used them, I'm honestly curious
sgarrity 5y ago
The Notion blog has had some quality posts, but it seems to be missing an RSS feed. You can subscribe via email (no thanks), and I realize there are some good ways to turn an email subscription into a feed (Feedbin.com handles this nicely). Still, I thought some public shaming here might encourage the to build this basic blog feature (that would help people follow their company!).
notJim 5y ago
Overall this sounds very similar to Etsy's sharding setup, which I think descends from Flickr's sharding setup. Interesting that this is still the way to do it.
leetrout 5y ago
What size data do y'all start worrying about vertical scaling? Im surprised at the 500gb table / 10tb db sizes listed. I start to worry when my db hits 1tb and any single "hot" table is over 100gb. I know there isnt a one size fits all answer but I'm genuinely curious if I'm just too behind living in lessons learned ~5 years ago and and we are in a better place now.
TruthWillHurt 5y ago
What is this, 2008? "Our team spent months architecting this migration". Maybe you should have gone with a cloud offering like Dynamo/RDS, or a serverless database like BigQuery/Aurora.
jordanthoms 5y ago
At Kami (kamiapp.com) we had to make a similar transition last year - we'd seen usage grow over 20x in a 6-month period on our classroom product due to the pandemic, and we were storing all our content in a single-primary Postgres database. We had plenty of headroom before but we were suddenly running out of write capacity, even after scaling vertically up to ~150 vcpu. Our workload is super write heavy so scaling out replicas wouldn't have helped much at all, and I was spending a lot of time manually vacuuming in our daily off-peak as we were burning transaction ids at an astonishing rate, ~800 million per day. I looked at a bunch of options but ultimately managing a sharded database seemed like too much overhead for our small (but expanding!) dev team - so we decided to move our heavily loaded tables to CockroachDB instead, as since it's mostly Postgres compatible our transition would be easier, and it would be much easier to manage as it automatically balances load and heals. We're running cockroach on GKE and it's a really nice fit for running on kubernetes. Ended up working well for us - we still have our smaller tables on PG but we want to move it all to Cockroach over time. There were some teething issues as we learnt what to monitor on it (read amplification is the big one here, though pebble has helped a lot vs the previous rocksdb), and it does need a lot of hardware at high scale (we're running 50 16-vcpu nodes for it) but overall I'm happy with it.
- Rapzid 5y ago
  How have you found moving from a default transaction isolation level of read committed on Postgres to serializable-only on CockroachDB? I know of some somewhat-standard usage patterns that lean heavily on more lax isolation levels for acceptable performance(and may just deadlock otherwise). I'm also curious how you have found dealing with legacy schemas and queries that might cause a lot of data shuffling due to poor data locality. Was it necessary to put a lot of time into the sharding strategy? Have you experienced new query hotspots?
  - jordanthoms 5y ago
    The transaction isolation level hasn't been an issue for us - Our read/write patterns aren't super 'transactional' which may be why, we don't get a lot of conflicting updates as usually only one user has the permission to edit their content on the document. Arguably these tables could be on Scylla/Cassandra with eventual consistency even, though that would have been a harder transition and we'd have to spend more time making sure our application could handle that correctly. We did have to make some schema changes in order to improve the data locality and avoid hotspots - CockroachDB did _work_ without doing those, but the changes improved performance massively. It wasn't super time consuming though, and the admin UI/monitoring in cockroach is nice for showing you any hot ranges. The biggest one was our table which stores annotations on a document - on PG it's primary key was only (annotation_id UUID), and we had a secondary index on (document_id). This meant that on Cockroach the data for a document was spread across many nodes and querying all the annotations for a document would take ~40ms. We changed the primary key to (document_id, annotation_id) which co-located the data and that came down to ~2ms. Composite primary keys aren't ideal for ActiveRecord but the performance win was worth it. We also had some tables with hotspots which was mostly just a matter of making sure they used uuid keys. It would be a problem though if we had a table with individual rows which were super hot or where the table didn't have enough data to get split into multiple ranges - though cockroach does have load based sharding to help here. It was only really necessary to worry about the higher load tables here, if it's below ~2000 qps or so it won't matter.
sillysaurusx 5y ago
Slightly unrelated, but notion's unofficial python API is cool AF! https://github.com/jamalex/notion-py https://github.com/jamalex/notion-py It's way more powerful than their official API. The official API can only interact with databases, which (as anyone who uses Notion knows) is a tiny subset of the overall things you want to use Notion for. The unofficial Python API lets you have complete programmatic access to all notion pages. The tradeoff is that certain things break. However, after reading the code, I found it quite easy to fix the problems I ran into (https://github.com/jamalex/notion-py/pull/345 https://github.com/jamalex/notion-py/pull/345) and I suspect you'll be able to do the same. (I empathize with maintainers not prioritizing their open source work ahead of family life, business, etc.) Was surprised that it was even possible to have a nice Python class for every possible Notion object, let alone control them and update them on the fly. I wish their official API would be as flexible. Maybe one day.
caffeinecrazy 5y ago
I'm curious about whether using Postgres native partitioning was considered? If so, I'd like to hear the rationale for not doing it. It seems like you made the database sharding decision as a result of vacuum taking too long. Partitioning the problematic table (assuming you're storing "blocks" in 1 giant table) would have enabled per-partition vacuuming, which will avoid long running vacuum processes.
DarthNebo 5y ago
This was so insightful, gave me a clearer picture of what all needs to be done during migration!
dikei 5y ago
I have 2 questions: * How do you setup hot stand-by for each database ? * Do you have coordinated backup for all the databases, or are they backed up individually ?

I still wonder so hard what a single bare metal server would have done in terms of performance. [0] May have saved months of planning that may never have been required. [0]: https://yourdatafitsinram.net/ https://yourdatafitsinram.net/

0xbadcafebee 5y ago
This can quickly become untenable depending on the availability and contractual requirements you have. You're literally building on a single point of failure. And if it's really big and you design wrong, recovery could take weeks.
- louwrentius 5y ago
  Put a slave next to it, that's not the point. AFAIK this is what stackoverflow still does and it works for them. Why not for notion?
  - notJim 5y ago
    Notion may have a much higher proportion of write volume than stackoverflow, given that most of stackoverflow's traffic must be from people who come in from Google, read the answers, and leave. Notion is productivity app, so I would guess most people are actively writing.
  - 0xbadcafebee 5y ago
    If StackOverflow jumped off a bridge, would you jump off one too? You can't base a huge decision like that on "well it worked for somebody else". How did it work for them? What was their platform? What was their data profile? What were their requirements? What was their acceptable level of risk? It could be that they have so many caching layers and lose so little money that they are fine with the database being down for 5 days while they recover. Or that it doesn't actually work that fine for them, because who wants to broadcast that their system is shitty? Or that they've just been lucky. It's not worth finding out the hard way. Build the best system you can with the time and money and expertise you have. Don't cheap out just because you think you can get away with it - especially on the critical stuff.
sa46 5y ago
You'll hit all sorts of other fun problems by vertically scaling. - Many concurrent connections to a single database which Postgres has traditionally not handled well (though improved in recent versions). - You're now on the hook for writing a database control plane. - Backup and restore is much flakier since the data volume is so much larger. Lots of weird shit starts happening when you download tens of TB. - In general, everything is a harder once you start nearing machine limits.
- jamesfinlayson 5y ago
  Yes I thought Postgres didn't recommend more than 700 connections per server (or maybe it was 300) and that was only three or four years ago.

endisneigh 5y ago

On a related note, how far can a single Postgres instance get you? If you had some behemoth with 32TB ram and 1PB storage could all of motion fit? I’m obviously ignoring the obvious single point of failure here, but sometimes the simplicity could be worth it if you’re willing to handle that. I’d be curious to hear about a site that’s in the Alexa 1000 architected how I describe.

aeyes 5y ago
> If you had some behemoth with 32TB ram and 1PB storage could all of motion fit? Postgres has a bunch of low level locks and buffers (protected by locks) which are essentially single threaded. So even if you had a 500 CPU instance at some point you'd not be able to get more throughput out of it. Of course it also depends on what you are doing. Large tables with high update rate are harder to handle than large tables which are insert only. My personal opinion is that Postgres tables with >100GB data (without indexes) are a starting to be a pain to work with, no matter how much RAM, CPU or IOPS you have.
- endisneigh 5y ago
  Is there a relational database that could efficiently leverage such a single powerful machine?
  - gfody 5y ago
    sql server scales incredibly well with the hardware but so does its price
  - haneefmubarak 5y ago
    I imagine you want NUMA aware software specifically designed for multi-socket systems; off the top of my head the ones I've heard of are IBM DB2, SAP HANA, and Microsoft SQL Server — although I've had a chance to actually use any of them.
    - jiggawatts 5y ago

alberth 5y ago

>> “Why 480 shards? … 480 is divisible by a lot of numbers”. This is an important point, doing things divisible by 12 gives you a lot of flexibility. It’s not a coincidence both time (clocks) and degrees (360) are multiples of 12.

infogulch 5y ago
Numberphile made a video about "highly composite numbers" / "anti-primes", which are numbers that have more factors than all smaller numbers. https://youtu.be/2JM2oImb9Qg https://youtu.be/2JM2oImb9Qg There is an oeis sequence of them that starts "1, 2, 4, 6, 12, 24, 36, 48, 60, 120, 180, 240, 360, 720, 840...", which notably does not include 480. https://oeis.org/A002182 https://oeis.org/A002182
- moralestapia 5y ago
  This is the kind of engineering that it's almost impossible to screen in interviews (particular for HR-minded pals) and is a big part of what makes a 10x-100x engineer (which definitely DO exist). Obviously not this thing just by itself, but this sort of knowledge, applied to everyday decisions that get compound over time.
- mikepurvis 5y ago
  But 480 is 2*240, so it's definitely a close contender.
- alberth 5y ago
  That’s super interesting. It’s counterintuitive but 480 has the same # of factors that 360 has (which is 24 factors) https://www.calculatorsoup.com/calculators/math/factors.php https://www.calculatorsoup.com/calculators/math/factors.php
  - _v7gu 5y ago
    so it's weakly highly composite?

bamazizi 5y ago

Curious about thought on using something totally different like column oriented db, like ClickHouse, would've suited a block data oriented app like Notion better? Does anyone with experience have any thoughts in favour or against such implementation?

akarki15 5y ago
+1. First thought i had when i saw this was like why is notion using a postgresdb in first place- i can see it being used for customer data (which probably doesn’t require sharding) but for the content itself, a nosql or colum oriented db is better fit. the question i have is what is notion storing in these sharded postgresdb?
dilyevsky 5y ago
Totally different usecase with clickhouse - the point queries there still slower than citus but it kicks its ass on multi row aggregations. For auto-sharded oltp db cockroachdb or tidb seem like way better options

capableweb 5y ago

Many companies I worked with stopped working with Notion in the past because of performance problems, the application was simply too slow. Has this changed with the new sharded architecture, as I haven't used it since probably last year?

methyl 5y ago
I can notice the performance no longer being the problem for me. The only gripe I have with Notion is too much flexibility, it’s easy to completely mess something up even by accident.
- sergiotapia 5y ago
  Agreed, our Notion ended up being a soup of disparate outdated notes all over the place with no rhyme or reason across a 480+ employee business. I'm founding engineer of a new startup now and I advocated for Guru. https://www.getguru.com/ https://www.getguru.com/ I like how information is marked stale/verified and the permission are much tighter. The data ladder is also better structured. I'll see how it works as our headcount increases.
- lucis 5y ago
  You mean editing documents you're just taking a look? Notion has a lock feature [1] on a document-level. [1] - https://www.notion.so/Lock-page-content-d2b995727c0b483f9f3518c1eb4fab75 https://www.notion.so/Lock-page-content-d2b995727c0b483f9f35...
tempest_ 5y ago
The app is a lot more snappy, not super fast, but usable. Search is still pretty slow.
alberth 5y ago
Curious, what app have you seen people migrate too?
hinkley 5y ago
From personal experience (not with Notion) once you fall behind it’s hard to catch up, and nobody declares an emergency until it’s already happening. Doing big surgery takes time, meanwhile your workload continues to grow and grow. It’s not uncommon for a major win to only reset the clock by 6-12 months, and then you either have to run to the next one, or you have two teams working on separate angles at the same time. All the while people are learning as they go because the business settled on catch-up instead of capacity planning. Being a little successful can be tough.

craigkerstiens 5y ago

This is a great read and primer for anyone looking to shard and know they need scale. Having helped a ton of people shard their data, my advice still remains don't do it until you know you have to. The old Citus data blog is a great resource for some guidance around best practices and how to's even if not using Citus. That said some of the things you can do from day 1 make things way easier when the time comes later. Workspace ID/Tenant ID/Customer ID, having your data grouped that way makes it very shardable and saves you a lot of work later on in the case you didn't have that in your data model. I'm not sure I buy that Citus/Vitess are magic, both are reasonably clear how they work and you can dig into it. At the same time I'd weigh the downsides of Citus (can't speak to Vitess) in that the online rebalancer isn't open source so at that point it's a proprietary product.

alberth 5y ago
>>” Workspace ID/Tenant ID/Customer ID, having your data grouped that way makes it very shardable” Dumb question, if someone starts off by grouping their data only by Customer ID and then later needs to shard. Couldn’t all of the sharding problems go away if they simply created a new Customer_Tenant table to map customer_id into tenant_group_id
- craigkerstiens 5y ago
  Sorry, wasn't clear enough, tenant ID = customer ID = workspace ID. There is often a benefit though to materializing that ID onto all child tables. If you take a salesforce.com example, every customer has an "orgID", from there have accounts and accounts have opportunities. It's not common for people to put the "orgID" on the opportunity from a data modeling perspective, but when it comes to sharding it gives you a big win. If you have that tenant discriminator on all tables then it's easy to route it to the appropriate physical and logical shard right away vs. having to do some DB join first in your request to figure out where it goes.
  - alberth 5y ago
    Are you suggesting creating entire database schemas specific to a customer and put the customer/orgID in all table names themselves?

theodric 5y ago

I sharded in my pance

garrettf 5y ago

Howdy all, author here! Sharding our Postgres monolith was a huge undertaking for a small team. We got a huge boost from folks that joined within weeks before the switch-over date. If you’re interested in thinking about problems like this, I’d love to chat. Plus we’re also hiring across all roles—check my profile for details. I’m happy to answer questions about the project here, feel free to reply below.

silentsea90 5y ago
> [2] In addition to packaged solutions, we considered a number of alternatives: switching to another database system such as DynamoDB (deemed too risky for our use case).. 1) Can you please talk more about the risks of using DynamoDB or a similar NoSQL solution? 2) Did you consider Spanner, which is the SQL DB of choice within Google and is available on Google Cloud. Thanks for the wonderful engineering blog posts!
- garrettf 5y ago
  Oh yes—this footnote is not knocking non-relational DBs, in fact we have deployed DynamoDB in production for other use cases today. We were on a tight timeline due to impending TXID wraparound. Switching database technologies would have required us to rewrite our queries, reexamine all of our indexes, and then validate that queries were both correct and performant on the new database. Even if we had the time to derisk those concerns, we'd be moving our most critical data from a system with scaling thresholds and failure modes we understand to a system we are less familiar with. Generally, we were already familiar with the performance characteristics of Postgres, and could leverage a decent amount of monitoring and tooling we built atop it. There is nothing inherent about non-relational DBs that make them unsuitable for our workload. If we were designing our data storage architecture from scratch, we'd consider databases that are optimized for our heaviest queries (vs. the flexibility afforded by Postgres today). A large number of those are simple key-value lookups, and a plain key/value store like DynamoDB is great for that. We're considering these alternatives going forward, especially as we optimize specific user workloads. Re: Cloud Spanner: we didn't consider a cross-cloud migration at the time due to the same time constraints. Still sounds like a wonderful product, we were just not ready at the time.

flowerlad 5y ago

Anyone starting a tech startup will face this decision: Should I start with a database that has built-in support for sharding? Or should I just start with MySQL or PostgreSQL and defer the sharding question for later? Notion chose to do manual sharding (aka application-level sharding). That's what you end up doing if you didn't choose a database that has built-in sharding from the get go, because it is extremely hard to switch to a different database technology. (Larry Ellison compares it to a catholic marriage -- there is no divorce!) I skimmed through the article to find the critical piece of info I was looking for: rationale for doing manual sharding. The rationale supplied in this article is "we wanted control over the distribution of our data." That's a weak explanation. It's the kind of thing you say to justify the bad choice made earlier on: failure to choose a database that supports automatic sharding from the get go.

nosefrog 5y ago
What open source databases support automatic sharding? I think their only other option would have been something like cockroachdb, which probably wasn't very mature when notion started.
- flowerlad 5y ago
  Check out Implementations section on this page: https://en.wikipedia.org/wiki/Shard_(database_architecture) https://en.wikipedia.org/wiki/Shard_(database_architecture)
zozbot234 5y ago
PostgreSQL supports native sharding out of the box. It's "manual" in that you have to set up table partitions and foreign data access yourself, but that's arguably still preferable to hacking together the whole thing as a pure application-layer concern.
- turbocon 5y ago
  It was my understanding that PostgreSQL supports table sharding out of the box on a single instance but not the ability to do distribute the processing across multiple instances.

mbell 5y ago

Notion seems like an interesting data storage problem. The vast majority of the data is in the `block` entity, which are organized into a sort of nested set, written/updated individually (user edits one at a time) but read in ranged chunks (a doc). Off hand this seems like an almost worst case for PG. Since the updates to blocks could contain large data (causing them to be moved often) and there is one big table; it seems likely that the blocks for a single notion document will end up being non-continuous on disk and thus require a lot of IO/memory trashing to read them back out. PG doesn't have a way to tell it how to organize data on disk so there is no good way around this (CLUSTER doesn't count, it's unusable in most use cases). Arm chair engineering of course - but my first thought would be to find another storage system for blocks that better fits the use case and leave the rest in PG. This does introduce other problems, but it just feels like storing data like this in PG is a bad fit. Maybe storing an entire doc's worth of block entities in a jsonb column would avoid a lot of this?

StreamBright 5y ago
I would try to use a simple k:v system that is much easier to scale. Even S3 would be a good candidate. Maybe I am missing the point.
- lapaz17 5y ago
  Could you explain that k:v system a little bit? Junior eng. here.
  - roguas 5y ago
    KV means key value. Essentially the suggestion is to organize your data as a big key value structure(hashmap-like). This term usually means data is not normalized/separated as in regular SQL. for instance, in SQL: User table, that has columns: id, email Article table, that has columns: id, text, user_id KV/noSQL equivalent: Article document, that has properties: id, text, user_email
rantanplan 5y ago
> PG doesn't have a way to tell it how to organize data on disk so there is no good way around this (CLUSTER doesn't count, it's unusable in most use cases). Aren't tablespaces (https://www.postgresql.org/docs/10/manage-ag-tablespaces.html https://www.postgresql.org/docs/10/manage-ag-tablespaces.htm...) supposed to help with that? Haven't used them, I'm honestly curious

sgarrity 5y ago

The Notion blog has had some quality posts, but it seems to be missing an RSS feed. You can subscribe via email (no thanks), and I realize there are some good ways to turn an email subscription into a feed (Feedbin.com handles this nicely). Still, I thought some public shaming here might encourage the to build this basic blog feature (that would help people follow their company!).

notJim 5y ago

Overall this sounds very similar to Etsy's sharding setup, which I think descends from Flickr's sharding setup. Interesting that this is still the way to do it.

leetrout 5y ago

What size data do y'all start worrying about vertical scaling? Im surprised at the 500gb table / 10tb db sizes listed. I start to worry when my db hits 1tb and any single "hot" table is over 100gb. I know there isnt a one size fits all answer but I'm genuinely curious if I'm just too behind living in lessons learned ~5 years ago and and we are in a better place now.

TruthWillHurt 5y ago

What is this, 2008? "Our team spent months architecting this migration". Maybe you should have gone with a cloud offering like Dynamo/RDS, or a serverless database like BigQuery/Aurora.

jordanthoms 5y ago

At Kami (kamiapp.com) we had to make a similar transition last year - we'd seen usage grow over 20x in a 6-month period on our classroom product due to the pandemic, and we were storing all our content in a single-primary Postgres database. We had plenty of headroom before but we were suddenly running out of write capacity, even after scaling vertically up to ~150 vcpu. Our workload is super write heavy so scaling out replicas wouldn't have helped much at all, and I was spending a lot of time manually vacuuming in our daily off-peak as we were burning transaction ids at an astonishing rate, ~800 million per day. I looked at a bunch of options but ultimately managing a sharded database seemed like too much overhead for our small (but expanding!) dev team - so we decided to move our heavily loaded tables to CockroachDB instead, as since it's mostly Postgres compatible our transition would be easier, and it would be much easier to manage as it automatically balances load and heals. We're running cockroach on GKE and it's a really nice fit for running on kubernetes. Ended up working well for us - we still have our smaller tables on PG but we want to move it all to Cockroach over time. There were some teething issues as we learnt what to monitor on it (read amplification is the big one here, though pebble has helped a lot vs the previous rocksdb), and it does need a lot of hardware at high scale (we're running 50 16-vcpu nodes for it) but overall I'm happy with it.

Rapzid 5y ago
How have you found moving from a default transaction isolation level of read committed on Postgres to serializable-only on CockroachDB? I know of some somewhat-standard usage patterns that lean heavily on more lax isolation levels for acceptable performance(and may just deadlock otherwise). I'm also curious how you have found dealing with legacy schemas and queries that might cause a lot of data shuffling due to poor data locality. Was it necessary to put a lot of time into the sharding strategy? Have you experienced new query hotspots?
- jordanthoms 5y ago
  The transaction isolation level hasn't been an issue for us - Our read/write patterns aren't super 'transactional' which may be why, we don't get a lot of conflicting updates as usually only one user has the permission to edit their content on the document. Arguably these tables could be on Scylla/Cassandra with eventual consistency even, though that would have been a harder transition and we'd have to spend more time making sure our application could handle that correctly. We did have to make some schema changes in order to improve the data locality and avoid hotspots - CockroachDB did _work_ without doing those, but the changes improved performance massively. It wasn't super time consuming though, and the admin UI/monitoring in cockroach is nice for showing you any hot ranges. The biggest one was our table which stores annotations on a document - on PG it's primary key was only (annotation_id UUID), and we had a secondary index on (document_id). This meant that on Cockroach the data for a document was spread across many nodes and querying all the annotations for a document would take ~40ms. We changed the primary key to (document_id, annotation_id) which co-located the data and that came down to ~2ms. Composite primary keys aren't ideal for ActiveRecord but the performance win was worth it. We also had some tables with hotspots which was mostly just a matter of making sure they used uuid keys. It would be a problem though if we had a table with individual rows which were super hot or where the table didn't have enough data to get split into multiple ranges - though cockroach does have load based sharding to help here. It was only really necessary to worry about the higher load tables here, if it's below ~2000 qps or so it won't matter.

sillysaurusx 5y ago

Slightly unrelated, but notion's unofficial python API is cool AF! https://github.com/jamalex/notion-py https://github.com/jamalex/notion-py It's way more powerful than their official API. The official API can only interact with databases, which (as anyone who uses Notion knows) is a tiny subset of the overall things you want to use Notion for. The unofficial Python API lets you have complete programmatic access to all notion pages. The tradeoff is that certain things break. However, after reading the code, I found it quite easy to fix the problems I ran into (https://github.com/jamalex/notion-py/pull/345 https://github.com/jamalex/notion-py/pull/345) and I suspect you'll be able to do the same. (I empathize with maintainers not prioritizing their open source work ahead of family life, business, etc.) Was surprised that it was even possible to have a nice Python class for every possible Notion object, let alone control them and update them on the fly. I wish their official API would be as flexible. Maybe one day.

caffeinecrazy 5y ago

I'm curious about whether using Postgres native partitioning was considered? If so, I'd like to hear the rationale for not doing it. It seems like you made the database sharding decision as a result of vacuum taking too long. Partitioning the problematic table (assuming you're storing "blocks" in 1 giant table) would have enabled per-partition vacuuming, which will avoid long running vacuum processes.

DarthNebo 5y ago

This was so insightful, gave me a clearer picture of what all needs to be done during migration!

dikei 5y ago

I have 2 questions: * How do you setup hot stand-by for each database ? * Do you have coordinated backup for all the databases, or are they backed up individually ?