9 ms·

Building and scaling Notion's data lake

239 points by alexzeitler 2y ago

philippemnoel 2y ago
This is one of the best blog posts I've seen that showcase the UPDATE-heavy, "surface data lakes data to users" type of workload. At ParadeDB, we're seeing more and more users want to maintain the Postgres interface while offloading data to S3 for cost and scalability reasons, which was the main reason behind the creation of pg_lakehouse.
mritchie712 2y ago
> Iceberg and Delta Lake, on the other hand, weren’t optimized for our update-heavy workload when we considered them in 2022 "when we considered them in 2022" is significant here because both Iceberg and Delta Lake have made rapid progress since then. I talk to a lot of companies making this decision and the consensus is swinging towards Iceberg. If they're already heavy Databricks users, then Delta is the obvious choice. For anyone that missed it, Databricks acquired Tabular[0] (which was founded by the creators of Iceberg). The public facing story is that both projects will continue independently and I really hope that's true. Shameless plug: this is the same infrastructure we're using at Definite[1] and we're betting a lot of companies want a setup like this, but can't afford to build it themselves. It's radically cheaper then the standard Snowflake + Fivetran + Looker stack and works day one. A lot of companies just want dashboards and it's pretty ridiculous the hoops you need to jump thru to get them running. We use iceberg for storage, duckdb as a query engine, a few open source projects for ETL and built a frontend to manage it all and create dashboards. 0 - https://www.definite.app/blog/databricks-tabular-acquisition https://www.definite.app/blog/databricks-tabular-acquisition 1 - https://www.youtube.com/watch?v=7FAJLc3k2Fo https://www.youtube.com/watch?v=7FAJLc3k2Fo
- deleted 2y ago
  [deleted]
whinvik 2y ago
Is there any advantage to having both a Data Lake setup as well as Snowflake. Why would one also want Snowflake after doing such an extensive data lake setup?
- luizfwolf 2y ago
  Saving money 100% also lower latency on distributed access. Accessing file partitioned S3 doesn’t require to spin a warehouse and wait for your query to go on a queue, so if every job runs in like k8s you don’t have to manage resources and auto scale in snowflake is a “paid feature” I believe just not having to handle a query queue system is already.
- mritchie712 2y ago
  Many BI / analytics tools don't have great support for Data Lakes, so part of the reason could be supporting those tools (e.g. they still load some of their data to snowflake to power BI / dashboards)
  - Lucasoato 2y ago
    We've solved that issue with Trino. Superset and a lot of other BI tools support connection to it and it's a very cost efficient engine (compared to DWH solutions). Another way to go even cheaper is using Athena, if you're on AWS.
    - jitl 2y ago
      Athena packages Trino - it’s in part a managed Trino service.
      - Lucasoato 2y ago
        They are several versions behind, support for delta was added just recently. Also consider that with Trino you can build a cache layer on Alluxio, making it really fast (especially on NVMe disks).
- 2y ago
deleted 2y ago
[deleted]
adolph 2y ago
They seem to be doing lots of work but I don’t understand what customer value this creates. What does a backing data lake afford a Notion user that can’t be done in a similar product, like Obsidian?
- jpalomaki 2y ago
  From the article: "Unlock AI, Search, and other product use cases that require denormalized data"
- SOLAR_FIELDS 2y ago
  Beyond the features that the sibling comment mentioned, this kind of data isn’t really for end users. It’s a way that you can package it up, “anonymize” it, and sell the data to interested parties.
  - sghiassy 2y ago
    Is that why they’re putting images in Postgres? I don’t understand that design decision yet.
    - bastawhiz 2y ago
      I... Don't think they are? If you look at the URL for images in notion, you can see the S3 hostname.
    - benaubin 2y ago
      Notion employee here. We don't put images themselves in Postgres- we use s3 to store them. The article is referring to image blocks, which are effectively pointers to the image.
  - Ozzie_osman 2y ago
    For someone like Notion, they probably aren't selling this data. The primary use case is internally for analysis (eg product usage, business analysis, etc). It can also be used to train AI models, of course.
SOLAR_FIELDS 2y ago
They didn’t say the quiet part out loud, which is almost certainly that the Fivetran and Snowflake bills for what they were doing were probably enormous and those were undoubtedly what got management’s attention about fixing this.
- deleted 2y ago
  [deleted]
- rorymalcolm 2y ago
  Found this comment (from Fivetran's CEO, so, with that in mind) regarding this article enlightening regarding the costs they were facing here https://twitter.com/frasergeorgew/status/1808326803796512865 https://twitter.com/frasergeorgew/status/1808326803796512865
  - pbd 2y ago
    Snowflake as destination is very very easy to work with on fivetran. Fivetran didn't have S3 as destination till late 2022. So it literally forces you to use one of BQ, Snowflake, redshift as destination. So fivetran CEO's defence is pretty stupid.
- sneak 2y ago
  I thought the quiet part was that they are data mining their customer data (and disclosing it to multiple third parties) because it’s not E2EE and they can read everyone’s private and proprietary notes. Otherwise, this is the perfect app for sharding/horizontal scalability. Your notes don’t need to be queried or joined with anyone else’s notes.
  - redpoint 2y ago
    This ^. This switch from managed to in house is a good example of only building when necessary.
  - altdataseller 2y ago
    Also whether this data lake is worth the costs/effort. How does this data lake add value to the user experience? What is this “AI” stuff that this data lake enables? For example, they mention search. But i imagine it is just searching only within your own docs. Which i presume should be fast and efficient if everything is sharded by user in Postgres. The tech stuff is all fine and good, but if it adds no value, its just playing with technology for technology sakes
methou 2y ago
> Data lake > Data warehouse These aren't something I would like to hear if I'm still using Notion. It's very bold to publish something like this on their own website.
- bnj 2y ago
  Could you expand on this?
  - lopkeny12ko 2y ago
    What's there to expand on? Do you not realize how bad of a look it is for a company to publicly admit, on their own blog, the amount of time and engineering effort they spent to package up, move, analyze, and sell all their customer's private data? This is why laws like CCPA "do not sell my personal information" exist, which I certainly hope Notion is abiding by, otherwise they'll have lawyers knocking on their door soon.
    - Cthulhu_ 2y ago
      Where do they say they sell it? Citation needed; that's a legal and reputational minefield that I don't think they would admit to, like you said.
      - lopkeny12ko 2y ago
        I would challenge you to find any broker who sells data (like the T-Mobile location data scandal) who says plainly and clearly they sell user data.
        quest88 2y ago
        This is not answering the question.
    - bnj 2y ago
      Right, yes, tone aside that’s very helpful- at first I didn’t understand the implication of the blog post for implementing customer hostile solutions, but you’ve helped me understand it now.
DataDaemon 2y ago
OK, thanks, when E2EE ?
deleted 2y ago
[deleted]
j45 2y ago
This was a nice read, interesting to see how far Postgres (largely alone) can get you. Also we see how at self hosting within a startup can make perfect sense. :) Devops that abstract away things in some cases to the cloud might just add to architectural and technical debt later, without the history of learning from working through the challenges Still, it might have been a great opportunity to figure out offline first use of notion. I have been forced to use anytype instead of notion for the offline first reason. Time to checkout to learn how they handle storage from the source code.
wejick 2y ago
I'm not familiar with S3 on datalake setup. When replicating a db table to S3, what format will be used? And I'm wondering if it's possible to update the S3 files to reflect latest incoming changes on the db table?
- mritchie712 2y ago
  The file format is often Parquet. The "table format" depend on what data lake you're using (e.g. Iceberg, Delta, etc.). If you know Python, here's[0] a practical example of how Iceberg works. 0 - https://www.definite.app/blog/iceberg-query-engine https://www.definite.app/blog/iceberg-query-engine
middlefing 2y ago
[flagged]
- marviel 2y ago
  what do you dislike specifically? They've invested a lot of energy into it when last I checked
  - TheDong 2y ago
    They advertise markdown support, the ability to export markdown, and the ability to import markdown... However, what they don't say, is that the export and import format aren't compatible and are different subsets of markdown with different features. If I export a notion page as markdown, then re-import that same markdown document back into notion, I get something wildly different. All I want is to not use the notion editor (which lags and sometimes crashes my browser), and to instead use my local text editor which has served me well for everything else. Failing at that, I want to edit plain text, like I can in github comments or wikipedia pages. Like, the fact that if I write 'foo`', and then go back and edit a backtick in before the word 'foo', it doesn't code-format it, but if I do it in the opposite order and type '`foo`', it code-formats it, makes it very clear I'm not editing text, and there is weird hidden state, which is annoying to reason about. Just let me edit something like markdown directly, with an optional preview window somewhere, and that would _also_ be vastly better than the mess they have.
  - middlefing 2y ago
    That's probably the problem. It's over engineered and just does things I didn't want or need it to do. I just want to type words and paste things in without having a bunch of bullshit happen.
- deleted 2y ago
  [deleted]
CyberDildonics 2y ago
What's the difference between a data lake and a database with a filesystem?
- anthomtb 2y ago
  Add data warehouse to this list too.
- OJFord 2y ago
  Normalisation, relations, the filesystem, ..? I'm more curious about your view of a 'data lake' which is met by adding a filesystem to a db?
  - CyberDildonics 2y ago
    Normalisation, relations, What does that mean?
- syndicatedjelly 2y ago
  I thought data lake was more of an advertising phrase, to show off how much data you have/can handle
  - Temporary_31337 2y ago
    Web scale
crux 2y ago
Hi all—I'm the EM for the Search team at Notion, and I want to chime in to clear up one unfortunate misconception I've seen a few times in this thread. Notion does not sell its users' data. Instead, I want to expand on one of the first use-cases for the Notion data lake, which was by my team. This is an elaboration of the description in TFA under the heading "Use case support". As is described there, Notion's block permissions are highly normalized at the source of truth. This is usually quite efficient and generally brings along all the benefits of normalization in application databases. However, we need to _denormalize_ all the permissions that relate to a specific document when we index it into our search index. When we transactionally reindex a document "online", this is no problem. However, when we need to reindex an entire search cluster from scratch, loading every ancestor of each page in order to collect all of its permissions is far too expensive. Thus, one of the primary needs that my team had from the new data lake is "tree traversal and permission data construction for each block". We rewrote our "offline" reindexer to read from the data lake instead of reading from RDS instances serving database snapshots. This allowed us to dramatically reduce the impact of iterating through every page when spinning up a new cluster (not to mention save a boatload in spinning up those ad-hoc RDS instances). I hope this miniature deep dive gives a little bit more color on the uses of this data store—as it is emphatically _not_ to sell our users' data!
- jzelinskie 2y ago
  This is a fantastic post that explains a lot of the end product, but I'd love to hear more about the journey specifically on denormalizing permissions at Notion. Scaling out authorization logic like this is actually very under-documented in industry. Mind if I email you to chat? Full disclosure: I'm a founder of authzed (W21), the company building SpiceDB, an open source project inspired by Google's internal scalable authorization system. We offer a product that streams changes to fully denormalized permissions for search engines to consume, but I'm not trying to pitch; you just don't often hear about other solutions built in this space!
- infogulch 2y ago
  Interesting! Now I'm curious how you handle live permission changes and indexes with stale permission data.
hobobaggins 2y ago
> Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake. Are they using this new data lake to train new AI models on? Or has Notion signed a deal with another LLM provider to provide customer data as a source for training data?
- gorgoiler 2y ago
  It’s not a direct answer but from what Notion tell us about their own business: * The team are based in the US, specifically California, and Notion Labs, Inc is a Delaware corporation. * Their investment comes from Venture Capital and individual wealth. The investors are listed on Notion’s about page and are open about how they themselves became rich through VC funded tech companies. There is a very open sense of panic in tech right now to climb to the top of the AI pile and not get crushed underneath. I would be amazed if there were any companies not enthralled by — and either already embracing or planning to embrace — the data-mining AI gold rush. Notion is a great product but one would be naive to use it while also harboring concerns about data privacy.
- jitl 2y ago
  (I work at Notion, but not on the data platform team) We do not and will never sell customer data to anyone. We do not train AI models on customer data. As we state in our privacy policy for AI features (https://www.notion.so/notion/Notion-AI-Supplementary-Terms-fa9034c8b5a04818a6baf3eac2adddbb https://www.notion.so/notion/Notion-AI-Supplementary-Terms-f...): > Notion does not use your Customer Data or permit others to use your Customer Data to train the machine learning models used to provide Notion AI Writing Suite or Notion AI Q&A [added: our AI features]. Your use of Notion AI Writing Suite or Notion AI Q&A does not grant Notion any right or license to your Customer Data to train our machine learning models. We do use various data infrastructure, including Postgres and the data lake, to index customer content both with traditional search infrastructure like Elasticsearch, as well as AI-based embedding search like Pinecone. We do this so you can search your own content when you're using Notion. We wrote this article to explain how Notion's AI features works with customer data: https://www.notion.so/help/notion-ai-security-practices https://www.notion.so/help/notion-ai-security-practices
jauntywundrkind 2y ago
Side-ish note, I really enjoyed a submission on Bufstream recently, a Kafka mq replacement. One of the things they mentioned is that they are working on building in Iceberg materialization, so Bufstream can automatically handle building a big analytics data lake out of incoming data. It feels like that could potentially tackle a bunch of the stack here. https://buf.build/blog/bufstream-kafka-lower-cost https://buf.build/blog/bufstream-kafka-lower-cost https://news.ycombinator.com/item?id=40919279 https://news.ycombinator.com/item?id=40919279 Versus what Notion is doing: > We ingest incrementally updated data from Postgres to Kafka using Debezium CDC connectors, then use Apache Hudi, an open-source data processing and storage framework, to write these updates from Kafka to S3. Feels like it would work about the same with Bufstream, replacing both Kafka & Hudi. I've heard great things about Hudi but it does seem to have significantly less adoption so far.
alexliu518 2y ago
Thank you for the clarification! It's great to hear more about the efficient data management practices at Notion. Your team's innovative use of the data lake to streamline the reindexing process while ensuring user data privacy is impressive. Keep up the excellent work!
HermitX 2y ago
Great article, thank you for sharing! I have a question I’d like to discuss with the author. Spark SQL is a great product and works perfectly for batch processing tasks. However, for handling ad hoc query tasks or more interactive data analysis tasks, Spark SQL might have some performance issues. If you have such workloads, I suggest trying data lake query engines like Trino or StarRocks, which offer faster speeds and a better query experience.
- jitl 2y ago
  (Notion employee) AWS Athena packages Trino, I’ve been using it for some queries like “find all blocks that contain @-mentions”. It’s a great tool.

239 points by alexzeitler 2y ago

This is one of the best blog posts I've seen that showcase the UPDATE-heavy, "surface data lakes data to users" type of workload. At ParadeDB, we're seeing more and more users want to maintain the Postgres interface while offloading data to S3 for cost and scalability reasons, which was the main reason behind the creation of pg_lakehouse.

mritchie712 2y ago

> Iceberg and Delta Lake, on the other hand, weren’t optimized for our update-heavy workload when we considered them in 2022 "when we considered them in 2022" is significant here because both Iceberg and Delta Lake have made rapid progress since then. I talk to a lot of companies making this decision and the consensus is swinging towards Iceberg. If they're already heavy Databricks users, then Delta is the obvious choice. For anyone that missed it, Databricks acquired Tabular[0] (which was founded by the creators of Iceberg). The public facing story is that both projects will continue independently and I really hope that's true. Shameless plug: this is the same infrastructure we're using at Definite[1] and we're betting a lot of companies want a setup like this, but can't afford to build it themselves. It's radically cheaper then the standard Snowflake + Fivetran + Looker stack and works day one. A lot of companies just want dashboards and it's pretty ridiculous the hoops you need to jump thru to get them running. We use iceberg for storage, duckdb as a query engine, a few open source projects for ETL and built a frontend to manage it all and create dashboards. 0 - https://www.definite.app/blog/databricks-tabular-acquisition https://www.definite.app/blog/databricks-tabular-acquisition 1 - https://www.youtube.com/watch?v=7FAJLc3k2Fo https://www.youtube.com/watch?v=7FAJLc3k2Fo

deleted 2y ago
[deleted]

whinvik 2y ago

Is there any advantage to having both a Data Lake setup as well as Snowflake. Why would one also want Snowflake after doing such an extensive data lake setup?

luizfwolf 2y ago
Saving money 100% also lower latency on distributed access. Accessing file partitioned S3 doesn’t require to spin a warehouse and wait for your query to go on a queue, so if every job runs in like k8s you don’t have to manage resources and auto scale in snowflake is a “paid feature” I believe just not having to handle a query queue system is already.
mritchie712 2y ago
Many BI / analytics tools don't have great support for Data Lakes, so part of the reason could be supporting those tools (e.g. they still load some of their data to snowflake to power BI / dashboards)
- Lucasoato 2y ago
  We've solved that issue with Trino. Superset and a lot of other BI tools support connection to it and it's a very cost efficient engine (compared to DWH solutions). Another way to go even cheaper is using Athena, if you're on AWS.
  - jitl 2y ago
    Athena packages Trino - it’s in part a managed Trino service.
    - Lucasoato 2y ago
      They are several versions behind, support for delta was added just recently. Also consider that with Trino you can build a cache layer on Alluxio, making it really fast (especially on NVMe disks).
2y ago

deleted 2y ago

[deleted]

adolph 2y ago

They seem to be doing lots of work but I don’t understand what customer value this creates. What does a backing data lake afford a Notion user that can’t be done in a similar product, like Obsidian?

jpalomaki 2y ago
From the article: "Unlock AI, Search, and other product use cases that require denormalized data"
SOLAR_FIELDS 2y ago
Beyond the features that the sibling comment mentioned, this kind of data isn’t really for end users. It’s a way that you can package it up, “anonymize” it, and sell the data to interested parties.
- sghiassy 2y ago
  Is that why they’re putting images in Postgres? I don’t understand that design decision yet.
  - bastawhiz 2y ago
    I... Don't think they are? If you look at the URL for images in notion, you can see the S3 hostname.
  - benaubin 2y ago
    Notion employee here. We don't put images themselves in Postgres- we use s3 to store them. The article is referring to image blocks, which are effectively pointers to the image.
- Ozzie_osman 2y ago
  For someone like Notion, they probably aren't selling this data. The primary use case is internally for analysis (eg product usage, business analysis, etc). It can also be used to train AI models, of course.

SOLAR_FIELDS 2y ago

They didn’t say the quiet part out loud, which is almost certainly that the Fivetran and Snowflake bills for what they were doing were probably enormous and those were undoubtedly what got management’s attention about fixing this.

deleted 2y ago
[deleted]
rorymalcolm 2y ago
Found this comment (from Fivetran's CEO, so, with that in mind) regarding this article enlightening regarding the costs they were facing here https://twitter.com/frasergeorgew/status/1808326803796512865 https://twitter.com/frasergeorgew/status/1808326803796512865
- pbd 2y ago
  Snowflake as destination is very very easy to work with on fivetran. Fivetran didn't have S3 as destination till late 2022. So it literally forces you to use one of BQ, Snowflake, redshift as destination. So fivetran CEO's defence is pretty stupid.
sneak 2y ago
I thought the quiet part was that they are data mining their customer data (and disclosing it to multiple third parties) because it’s not E2EE and they can read everyone’s private and proprietary notes. Otherwise, this is the perfect app for sharding/horizontal scalability. Your notes don’t need to be queried or joined with anyone else’s notes.
- redpoint 2y ago
  This ^. This switch from managed to in house is a good example of only building when necessary.
- altdataseller 2y ago
  Also whether this data lake is worth the costs/effort. How does this data lake add value to the user experience? What is this “AI” stuff that this data lake enables? For example, they mention search. But i imagine it is just searching only within your own docs. Which i presume should be fast and efficient if everything is sharded by user in Postgres. The tech stuff is all fine and good, but if it adds no value, its just playing with technology for technology sakes

methou 2y ago

> Data lake > Data warehouse These aren't something I would like to hear if I'm still using Notion. It's very bold to publish something like this on their own website.

bnj 2y ago
Could you expand on this?
- lopkeny12ko 2y ago
  What's there to expand on? Do you not realize how bad of a look it is for a company to publicly admit, on their own blog, the amount of time and engineering effort they spent to package up, move, analyze, and sell all their customer's private data? This is why laws like CCPA "do not sell my personal information" exist, which I certainly hope Notion is abiding by, otherwise they'll have lawyers knocking on their door soon.
  - Cthulhu_ 2y ago
    Where do they say they sell it? Citation needed; that's a legal and reputational minefield that I don't think they would admit to, like you said.
    - lopkeny12ko 2y ago
      I would challenge you to find any broker who sells data (like the T-Mobile location data scandal) who says plainly and clearly they sell user data.
      - quest88 2y ago
        This is not answering the question.
  - bnj 2y ago
    Right, yes, tone aside that’s very helpful- at first I didn’t understand the implication of the blog post for implementing customer hostile solutions, but you’ve helped me understand it now.

DataDaemon 2y ago

OK, thanks, when E2EE ?

deleted 2y ago

[deleted]

j45 2y ago

This was a nice read, interesting to see how far Postgres (largely alone) can get you. Also we see how at self hosting within a startup can make perfect sense. :) Devops that abstract away things in some cases to the cloud might just add to architectural and technical debt later, without the history of learning from working through the challenges Still, it might have been a great opportunity to figure out offline first use of notion. I have been forced to use anytype instead of notion for the offline first reason. Time to checkout to learn how they handle storage from the source code.

wejick 2y ago

I'm not familiar with S3 on datalake setup. When replicating a db table to S3, what format will be used? And I'm wondering if it's possible to update the S3 files to reflect latest incoming changes on the db table?

mritchie712 2y ago
The file format is often Parquet. The "table format" depend on what data lake you're using (e.g. Iceberg, Delta, etc.). If you know Python, here's[0] a practical example of how Iceberg works. 0 - https://www.definite.app/blog/iceberg-query-engine https://www.definite.app/blog/iceberg-query-engine

middlefing 2y ago

[flagged]

marviel 2y ago
what do you dislike specifically? They've invested a lot of energy into it when last I checked
- TheDong 2y ago
  They advertise markdown support, the ability to export markdown, and the ability to import markdown... However, what they don't say, is that the export and import format aren't compatible and are different subsets of markdown with different features. If I export a notion page as markdown, then re-import that same markdown document back into notion, I get something wildly different. All I want is to not use the notion editor (which lags and sometimes crashes my browser), and to instead use my local text editor which has served me well for everything else. Failing at that, I want to edit plain text, like I can in github comments or wikipedia pages. Like, the fact that if I write 'foo`', and then go back and edit a backtick in before the word 'foo', it doesn't code-format it, but if I do it in the opposite order and type '`foo`', it code-formats it, makes it very clear I'm not editing text, and there is weird hidden state, which is annoying to reason about. Just let me edit something like markdown directly, with an optional preview window somewhere, and that would _also_ be vastly better than the mess they have.
- middlefing 2y ago
  That's probably the problem. It's over engineered and just does things I didn't want or need it to do. I just want to type words and paste things in without having a bunch of bullshit happen.
deleted 2y ago
[deleted]

CyberDildonics 2y ago

What's the difference between a data lake and a database with a filesystem?

anthomtb 2y ago
Add data warehouse to this list too.
OJFord 2y ago
Normalisation, relations, the filesystem, ..? I'm more curious about your view of a 'data lake' which is met by adding a filesystem to a db?
- CyberDildonics 2y ago
  Normalisation, relations, What does that mean?
syndicatedjelly 2y ago
I thought data lake was more of an advertising phrase, to show off how much data you have/can handle
- Temporary_31337 2y ago
  Web scale

crux 2y ago

Hi all—I'm the EM for the Search team at Notion, and I want to chime in to clear up one unfortunate misconception I've seen a few times in this thread. Notion does not sell its users' data. Instead, I want to expand on one of the first use-cases for the Notion data lake, which was by my team. This is an elaboration of the description in TFA under the heading "Use case support". As is described there, Notion's block permissions are highly normalized at the source of truth. This is usually quite efficient and generally brings along all the benefits of normalization in application databases. However, we need to _denormalize_ all the permissions that relate to a specific document when we index it into our search index. When we transactionally reindex a document "online", this is no problem. However, when we need to reindex an entire search cluster from scratch, loading every ancestor of each page in order to collect all of its permissions is far too expensive. Thus, one of the primary needs that my team had from the new data lake is "tree traversal and permission data construction for each block". We rewrote our "offline" reindexer to read from the data lake instead of reading from RDS instances serving database snapshots. This allowed us to dramatically reduce the impact of iterating through every page when spinning up a new cluster (not to mention save a boatload in spinning up those ad-hoc RDS instances). I hope this miniature deep dive gives a little bit more color on the uses of this data store—as it is emphatically _not_ to sell our users' data!

jzelinskie 2y ago
This is a fantastic post that explains a lot of the end product, but I'd love to hear more about the journey specifically on denormalizing permissions at Notion. Scaling out authorization logic like this is actually very under-documented in industry. Mind if I email you to chat? Full disclosure: I'm a founder of authzed (W21), the company building SpiceDB, an open source project inspired by Google's internal scalable authorization system. We offer a product that streams changes to fully denormalized permissions for search engines to consume, but I'm not trying to pitch; you just don't often hear about other solutions built in this space!
infogulch 2y ago
Interesting! Now I'm curious how you handle live permission changes and indexes with stale permission data.

hobobaggins 2y ago

> Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake. Are they using this new data lake to train new AI models on? Or has Notion signed a deal with another LLM provider to provide customer data as a source for training data?

gorgoiler 2y ago
It’s not a direct answer but from what Notion tell us about their own business: * The team are based in the US, specifically California, and Notion Labs, Inc is a Delaware corporation. * Their investment comes from Venture Capital and individual wealth. The investors are listed on Notion’s about page and are open about how they themselves became rich through VC funded tech companies. There is a very open sense of panic in tech right now to climb to the top of the AI pile and not get crushed underneath. I would be amazed if there were any companies not enthralled by — and either already embracing or planning to embrace — the data-mining AI gold rush. Notion is a great product but one would be naive to use it while also harboring concerns about data privacy.
jitl 2y ago
(I work at Notion, but not on the data platform team) We do not and will never sell customer data to anyone. We do not train AI models on customer data. As we state in our privacy policy for AI features (https://www.notion.so/notion/Notion-AI-Supplementary-Terms-fa9034c8b5a04818a6baf3eac2adddbb https://www.notion.so/notion/Notion-AI-Supplementary-Terms-f...): > Notion does not use your Customer Data or permit others to use your Customer Data to train the machine learning models used to provide Notion AI Writing Suite or Notion AI Q&A [added: our AI features]. Your use of Notion AI Writing Suite or Notion AI Q&A does not grant Notion any right or license to your Customer Data to train our machine learning models. We do use various data infrastructure, including Postgres and the data lake, to index customer content both with traditional search infrastructure like Elasticsearch, as well as AI-based embedding search like Pinecone. We do this so you can search your own content when you're using Notion. We wrote this article to explain how Notion's AI features works with customer data: https://www.notion.so/help/notion-ai-security-practices https://www.notion.so/help/notion-ai-security-practices

jauntywundrkind 2y ago

Side-ish note, I really enjoyed a submission on Bufstream recently, a Kafka mq replacement. One of the things they mentioned is that they are working on building in Iceberg materialization, so Bufstream can automatically handle building a big analytics data lake out of incoming data. It feels like that could potentially tackle a bunch of the stack here. https://buf.build/blog/bufstream-kafka-lower-cost https://buf.build/blog/bufstream-kafka-lower-cost https://news.ycombinator.com/item?id=40919279 https://news.ycombinator.com/item?id=40919279 Versus what Notion is doing: > We ingest incrementally updated data from Postgres to Kafka using Debezium CDC connectors, then use Apache Hudi, an open-source data processing and storage framework, to write these updates from Kafka to S3. Feels like it would work about the same with Bufstream, replacing both Kafka & Hudi. I've heard great things about Hudi but it does seem to have significantly less adoption so far.

alexliu518 2y ago

Thank you for the clarification! It's great to hear more about the efficient data management practices at Notion. Your team's innovative use of the data lake to streamline the reindexing process while ensuring user data privacy is impressive. Keep up the excellent work!

HermitX 2y ago

Great article, thank you for sharing! I have a question I’d like to discuss with the author. Spark SQL is a great product and works perfectly for batch processing tasks. However, for handling ad hoc query tasks or more interactive data analysis tasks, Spark SQL might have some performance issues. If you have such workloads, I suggest trying data lake query engines like Trino or StarRocks, which offer faster speeds and a better query experience.

jitl 2y ago
(Notion employee) AWS Athena packages Trino, I’ve been using it for some queries like “find all blocks that contain @-mentions”. It’s a great tool.