3 ms·

They seem to be doing lots of work but I don’t understand what customer value this creates. What does a backing data lake afford a Notion user that can’t be do

by adolph 2y ago

They seem to be doing lots of work but I don’t understand what customer value this creates. What does a backing data lake afford a Notion user that can’t be done in a similar product, like Obsidian?

jpalomaki 2y ago
From the article: "Unlock AI, Search, and other product use cases that require denormalized data"
SOLAR_FIELDS 2y ago
Beyond the features that the sibling comment mentioned, this kind of data isn’t really for end users. It’s a way that you can package it up, “anonymize” it, and sell the data to interested parties.
- sghiassy 2y ago
  Is that why they’re putting images in Postgres? I don’t understand that design decision yet.
  - bastawhiz 2y ago
    I... Don't think they are? If you look at the URL for images in notion, you can see the S3 hostname.
  - benaubin 2y ago
    Notion employee here. We don't put images themselves in Postgres- we use s3 to store them. The article is referring to image blocks, which are effectively pointers to the image.
- Ozzie_osman 2y ago
  For someone like Notion, they probably aren't selling this data. The primary use case is internally for analysis (eg product usage, business analysis, etc). It can also be used to train AI models, of course.
  - Cthulhu_ 2y ago
    That "probably" is doing a lot of heavy lifting. That said, whether they sell it or not, it's all that data that is their primary value store at the moment. They will either go public or sell, eventually. If they go public, it'll likely be similar to Dropbox; a single fairly successful product, but failing attempts to diversify.
    - TeMPOraL 2y ago
      "Selling" is a load-bearing word, too. They're probably not literally selling SQL dumps for hard cash. But there are many ways of indirectly selling data, that are almost equivalent to trading database dumps, but indirect enough that the company can say they're not selling data, and be technically correct.
ambicapter 2y ago
1st paragraph: "Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake."
bastawhiz 2y ago
The whole point of a data warehouse is that you can rapidly query a huge amount of data with ad hoc queries. When your data is in Postgres, running an arbitrary query might take hours or days (or longer). Postgres does very poorly for queries that read huge amounts of data when there's no preexisting index (and you're not going to be building one-off indexes for ad hoc queries—that defeats the point). A data warehouse is slower for basic queries but substantially faster for queries that run against terabytes or petabytes of data. I can imagine some use cases at Notion: - You want to know the most popular syntax highlighting languages - You're searching for data corruption, where blocks form a cycle - You're looking for users who are committing fraud or abuse (like using bots in violation of your tos)

by adolph 2y ago

jpalomaki 2y ago

From the article: "Unlock AI, Search, and other product use cases that require denormalized data"

SOLAR_FIELDS 2y ago

Beyond the features that the sibling comment mentioned, this kind of data isn’t really for end users. It’s a way that you can package it up, “anonymize” it, and sell the data to interested parties.

sghiassy 2y ago
Is that why they’re putting images in Postgres? I don’t understand that design decision yet.
- bastawhiz 2y ago
  I... Don't think they are? If you look at the URL for images in notion, you can see the S3 hostname.
- benaubin 2y ago
  Notion employee here. We don't put images themselves in Postgres- we use s3 to store them. The article is referring to image blocks, which are effectively pointers to the image.
Ozzie_osman 2y ago
For someone like Notion, they probably aren't selling this data. The primary use case is internally for analysis (eg product usage, business analysis, etc). It can also be used to train AI models, of course.
- Cthulhu_ 2y ago
  That "probably" is doing a lot of heavy lifting. That said, whether they sell it or not, it's all that data that is their primary value store at the moment. They will either go public or sell, eventually. If they go public, it'll likely be similar to Dropbox; a single fairly successful product, but failing attempts to diversify.
  - TeMPOraL 2y ago
    "Selling" is a load-bearing word, too. They're probably not literally selling SQL dumps for hard cash. But there are many ways of indirectly selling data, that are almost equivalent to trading database dumps, but indirect enough that the company can say they're not selling data, and be technically correct.

ambicapter 2y ago

1st paragraph: "Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake."

bastawhiz 2y ago

The whole point of a data warehouse is that you can rapidly query a huge amount of data with ad hoc queries. When your data is in Postgres, running an arbitrary query might take hours or days (or longer). Postgres does very poorly for queries that read huge amounts of data when there's no preexisting index (and you're not going to be building one-off indexes for ad hoc queries—that defeats the point). A data warehouse is slower for basic queries but substantially faster for queries that run against terabytes or petabytes of data. I can imagine some use cases at Notion: - You want to know the most popular syntax highlighting languages - You're searching for data corruption, where blocks form a cycle - You're looking for users who are committing fraud or abuse (like using bots in violation of your tos)