5 ms·

> Data lake > Data warehouse These aren't something I would like to hear if I'm still using Notion. It's very bold to publish something like this on their own

by methou 2y ago

> Data lake > Data warehouse These aren't something I would like to hear if I'm still using Notion. It's very bold to publish something like this on their own website.

bnj 2y ago
Could you expand on this?
- lopkeny12ko 2y ago
  What's there to expand on? Do you not realize how bad of a look it is for a company to publicly admit, on their own blog, the amount of time and engineering effort they spent to package up, move, analyze, and sell all their customer's private data? This is why laws like CCPA "do not sell my personal information" exist, which I certainly hope Notion is abiding by, otherwise they'll have lawyers knocking on their door soon.
  - Cthulhu_ 2y ago
    Where do they say they sell it? Citation needed; that's a legal and reputational minefield that I don't think they would admit to, like you said.
    - lopkeny12ko 2y ago
      I would challenge you to find any broker who sells data (like the T-Mobile location data scandal) who says plainly and clearly they sell user data.
      - quest88 2y ago
        This is not answering the question.
  - bnj 2y ago
    Right, yes, tone aside that’s very helpful- at first I didn’t understand the implication of the blog post for implementing customer hostile solutions, but you’ve helped me understand it now.
bastawhiz 2y ago
Those are just different words for "database". What do you care what kind of database your Notion data is sitting in?
- TeMPOraL 2y ago
  A "data lake" strongly suggests there's lot of information the company needs to aggregate and process globally, which should very much not be the case with a semi-private rich notebook product.
  - deleted 2y ago
    [deleted]
  - bastawhiz 2y ago
    They literally explained in the article why they have a data lake instead of just a data warehouse: their data model means it's slow and expensive to ingest that data into the warehouse from Postgres. The data lake is serving the same functions that the data warehouse did, but now that the volume of data has exceeded what the warehouse can handle, the data lake fills that gap. I wrote another comment about why you'd need this in the first place: https://news.ycombinator.com/item?id=40961622 https://news.ycombinator.com/item?id=40961622 Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. If you don't want your data processed for the features and services the company offers, don't use them.
    - anoncareer0212 2y ago
      > Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. Neutral party here: that's not what they said. A) Quotes shouldn't be there. B) Heuristic I've started applying to my comments: if I'm tempted to "quote" something that isn't a quote, it means I don't fully understand what they mean and should ask a question. This dovetails nicely with the spirit of HN's "come with curiosity" It is disquieting because: A) This are very much ill-defined terms (what, exactly, is data lake, vs. data warehouse, vs. database?), and as far as I've had to understand this stuff, and a quick spot check of Google shows, it's about making it so you're accumulating more data in one place. B) This is antithetical to a consumer's desired approach to data, which will described parodically as: stored individually, on one computer, behind 3 locked doors and 20 layers of encryption.
zarmin 2y ago
Given how infuriating their implementation is of an in-app database, perhaps it's not that surprising.
wodenokoto 2y ago
That’s definitely something you want to do. Datalake can be home for raw and lightly refined data in an “analytics” database such as big query or just raw parquets. This is fast for large queries but slow for small queries. So you want refined data in a “regular” database like Postgres or mssql to serve all the dashboards.

by methou 2y ago

> Data lake > Data warehouse These aren't something I would like to hear if I'm still using Notion. It's very bold to publish something like this on their own website.

bnj 2y ago

Could you expand on this?

lopkeny12ko 2y ago
What's there to expand on? Do you not realize how bad of a look it is for a company to publicly admit, on their own blog, the amount of time and engineering effort they spent to package up, move, analyze, and sell all their customer's private data? This is why laws like CCPA "do not sell my personal information" exist, which I certainly hope Notion is abiding by, otherwise they'll have lawyers knocking on their door soon.
- Cthulhu_ 2y ago
  Where do they say they sell it? Citation needed; that's a legal and reputational minefield that I don't think they would admit to, like you said.
  - lopkeny12ko 2y ago
    I would challenge you to find any broker who sells data (like the T-Mobile location data scandal) who says plainly and clearly they sell user data.
    - quest88 2y ago
      This is not answering the question.
- bnj 2y ago
  Right, yes, tone aside that’s very helpful- at first I didn’t understand the implication of the blog post for implementing customer hostile solutions, but you’ve helped me understand it now.

bastawhiz 2y ago

Those are just different words for "database". What do you care what kind of database your Notion data is sitting in?

TeMPOraL 2y ago
A "data lake" strongly suggests there's lot of information the company needs to aggregate and process globally, which should very much not be the case with a semi-private rich notebook product.
- deleted 2y ago
  [deleted]
- bastawhiz 2y ago
  They literally explained in the article why they have a data lake instead of just a data warehouse: their data model means it's slow and expensive to ingest that data into the warehouse from Postgres. The data lake is serving the same functions that the data warehouse did, but now that the volume of data has exceeded what the warehouse can handle, the data lake fills that gap. I wrote another comment about why you'd need this in the first place: https://news.ycombinator.com/item?id=40961622 https://news.ycombinator.com/item?id=40961622 Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. If you don't want your data processed for the features and services the company offers, don't use them.
  - anoncareer0212 2y ago
    > Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. Neutral party here: that's not what they said. A) Quotes shouldn't be there. B) Heuristic I've started applying to my comments: if I'm tempted to "quote" something that isn't a quote, it means I don't fully understand what they mean and should ask a question. This dovetails nicely with the spirit of HN's "come with curiosity" It is disquieting because: A) This are very much ill-defined terms (what, exactly, is data lake, vs. data warehouse, vs. database?), and as far as I've had to understand this stuff, and a quick spot check of Google shows, it's about making it so you're accumulating more data in one place. B) This is antithetical to a consumer's desired approach to data, which will described parodically as: stored individually, on one computer, behind 3 locked doors and 20 layers of encryption.

zarmin 2y ago

Given how infuriating their implementation is of an in-app database, perhaps it's not that surprising.

wodenokoto 2y ago

That’s definitely something you want to do. Datalake can be home for raw and lightly refined data in an “analytics” database such as big query or just raw parquets. This is fast for large queries but slow for small queries. So you want refined data in a “regular” database like Postgres or mssql to serve all the dashboards.