4 ms·

Those are just different words for "database". What do you care what kind of database your Notion data is sitting in?

TeMPOraL 2y ago
A "data lake" strongly suggests there's lot of information the company needs to aggregate and process globally, which should very much not be the case with a semi-private rich notebook product.
- deleted 2y ago
  [deleted]
- bastawhiz 2y ago
  They literally explained in the article why they have a data lake instead of just a data warehouse: their data model means it's slow and expensive to ingest that data into the warehouse from Postgres. The data lake is serving the same functions that the data warehouse did, but now that the volume of data has exceeded what the warehouse can handle, the data lake fills that gap. I wrote another comment about why you'd need this in the first place: https://news.ycombinator.com/item?id=40961622 https://news.ycombinator.com/item?id=40961622 Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. If you don't want your data processed for the features and services the company offers, don't use them.
  - anoncareer0212 2y ago
    > Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. Neutral party here: that's not what they said. A) Quotes shouldn't be there. B) Heuristic I've started applying to my comments: if I'm tempted to "quote" something that isn't a quote, it means I don't fully understand what they mean and should ask a question. This dovetails nicely with the spirit of HN's "come with curiosity" It is disquieting because: A) This are very much ill-defined terms (what, exactly, is data lake, vs. data warehouse, vs. database?), and as far as I've had to understand this stuff, and a quick spot check of Google shows, it's about making it so you're accumulating more data in one place. B) This is antithetical to a consumer's desired approach to data, which will described parodically as: stored individually, on one computer, behind 3 locked doors and 20 layers of encryption.
- nojvek 2y ago
  At the scale of Notion, with millions of users, they’d have that much data. I’ve seen 100TB+ workloads at smaller companies. Not unusual.
  - iLoveOncall 2y ago
    The concern isn't the scale, it's the use. What is there to _process_ when they're supposed to only store and retrieve to show to users?
    - ctippett 2y ago
      The data doesn't have to be the content of user's notes. Think of all the metadata they're likely collecting per user/notebook/interaction – the data's likely useful for things like flagging security events, calculating the graph of interconnected notes, indexing hashed content for search (or AI embeddings?) ... these are just a few use-cases that come to mind from the top of my head.
      - TeMPOraL 2y ago
        Of which security and stability seems like the only reasonable use cases. Indexing content for search globally? Embeddings? They just can't help themselves, can they? All that juicy data, can't possibly leave it alone.
    - bastawhiz 2y ago
      Great, you build only store and retrieve functionality. How: 1. Do you identify which types of content your users use the most? 2. Do you find users who are abusing your system? 3. Do you load and process data (even on a customer by customer basis) to fine tune models for the QA service that you offer as an optional upgrade? Especially when there could be gigabytes of data for a single customer 4. Identify corrupt data caused by a bug in your code that saves data to the db? You're not doing a full table scan over hundreds of billions of records across almost 500 logical shares in your production fleet These are just the examples I came up with off the dome. The job of the business is to operate on the data. If you can't even query it, you can't operate on it. Running a business is far more than just being a dumb CRUD API.
- deleted 2y ago
  [deleted]

A "data lake" strongly suggests there's lot of information the company needs to aggregate and process globally, which should very much not be the case with a semi-private rich notebook product.

deleted 2y ago
[deleted]
bastawhiz 2y ago
They literally explained in the article why they have a data lake instead of just a data warehouse: their data model means it's slow and expensive to ingest that data into the warehouse from Postgres. The data lake is serving the same functions that the data warehouse did, but now that the volume of data has exceeded what the warehouse can handle, the data lake fills that gap. I wrote another comment about why you'd need this in the first place: https://news.ycombinator.com/item?id=40961622 https://news.ycombinator.com/item?id=40961622 Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. If you don't want your data processed for the features and services the company offers, don't use them.
- anoncareer0212 2y ago
  > Frankly the argument "they shouldn't need to query the data in their system" is kind of silly. Neutral party here: that's not what they said. A) Quotes shouldn't be there. B) Heuristic I've started applying to my comments: if I'm tempted to "quote" something that isn't a quote, it means I don't fully understand what they mean and should ask a question. This dovetails nicely with the spirit of HN's "come with curiosity" It is disquieting because: A) This are very much ill-defined terms (what, exactly, is data lake, vs. data warehouse, vs. database?), and as far as I've had to understand this stuff, and a quick spot check of Google shows, it's about making it so you're accumulating more data in one place. B) This is antithetical to a consumer's desired approach to data, which will described parodically as: stored individually, on one computer, behind 3 locked doors and 20 layers of encryption.
nojvek 2y ago
At the scale of Notion, with millions of users, they’d have that much data. I’ve seen 100TB+ workloads at smaller companies. Not unusual.
- iLoveOncall 2y ago
  The concern isn't the scale, it's the use. What is there to _process_ when they're supposed to only store and retrieve to show to users?
  - ctippett 2y ago
    The data doesn't have to be the content of user's notes. Think of all the metadata they're likely collecting per user/notebook/interaction – the data's likely useful for things like flagging security events, calculating the graph of interconnected notes, indexing hashed content for search (or AI embeddings?) ... these are just a few use-cases that come to mind from the top of my head.
    - TeMPOraL 2y ago
      Of which security and stability seems like the only reasonable use cases. Indexing content for search globally? Embeddings? They just can't help themselves, can they? All that juicy data, can't possibly leave it alone.
  - bastawhiz 2y ago
    Great, you build only store and retrieve functionality. How: 1. Do you identify which types of content your users use the most? 2. Do you find users who are abusing your system? 3. Do you load and process data (even on a customer by customer basis) to fine tune models for the QA service that you offer as an optional upgrade? Especially when there could be gigabytes of data for a single customer 4. Identify corrupt data caused by a bug in your code that saves data to the db? You're not doing a full table scan over hundreds of billions of records across almost 500 logical shares in your production fleet These are just the examples I came up with off the dome. The job of the business is to operate on the data. If you can't even query it, you can't operate on it. Running a business is far more than just being a dumb CRUD API.
deleted 2y ago
[deleted]