2 ms·

My queston is on scalability. You say "large datasets" on the website. What is large? 1x/10x/100x Terabytes? 1x/10x/100x Petabytes? What kind of access rat

• aboodman 10y ago
  Honest answer is: we don't know yet -- we're working our way up from the bottom. But we (cautiously) don't see any reason why the basic design shouldn't scale to very large (e.g. petabyte) datasets, and that is our eventual goal. That said, we do think there are a lot (even maybe the majority) of use cases in the GB-TB range.
  • aartur 10y ago
    Isn't the append-only design unsuitable for scenarios where many updates/deletes are made? If you update/delete 1GB of your 2GB database each day, then after a year the database is 365GB in size, but the live data is only 2GB. I think the git-like features (history, merging) are very helpful for internal work, but when the dataset must be published, I think in most cases only the newest snapshot should be made available. But then the question is what format should it have...?
    • aboodman 10y ago
      It just depends on the details. If you have a dataset in which 50% of values changes every day, and it doesn't compress well, then yeah, your Noms archive of that entire dataset is going to grow quickly. In such situations, you could either (eventually, when it is implemented) prune old data, or aggregate the changes into bigger blocks.