5 ms·

Agreed. For certain use cases (session store, user profile store), the lower cost of memory and the relatively slow growth of data will mean that they will usua

Agreed. For certain use cases (session store, user profile store), the lower cost of memory and the relatively slow growth of data will mean that they will usually fit in memory. But, in my experience, there's a larger, and growing, class of use cases (ML, IOT, Fraud detection, Gaming data) where both the amount of raw data captured and the expectations for data retention are causing the total dataset to spiral into the dozens or hundreds of TB, with expectations for real-time SLAs on query. That's what I was talking about.

amelius 9y ago
Yes, but data on the order of TBs is usually stored on platters (as opposed to SSD). In that case, a fast data access library will not buy you much, because the disk will be slow anyway. (By the way it is common that an in-memory database is combined with on-disk storage for large blobs; those blobs are just flat files, and the OS already offers the fastest way to access them without any library)
- seastarer 9y ago
  Multi-Terabyte SSDs are very common. An i3.16xlarge instance on AWS, for example, has 15TB of SSD storage.
  - amelius 9y ago
    Ok, but it doesn't take away the point that very often the database can still be stored in memory, except for large blobs which are stored in flat files. Access of those files is usually sequential, so you will not need a special library.
    - seastarer 9y ago
      It happens sometimes. Very often? Not in my experience
    - biggestdummy 9y ago
      Yes, video streaming and management is a use case as you describe. Metadata is kept in memory, and the video files, themselves, are stored on an inexpensive media - usually as file system. So queries about "what can i watch" are quick, but loading the video can take seconds. But any use case where you want to keep state over time - sums, averages, maxes, histories - requires you to keep all the individual points of data in the database. Those are the (increasingly common) use cases that I run into where very fast disk storage like Scylla are important.

amelius 9y ago

Yes, but data on the order of TBs is usually stored on platters (as opposed to SSD). In that case, a fast data access library will not buy you much, because the disk will be slow anyway. (By the way it is common that an in-memory database is combined with on-disk storage for large blobs; those blobs are just flat files, and the OS already offers the fastest way to access them without any library)

seastarer 9y ago
Multi-Terabyte SSDs are very common. An i3.16xlarge instance on AWS, for example, has 15TB of SSD storage.
- amelius 9y ago
  Ok, but it doesn't take away the point that very often the database can still be stored in memory, except for large blobs which are stored in flat files. Access of those files is usually sequential, so you will not need a special library.
  - seastarer 9y ago
    It happens sometimes. Very often? Not in my experience
  - biggestdummy 9y ago
    Yes, video streaming and management is a use case as you describe. Metadata is kept in memory, and the video files, themselves, are stored on an inexpensive media - usually as file system. So queries about "what can i watch" are quick, but loading the video can take seconds. But any use case where you want to keep state over time - sums, averages, maxes, histories - requires you to keep all the individual points of data in the database. Those are the (increasingly common) use cases that I run into where very fast disk storage like Scylla are important.