4 ms·

I'm curious how scalable this is. If it can't go beyond what you can load into memory, it doesn't seem that useful. If you already know SQL, you could just as

by scribu 7y ago

I'm curious how scalable this is. If it can't go beyond what you can load into memory, it doesn't seem that useful. If you already know SQL, you could just as well use Pandas (the Python library) to load data from various sources and query it. Also: AWS Athena

foxish 7y ago
Hi scribu, I'm Anirudh from the product team at Rockset. The data is indexed onto SSDs in the cloud. The sweet spot is 10s of terabytes of data that you want to build a live application on top of. From an architectural standpoint, we can scale even further. https://rockset.com/Rockset_Concepts_Design_Architecture.pdf https://rockset.com/Rockset_Concepts_Design_Architecture.pdf
- foxish 7y ago
  In terms of use-cases, we see things differently from a product like Athena. We are focused on ETL-free real-time analytics and applications which make use of the fact that we construct multiple indexes automatically behind the scenes to enable low latency and scalable query serving.
- manigandham 7y ago
  1TB costs $6000/month at your pricing. 10s of TB is rather expensive. It's great if you can get those rates but I'm having trouble seeing how these numbers can work on large datasets.
  - Aeolun 7y ago
    If you are live querying 1TB of data in memory, $6000/month does not seem that crazy at all.
    - manigandham 7y ago
      It's in memory now? The prior comment says SSDs.
  - foxish 7y ago
    You are right in that it uses SSD and indexes under the hood to power the SQL queries. There are volume discounts available at higher storage volumes in the 10s of TB range, so, the cost wouldn't scale linearly. Rockset builds multiple indexes to enable low latency SQL queries that can be served directly into applications. This might not make sense for say - storing a lot of log data and querying it rarely, and makes more sense for data that needs to be actively queried with low latency. (I work on product at Rockset)

by scribu 7y ago

foxish 7y ago

Hi scribu, I'm Anirudh from the product team at Rockset. The data is indexed onto SSDs in the cloud. The sweet spot is 10s of terabytes of data that you want to build a live application on top of. From an architectural standpoint, we can scale even further. https://rockset.com/Rockset_Concepts_Design_Architecture.pdf https://rockset.com/Rockset_Concepts_Design_Architecture.pdf

foxish 7y ago
In terms of use-cases, we see things differently from a product like Athena. We are focused on ETL-free real-time analytics and applications which make use of the fact that we construct multiple indexes automatically behind the scenes to enable low latency and scalable query serving.
manigandham 7y ago
1TB costs $6000/month at your pricing. 10s of TB is rather expensive. It's great if you can get those rates but I'm having trouble seeing how these numbers can work on large datasets.
- Aeolun 7y ago
  If you are live querying 1TB of data in memory, $6000/month does not seem that crazy at all.
  - manigandham 7y ago
    It's in memory now? The prior comment says SSDs.
- foxish 7y ago
  You are right in that it uses SSD and indexes under the hood to power the SQL queries. There are volume discounts available at higher storage volumes in the 10s of TB range, so, the cost wouldn't scale linearly. Rockset builds multiple indexes to enable low latency SQL queries that can be served directly into applications. This might not make sense for say - storing a lot of log data and querying it rarely, and makes more sense for data that needs to be actively queried with low latency. (I work on product at Rockset)