4 ms·

Correct title is "How to Analyze Billions of Records in 20 Minutes and One Second" All of these "fast" databases have a fatal flaw - it takes forever to load t

Correct title is "How to Analyze Billions of Records in 20 Minutes and One Second" All of these "fast" databases have a fatal flaw - it takes forever to load the data in the first place. In this case loading takes >1000x longer than the query so loading is all that matters.

placebo 8y ago
Have you experienced this issue with the other databases mentioned in the article?
- chiph 8y ago
  I think he was implying that if you access the disk at all, your performance is hidden behind that cost.
t0mbstone 8y ago
Loading data from cold storage (HDD/SSD, etc) into RAM only has to be done once on startup, though. After that single slow startup, all of the additional queries happen super fast. Your complaint only makes sense if you only intend to perform one single query against a dataset.
- paulsutter 8y ago
  My complaint makes sense because I might have 100B records per day to process, and the delay to load them up is significant
  - wmf 8y ago
    Could you stream the data into memory as it is created so that loading data isn't in the critical path?
    - AboutTheWhisles 8y ago
      Does this software do that?
      - gricardo99 8y ago
        Yes, that’s common use case for kdb+
      - edf13 8y ago
        Common for any analytical setup... you'd integrate with many of the other stream processing engines available (Storm, Flink, Kafka, Spark... all depending on what you want to do).
- skgoa 8y ago
  That assumes that we have enough RAM to hold all of the data we could ever want to analyse and that we have this data at startup. Neither of which has ever been the case as far as I have personally experienced.
  - xgbi 8y ago
    My friend works for $financial_subcontractor and he told me they have an XmX of 1,46TB on their servers so that they can load the data needed for market analysis. The response time is quite fantastic (less than a minute), but there are drawbacks: - GC takes in the order of minutes to complete - loading the data takes 90 min from spinning rust So yeah, it's possible, but your bottleneck becomes the storage driver.
    - neuro 8y ago
      I've seen that setup in a number of hedge funds
  - majewsky 8y ago
    It's not uncommon for in-memory database servers to have installed RAM in the order of multiple terabytes. Source: I work at SAP's internal cloud team. For our internal customers running SAP HANA, we offer baremetal machines with up to 6 TiB RAM.
bduerst 8y ago
Isn't this how all indexing works? Heavy fixed processing costs up front with the benefit of light variable processing moving forward.
zeroxfe 8y ago
That isn't a flaw, it's a constraint, which applies to practically all databases. Good databases are the ones that can optimize around these constraints for arbitrary data analysis and manipulation (e.g., by caching, read-ahead, auto-indexing, deduplication, aggregation, batching, contention-avoidance etc. etc.) There's a lot more to data analysis than loading data off disk.
edf13 8y ago
I'd down vote this if I could... It's far from the correct title. A good part of the optimisation of all modern analysis DBs is in-memory... this is no different and as such he's compared his work against 2 of the major other systems in the space. I see a lot of critics here... I think the guy should be congratulated for his work - which if you've read through all pages, appears to be done single handed!

placebo 8y ago

Have you experienced this issue with the other databases mentioned in the article?

chiph 8y ago
I think he was implying that if you access the disk at all, your performance is hidden behind that cost.

t0mbstone 8y ago

Loading data from cold storage (HDD/SSD, etc) into RAM only has to be done once on startup, though. After that single slow startup, all of the additional queries happen super fast. Your complaint only makes sense if you only intend to perform one single query against a dataset.

paulsutter 8y ago
My complaint makes sense because I might have 100B records per day to process, and the delay to load them up is significant
- wmf 8y ago
  Could you stream the data into memory as it is created so that loading data isn't in the critical path?
  - AboutTheWhisles 8y ago
    Does this software do that?
    - gricardo99 8y ago
      Yes, that’s common use case for kdb+
    - edf13 8y ago
      Common for any analytical setup... you'd integrate with many of the other stream processing engines available (Storm, Flink, Kafka, Spark... all depending on what you want to do).
skgoa 8y ago
That assumes that we have enough RAM to hold all of the data we could ever want to analyse and that we have this data at startup. Neither of which has ever been the case as far as I have personally experienced.
- xgbi 8y ago
  My friend works for $financial_subcontractor and he told me they have an XmX of 1,46TB on their servers so that they can load the data needed for market analysis. The response time is quite fantastic (less than a minute), but there are drawbacks: - GC takes in the order of minutes to complete - loading the data takes 90 min from spinning rust So yeah, it's possible, but your bottleneck becomes the storage driver.
  - neuro 8y ago
    I've seen that setup in a number of hedge funds
- majewsky 8y ago
  It's not uncommon for in-memory database servers to have installed RAM in the order of multiple terabytes. Source: I work at SAP's internal cloud team. For our internal customers running SAP HANA, we offer baremetal machines with up to 6 TiB RAM.

bduerst 8y ago

Isn't this how all indexing works? Heavy fixed processing costs up front with the benefit of light variable processing moving forward.

zeroxfe 8y ago

That isn't a flaw, it's a constraint, which applies to practically all databases. Good databases are the ones that can optimize around these constraints for arbitrary data analysis and manipulation (e.g., by caching, read-ahead, auto-indexing, deduplication, aggregation, batching, contention-avoidance etc. etc.) There's a lot more to data analysis than loading data off disk.

edf13 8y ago

I'd down vote this if I could... It's far from the correct title. A good part of the optimisation of all modern analysis DBs is in-memory... this is no different and as such he's compared his work against 2 of the major other systems in the space. I see a lot of critics here... I think the guy should be congratulated for his work - which if you've read through all pages, appears to be done single handed!