5 ms·
Loading data from cold storage (HDD/SSD, etc) into RAM only has to be done once on startup, though. After that single slow startup, all of the additional querie
by t0mbstone 8y ago
Loading data from cold storage (HDD/SSD, etc) into RAM only has to be done once on startup, though. After that single slow startup, all of the additional queries happen super fast.
Your complaint only makes sense if you only intend to perform one single query against a dataset.
- paulsutter 8y agoMy complaint makes sense because I might have 100B records per day to process, and the delay to load them up is significant
- wmf 8y agoCould you stream the data into memory as it is created so that loading data isn't in the critical path?
- AboutTheWhisles 8y agoDoes this software do that?
- gricardo99 8y agoYes, that’s common use case for kdb+
- edf13 8y agoCommon for any analytical setup... you'd integrate with many of the other stream processing engines available (Storm, Flink, Kafka, Spark... all depending on what you want to do).
- skgoa 8y agoThat assumes that we have enough RAM to hold all of the data we could ever want to analyse and that we have this data at startup. Neither of which has ever been the case as far as I have personally experienced.
- xgbi 8y agoMy friend works for $financial_subcontractor and he told me they have an XmX of 1,46TB on their servers so that they can load the data needed for market analysis. The response time is quite fantastic (less than a minute), but there are drawbacks: - GC takes in the order of minutes to complete - loading the data takes 90 min from spinning rust So yeah, it's possible, but your bottleneck becomes the storage driver.
- neuro 8y agoI've seen that setup in a number of hedge funds
- majewsky 8y agoIt's not uncommon for in-memory database servers to have installed RAM in the order of multiple terabytes. Source: I work at SAP's internal cloud team. For our internal customers running SAP HANA, we offer baremetal machines with up to 6 TiB RAM.