3 ms·
How many TBs of data are we really talking here, if it's just 6M rows? Surely this processing could easily be done on a single machine.
by sagarm 5y ago
How many TBs of data are we really talking here, if it's just 6M rows? Surely this processing could easily be done on a single machine.
- sztanko 5y agoIt doesn't say it is just 6m rows. It is 6m patient, which only hints that one of the dimension tables is 6m. Facts gathered in patients might be significantly larger. Also, my experience is saying, if you have hundreds of queries running simultaneously, it is not the volume of data that can be a bottleneck. Depending on the system it can be anything, starting from acquiring lock on a record or initiating a transaction.
- sagarm 5y agoTrue, the number of events / records is probably significantly larger than 6M. I still have a hard time believing any reasonably modern datawarehouse would struggle with queries on this dataset. If you've got 1400 analysts, I hope you've exported your data from an OLTP database to an OLAP database. Those generally are much easier to scale to many users.
- ivalm 5y agoI work for a large healthcare org and from personal experience these things can get large. 6m patients is prob 20m encounters, each encounter may have 10 different kinds of meta data (each one it’s own table) and each kind has 10-100 rows. So really you are doing joins between 10 tables each with ~1-10 billion rows. It actually does slow down even running on expensive hardware.
- sim_singh 5y agoHi, I saw your comment on the wechat page, I was wondering if you have the account and can verify me?
- sagarm 5y agoAh yes that does sound painful. I'd generally recommend that data be denormalized in the warehouse so the joins don't need to be done at query time. That can make the queries more complex, however.