4 ms·
None of that matters. It's a hundred gigs. You can store it in a textfile and read it in its entirety if you want. It fits in RAM. It is perfectly reasonable t
by xorcist 8y ago
None of that matters. It's a hundred gigs. You can store it in a textfile and read it in its entirety if you want. It fits in RAM.
It is perfectly reasonable to store this in a database. If and when you change your mind about the data format you can just scrap it and start over.
- aeroevan 8y agoIt's 100GB compressed. Parquet does a very good job of compressing most data so that's where the estimate of 10x (so 1TB) uncompressed was mentioned as a rule of thumb. Parquet also supports much better access mechanisms, like being able to deserialize a single column without having to read in entire rows. But like you mentioned, 1TB of data in a traditional database isn't that bad.
- usgroup 8y ago... also remembering that a traditional dB will typically not store data raw. Row compression is normal and disk compression is normal . The typical column store advantage is block compression, predicate pushdown and column order storage.
- sixdimensional 8y agoRegular databases such as SQL Server and Oracle have had columnar compression built in as an option along with the row stores for years now. I use it in SQL Server a lot and it works great.
- riku_iki 8y agoyou can run sql DB over compressed filesystem, and some DBs allow you to compress tables too > like being able to deserialize a single column without having to read in entire rows. and it reads filesystem's whole page anyway
- aeroevan 8y agoSorry for the late reply, but parquet is a columnar format so if it's big enough data, you should have multiple pages/blocks of data in a single column for a specific row group, and then be able to seek to the next row group and sequentially read the next set of blocks.