4 ms·
No, of course you're right, I'm just bit too comfortable in my own domain. For smaller data (and here that would mean few hundreds GB which is pretty huge by n
by glogla 6y ago
No, of course you're right, I'm just bit too comfortable in my own domain.
For smaller data (and here that would mean few hundreds GB which is pretty huge by normal standards) OLTP databases would cover you pretty well. Oracle has some fancy bitmap indexing, MS SQL even has columnar tables, and pretty much anything has partitioning.
Parquet is really cool - especially there's like 30 x ratio between "generic oracle table" and gzip compressed parquet file so scan time are really in a different world.
But by itself parquet doesn't solve a whole lot - where are the files stored and what scans them? What happens if someone is updating the files while someone else is reading them?
- mb7733 6y agoYeah the storage size was a real clincher for me, as much as I appreciate a SQL database. Re: concurrent reading and writing, for my use case the files are immutable so that isn't a concern. But I agree, I don't think pure parquet is a good fit there.