3 ms·

OLAP is kind of a different world - none of the typical Postgres "competition" (like MySQL or Oracle) works really any better in this domain. There really isn'

by glogla 6y ago

OLAP is kind of a different world - none of the typical Postgres "competition" (like MySQL or Oracle) works really any better in this domain. There really isn't a very good free one part solution here - so either you pay big bucks for the likes of Google BigQuery or Snowflake so they can become gatekeepers to your own data, or you end up burning a lot of engineering time to get the likes of Hadoop or Spark on K8s or Trino working.

mb7733 6y ago
In common usage does OLAP automatically imply data that is too large for a single node to handle? I'm wondering if a lot of OLAP workloads couldn't be handled with some parquet files (or some other column based storage)? I realize parquet is used by the hadoop/spark ecosystem, but do you really need those systems? I'm thinking that a lot of companies reach for a hadoop cluster when some parquet files in a regular for system would be much simpler. I've done things like this and in my experience it works quite well. But only for personal projects.
- glogla 6y ago
  No, of course you're right, I'm just bit too comfortable in my own domain. For smaller data (and here that would mean few hundreds GB which is pretty huge by normal standards) OLTP databases would cover you pretty well. Oracle has some fancy bitmap indexing, MS SQL even has columnar tables, and pretty much anything has partitioning. Parquet is really cool - especially there's like 30 x ratio between "generic oracle table" and gzip compressed parquet file so scan time are really in a different world. But by itself parquet doesn't solve a whole lot - where are the files stored and what scans them? What happens if someone is updating the files while someone else is reading them?
  - mb7733 6y ago
    Yeah the storage size was a real clincher for me, as much as I appreciate a SQL database. Re: concurrent reading and writing, for my use case the files are immutable so that isn't a concern. But I agree, I don't think pure parquet is a good fit there.

by glogla 6y ago

mb7733 6y ago

In common usage does OLAP automatically imply data that is too large for a single node to handle? I'm wondering if a lot of OLAP workloads couldn't be handled with some parquet files (or some other column based storage)? I realize parquet is used by the hadoop/spark ecosystem, but do you really need those systems? I'm thinking that a lot of companies reach for a hadoop cluster when some parquet files in a regular for system would be much simpler. I've done things like this and in my experience it works quite well. But only for personal projects.

glogla 6y ago
No, of course you're right, I'm just bit too comfortable in my own domain. For smaller data (and here that would mean few hundreds GB which is pretty huge by normal standards) OLTP databases would cover you pretty well. Oracle has some fancy bitmap indexing, MS SQL even has columnar tables, and pretty much anything has partitioning. Parquet is really cool - especially there's like 30 x ratio between "generic oracle table" and gzip compressed parquet file so scan time are really in a different world. But by itself parquet doesn't solve a whole lot - where are the files stored and what scans them? What happens if someone is updating the files while someone else is reading them?
- mb7733 6y ago
  Yeah the storage size was a real clincher for me, as much as I appreciate a SQL database. Re: concurrent reading and writing, for my use case the files are immutable so that isn't a concern. But I agree, I don't think pure parquet is a good fit there.