4 ms·
This split of compute and storage into distinct layers is starting to become more common, and it has some pretty neat advantages like much better scalability an
by manigandham 9y ago
This split of compute and storage into distinct layers is starting to become more common, and it has some pretty neat advantages like much better scalability and efficiency.
Google's BigQuery and Snowflake Data are examples of data warehouses, similar to DIY presto/drill/spark on S3. Apache Pulsar brings that to messaging and distributed logs. It'll be interesting to see how it applies to more OLTP database engines, although there are examples like TiDB which seem to work well enough.
- vgt 9y agoBigQuery specifically pioneered many of these concepts since it's release in 2012 - "serverless manageability", pay-per-query consumption-based pricing, pure separation of compute and storage (no intermediate ssh mesh), and more recently separation of compute and intermediate state (wonders for scalability and complex query performance). (work at G and used to work on BQ)
- hueving 9y agoSeparation of compute and storage has been around since before Google was even relevant as a company (see NFS). If you think bigquery pioneered all of these concepts then your team did a very poor job of researching prior art. Maybe that was intentional for a good green-field design, but it's certainly not pioneering at that point.