3 ms·

At the end of the day, all the data warehouses run on SQL, with a bit of customization around ingestion and export. Most of them are backed by object storage (

At the end of the day, all the data warehouses run on SQL, with a bit of customization around ingestion and export. Most of them are backed by object storage (S3/GCS) and those integrations look very similar. I wouldn't be that worried about lock-in or being made obsolete. Business logic is going to be pretty easy to port between Redshift, BigQuery, Snowflake, or whatever comes next.

worker767424 6y ago
> Most of them are backed by object storage (S3/GCS) Redshift is backed by worker instances that have their own stores in what's basically an EC2 instance. It's definitely not backed by S3 like Athena. Bigquery and GCS are both built on top of Colossus, but they have different layers in between them.
- AtlasLion 6y ago
  Same applies to Teradata vantage on cloud.
- bpodgursky 6y ago
  Sorry, probably should have been more precise. Meant to say: most users are going to interact with the warehouses via object storage for import and export of data. Since the object store APIs are almost identical across platforms, it doesn't matter that much which warehouse you actually use for production work. It's something that does massive SQL, imports data from S3, and exports data to S3.
  - tomnipotent 6y ago
    > most users are going to interact with the warehouses via object storage for import and export of data. No, most are going to be using SQL IDE's to query and export data.
- EwanToo 6y ago
  With the newer Redshift ra3 instances you use S3 backed storage with local SSD caching https://aws.amazon.com/redshift/features/ra3/ https://aws.amazon.com/redshift/features/ra3/
tomnipotent 6y ago
> going to be pretty easy to port between Redshift, BigQuery, Snowflake, or whatever comes next. This isn't even remotely true. Each has unique SQL syntax, and once you have few hundred or thousand queries written using vendor-specific SQL (be it date functions or JSON), it is non-trivial to migrate.

worker767424 6y ago

> Most of them are backed by object storage (S3/GCS) Redshift is backed by worker instances that have their own stores in what's basically an EC2 instance. It's definitely not backed by S3 like Athena. Bigquery and GCS are both built on top of Colossus, but they have different layers in between them.

AtlasLion 6y ago
Same applies to Teradata vantage on cloud.
bpodgursky 6y ago
Sorry, probably should have been more precise. Meant to say: most users are going to interact with the warehouses via object storage for import and export of data. Since the object store APIs are almost identical across platforms, it doesn't matter that much which warehouse you actually use for production work. It's something that does massive SQL, imports data from S3, and exports data to S3.
- tomnipotent 6y ago
  > most users are going to interact with the warehouses via object storage for import and export of data. No, most are going to be using SQL IDE's to query and export data.
EwanToo 6y ago
With the newer Redshift ra3 instances you use S3 backed storage with local SSD caching https://aws.amazon.com/redshift/features/ra3/ https://aws.amazon.com/redshift/features/ra3/

tomnipotent 6y ago

> going to be pretty easy to port between Redshift, BigQuery, Snowflake, or whatever comes next. This isn't even remotely true. Each has unique SQL syntax, and once you have few hundred or thousand queries written using vendor-specific SQL (be it date functions or JSON), it is non-trivial to migrate.