17 ms·

I've used Snowflake a fair amount. It's a decent product, probably on par with Redshift / BigQuery. Obviously theres a lot of hype and free money floating aroun

I've used Snowflake a fair amount. It's a decent product, probably on par with Redshift / BigQuery. Obviously theres a lot of hype and free money floating around but my take on why they are popular is that they are basically a replacement for large Hadoop installations that have become untenable to manage over the past decade. If a company is already using Redshift or BigQuery I'm not sure why they would switch. I would be apprehensive in investing in Snowflake long term purely because their product is highly susceptible to being obsoleted in the next 5-10 years.

cgenschwap 6y ago
I was at a company that switched from Redshift to Snowflake. It was a night and day difference. Faster (orders of magnitude!), cheaper, and significantly easier to work with (since everyone had their own personal view of the data to mutate/work with). As far as I can tell, it is a unique product in the database space. Extremely well executed ideas and design.
- deleted 6y ago
  [deleted]
- jeffffff 6y ago
  yeah redshift is not at all comparable to snowflake. big query is much closer, it's ahead in some areas and in the last year has closed some of the gaps where it wasn't. big query's biggest problem is that it's tied to gcp which is a distant 3rd in cloud marketshare. they have big query omni coming which is multi-cloud but it'll probably be a while before it's comparable to big query in gcp.
  - philjohn 6y ago
    The other problem with BigQuery is that you can very easily write a query that's going to cost you a lot of money to run - with Snowflake you can let it run for an hour or so, and then realise it was a bad idea and you're only out a few credits, a handful of dollars. The killer feature for me was the query profiler - you can see WHY a query is taking a long time and optimise it - BigQuery just felt like Google were brute forcing the performance, and then charging you accordingly. When the project I was on switched, the micro-clusters (and the ability to recluster a table) as well as the MERGE semantics beat BigQuery hands down - although those features my be out of beta now (but I've moved on to a new gig).
    - m0zg 6y ago
      That's also a problem that it'd be fairly straightforward for Google to solve by automatically spinning up smaller, entirely separate serving clusters for customers who are worried about such a blowout (for a fee, obvs). It's just the serving tree (+ whatever in-memory storage service they use to do distributed joins nowadays), no need to duplicate the rest of the service. The caveat is, a smaller cluster will favor query optimizations specific to that smaller cluster. Some of those "small cluster" optimizations could hurt query performance when deployed against BQ proper with its tens of thousands of workers. Also, BQ does explain the query plan to some extent: https://cloud.google.com/bigquery/query-plan-explanation https://cloud.google.com/bigquery/query-plan-explanation. Not quite at the level of a "regular" SQL DB, but it does give you some info to work with when optimizing queries. If you haven't used it in a while I'd give it another try.
      - statusgraph 6y ago
        BQ Slots lets you do essentially that (pre-commit to a particular cluster size)
      - quirmian 6y ago
        I believe this is exactly what slot reservations in BigQuery achieve. Instead of paying on-demand pricing that is determined by data read, you purchase a fixed number of “slots” that are shared by queries running within that particular project.
        m0zg 6y ago
        Ah OK, after reading their docs I see they've changed what "slots" used to mean in Dremel (internal version of BQ). It used to be that slots _guaranteed_ capacity, but did not limit it. Meaning that you could rely on having a certain number of workers in the cluster when you issue a query, but if Dremel had more it'd give you all it's got. Obviously this is not viable when people have to pay per terabyte read, because a ton can be read. What they have now strikes me as an even better solution to the problem of bankrupting someone with a query IMO. Not sure how pricing compares to redshift et al, but pricing is the easiest thing for Google to change.
- MrPowers 6y ago
  Snowflake seems like a unique product and I can only imagine the complex math they're doing under the hood to achieve these incredible query times. memsql is the only real competitor I know of. Redshift is a lot less user friendly (constant need to run vacuum queries). Parquet lakes / Delta lakes don't have anything close to the performance. Predicate pushdown filtering enabled by the Snowflake Spark connector seems really promising. Lots of companies are currently running big data analyses on Parquet files in S3. Snowflake has the opportunity to grab a huge slice of the big data market.
  - javajosh 6y ago
    >I can only imagine the complex math they're doing under the hood to achieve these incredible query times Maybe its cynical/paranoid, but in this age of Theranos I must ask: is it possible their algorithm excels at showing you a reasonable looking number, rather than an accurate one?
    - dumbfounder 6y ago
      It's SQL, if they were giving wrong answers people would notice.
    - kgbdrop1 6y ago
      It's not too terribly difficult to load test Snowflake to get a sense of scaling. Jmeter does the job well. Heck I can pass you along some sample projects I've done against them if you really wanted.
    - deleted 6y ago
      [deleted]
  - ruang 6y ago
    What kind of math is involved in building a faster database? Genuinely curious. I would guess maybe linear algebra, indirectly.
    - acidbaseextract 6y ago
ghgdynb1 6y ago
I think the obsolescence issue is complicated. I recently saw a criticism of Palantir which went: "The company has largely succeeded, they say, not because of its technological wizardry but because its interface is slicker and more user friendly than the alternatives created by defense contractors." A lot of the most successful tech firms started post-dot-com are decent interfaces to not-particularly-revolutionary databases. In high-end consulting and investment banking, appearances are hugely important. You can't have trash decks. It's unsurprising to me that the same is true in defense and intelligence. You can get a roof over your head and breakfast at a trashy motel or the Ritz. Everybody knows the Ritz can command a much higher price because "its interface is slicker and more user friendly than the alternatives." I think the same thing is true here.
- novok 6y ago
  The ritz has far better beds, cleaner & safer rooms, better food and is far more likely to deliver that consistently. It's not just the appearance.
  - ethbr0 6y ago
    Maybe your Ritz experiences have been different than mine, but IMHO all hotel rooms are concrete boxes with a facsimile of home stuffed inside them, copied and pasted as many times as local demand will merit. Hotel restaurants are the same principle, except replace furnishing with food.
    - Spooky23 6y ago
      Stay at an aging Courtyard Marriott. Some boxes are nicer than others.
      - ethbr0 6y ago
        I've stayed at everything from a Motel 6, to Courtyards / Residence Inns / Sheratons between NYC and San Diego, to Four Seasons / Ritz Carltons. I stand by my claim. The relative differentiation in niceness is swamped by their mass produced boxness. Ironically, my favorite road chain tends to be Aloft. At least they're upfront about their capsule-esque nature, in a sort of ironic/not-ironic way? Least favorite: Embassy Suites. shudders It's like every Disney vacationing family's fantasy about what a hotel should be... packed with every Disney vacationing family. Omelette?
bpodgursky 6y ago
At the end of the day, all the data warehouses run on SQL, with a bit of customization around ingestion and export. Most of them are backed by object storage (S3/GCS) and those integrations look very similar. I wouldn't be that worried about lock-in or being made obsolete. Business logic is going to be pretty easy to port between Redshift, BigQuery, Snowflake, or whatever comes next.
- worker767424 6y ago
  > Most of them are backed by object storage (S3/GCS) Redshift is backed by worker instances that have their own stores in what's basically an EC2 instance. It's definitely not backed by S3 like Athena. Bigquery and GCS are both built on top of Colossus, but they have different layers in between them.
  - AtlasLion 6y ago
    Same applies to Teradata vantage on cloud.
  - bpodgursky 6y ago
    Sorry, probably should have been more precise. Meant to say: most users are going to interact with the warehouses via object storage for import and export of data. Since the object store APIs are almost identical across platforms, it doesn't matter that much which warehouse you actually use for production work. It's something that does massive SQL, imports data from S3, and exports data to S3.
    - tomnipotent 6y ago
      > most users are going to interact with the warehouses via object storage for import and export of data. No, most are going to be using SQL IDE's to query and export data.
  - EwanToo 6y ago
    With the newer Redshift ra3 instances you use S3 backed storage with local SSD caching https://aws.amazon.com/redshift/features/ra3/ https://aws.amazon.com/redshift/features/ra3/
zeroxfe 6y ago
> I would be apprehensive in investing in Snowflake long term purely because their product is highly susceptible to being obsoleted in the next 5-10 years. This can be said about most products and companies. What keeps them alive is how robustly they capture (and hold on to) the market, reduce costs through economies of scale, and innovate. This specific market is also very rapidly growing.
paxys 6y ago
Nothing ever gets obsolete once it gains a large foothold in the enterprise space. There's a reason why Oracle and IBM are worth what they are today.
- mr_toad 6y ago
  > Nothing ever gets obsolete once it gains a large foothold in the enterprise space. Lotus? Delphi?
  - ncmncm 6y ago
    Both still in very heavy use. In 2014, anyway, every single IBM employee had to keep a Lotus Notes window open. It was hellish. Dunno if that's changed since Red Hat took them over.
  - Spooky23 6y ago
    Lotus is all over in government and insurance. As a mail client it is mostly dead, but the apps live on.
  - hkmurakami 6y ago
    Used Lotus notes as recently as 2010, I am pretty sure it's going strong in my megacorp former employer.
- scarface74 6y ago
  Novell, Word Perfect
  - bawolff 6y ago
    Wordperfect was used in certain industries (legal especially i think) long after it started dying everywhere else. I don't think its an exception to this rule.
    - scarface74 6y ago
      Yes but it’s dead now.
boh 6y ago
I would think it wouldn't be the same product in 5-10 years.
sjg007 6y ago
Lots of companies have built on top of snowflake.

cgenschwap 6y ago

I was at a company that switched from Redshift to Snowflake. It was a night and day difference. Faster (orders of magnitude!), cheaper, and significantly easier to work with (since everyone had their own personal view of the data to mutate/work with). As far as I can tell, it is a unique product in the database space. Extremely well executed ideas and design.

deleted 6y ago
[deleted]
jeffffff 6y ago
yeah redshift is not at all comparable to snowflake. big query is much closer, it's ahead in some areas and in the last year has closed some of the gaps where it wasn't. big query's biggest problem is that it's tied to gcp which is a distant 3rd in cloud marketshare. they have big query omni coming which is multi-cloud but it'll probably be a while before it's comparable to big query in gcp.
- philjohn 6y ago
  The other problem with BigQuery is that you can very easily write a query that's going to cost you a lot of money to run - with Snowflake you can let it run for an hour or so, and then realise it was a bad idea and you're only out a few credits, a handful of dollars. The killer feature for me was the query profiler - you can see WHY a query is taking a long time and optimise it - BigQuery just felt like Google were brute forcing the performance, and then charging you accordingly. When the project I was on switched, the micro-clusters (and the ability to recluster a table) as well as the MERGE semantics beat BigQuery hands down - although those features my be out of beta now (but I've moved on to a new gig).
  - m0zg 6y ago
    That's also a problem that it'd be fairly straightforward for Google to solve by automatically spinning up smaller, entirely separate serving clusters for customers who are worried about such a blowout (for a fee, obvs). It's just the serving tree (+ whatever in-memory storage service they use to do distributed joins nowadays), no need to duplicate the rest of the service. The caveat is, a smaller cluster will favor query optimizations specific to that smaller cluster. Some of those "small cluster" optimizations could hurt query performance when deployed against BQ proper with its tens of thousands of workers. Also, BQ does explain the query plan to some extent: https://cloud.google.com/bigquery/query-plan-explanation https://cloud.google.com/bigquery/query-plan-explanation. Not quite at the level of a "regular" SQL DB, but it does give you some info to work with when optimizing queries. If you haven't used it in a while I'd give it another try.
    - statusgraph 6y ago
      BQ Slots lets you do essentially that (pre-commit to a particular cluster size)
    - quirmian 6y ago
      I believe this is exactly what slot reservations in BigQuery achieve. Instead of paying on-demand pricing that is determined by data read, you purchase a fixed number of “slots” that are shared by queries running within that particular project.
      - m0zg 6y ago
        Ah OK, after reading their docs I see they've changed what "slots" used to mean in Dremel (internal version of BQ). It used to be that slots _guaranteed_ capacity, but did not limit it. Meaning that you could rely on having a certain number of workers in the cluster when you issue a query, but if Dremel had more it'd give you all it's got. Obviously this is not viable when people have to pay per terabyte read, because a ton can be read. What they have now strikes me as an even better solution to the problem of bankrupting someone with a query IMO. Not sure how pricing compares to redshift et al, but pricing is the easiest thing for Google to change.
MrPowers 6y ago
Snowflake seems like a unique product and I can only imagine the complex math they're doing under the hood to achieve these incredible query times. memsql is the only real competitor I know of. Redshift is a lot less user friendly (constant need to run vacuum queries). Parquet lakes / Delta lakes don't have anything close to the performance. Predicate pushdown filtering enabled by the Snowflake Spark connector seems really promising. Lots of companies are currently running big data analyses on Parquet files in S3. Snowflake has the opportunity to grab a huge slice of the big data market.
- javajosh 6y ago
  >I can only imagine the complex math they're doing under the hood to achieve these incredible query times Maybe its cynical/paranoid, but in this age of Theranos I must ask: is it possible their algorithm excels at showing you a reasonable looking number, rather than an accurate one?
  - dumbfounder 6y ago
    It's SQL, if they were giving wrong answers people would notice.
  - kgbdrop1 6y ago
    It's not too terribly difficult to load test Snowflake to get a sense of scaling. Jmeter does the job well. Heck I can pass you along some sample projects I've done against them if you really wanted.
  - deleted 6y ago
    [deleted]
- ruang 6y ago
  What kind of math is involved in building a faster database? Genuinely curious. I would guess maybe linear algebra, indirectly.
  - acidbaseextract 6y ago

ghgdynb1 6y ago

I think the obsolescence issue is complicated. I recently saw a criticism of Palantir which went: "The company has largely succeeded, they say, not because of its technological wizardry but because its interface is slicker and more user friendly than the alternatives created by defense contractors." A lot of the most successful tech firms started post-dot-com are decent interfaces to not-particularly-revolutionary databases. In high-end consulting and investment banking, appearances are hugely important. You can't have trash decks. It's unsurprising to me that the same is true in defense and intelligence. You can get a roof over your head and breakfast at a trashy motel or the Ritz. Everybody knows the Ritz can command a much higher price because "its interface is slicker and more user friendly than the alternatives." I think the same thing is true here.

novok 6y ago
The ritz has far better beds, cleaner & safer rooms, better food and is far more likely to deliver that consistently. It's not just the appearance.
- ethbr0 6y ago
  Maybe your Ritz experiences have been different than mine, but IMHO all hotel rooms are concrete boxes with a facsimile of home stuffed inside them, copied and pasted as many times as local demand will merit. Hotel restaurants are the same principle, except replace furnishing with food.
  - Spooky23 6y ago
    Stay at an aging Courtyard Marriott. Some boxes are nicer than others.
    - ethbr0 6y ago
      I've stayed at everything from a Motel 6, to Courtyards / Residence Inns / Sheratons between NYC and San Diego, to Four Seasons / Ritz Carltons. I stand by my claim. The relative differentiation in niceness is swamped by their mass produced boxness. Ironically, my favorite road chain tends to be Aloft. At least they're upfront about their capsule-esque nature, in a sort of ironic/not-ironic way? Least favorite: Embassy Suites. shudders It's like every Disney vacationing family's fantasy about what a hotel should be... packed with every Disney vacationing family. Omelette?

bpodgursky 6y ago

At the end of the day, all the data warehouses run on SQL, with a bit of customization around ingestion and export. Most of them are backed by object storage (S3/GCS) and those integrations look very similar. I wouldn't be that worried about lock-in or being made obsolete. Business logic is going to be pretty easy to port between Redshift, BigQuery, Snowflake, or whatever comes next.

worker767424 6y ago
> Most of them are backed by object storage (S3/GCS) Redshift is backed by worker instances that have their own stores in what's basically an EC2 instance. It's definitely not backed by S3 like Athena. Bigquery and GCS are both built on top of Colossus, but they have different layers in between them.
- AtlasLion 6y ago
  Same applies to Teradata vantage on cloud.
- bpodgursky 6y ago
  Sorry, probably should have been more precise. Meant to say: most users are going to interact with the warehouses via object storage for import and export of data. Since the object store APIs are almost identical across platforms, it doesn't matter that much which warehouse you actually use for production work. It's something that does massive SQL, imports data from S3, and exports data to S3.
  - tomnipotent 6y ago
    > most users are going to interact with the warehouses via object storage for import and export of data. No, most are going to be using SQL IDE's to query and export data.
- EwanToo 6y ago
  With the newer Redshift ra3 instances you use S3 backed storage with local SSD caching https://aws.amazon.com/redshift/features/ra3/ https://aws.amazon.com/redshift/features/ra3/

zeroxfe 6y ago

> I would be apprehensive in investing in Snowflake long term purely because their product is highly susceptible to being obsoleted in the next 5-10 years. This can be said about most products and companies. What keeps them alive is how robustly they capture (and hold on to) the market, reduce costs through economies of scale, and innovate. This specific market is also very rapidly growing.

paxys 6y ago

Nothing ever gets obsolete once it gains a large foothold in the enterprise space. There's a reason why Oracle and IBM are worth what they are today.

mr_toad 6y ago
> Nothing ever gets obsolete once it gains a large foothold in the enterprise space. Lotus? Delphi?
- ncmncm 6y ago
  Both still in very heavy use. In 2014, anyway, every single IBM employee had to keep a Lotus Notes window open. It was hellish. Dunno if that's changed since Red Hat took them over.
- Spooky23 6y ago
  Lotus is all over in government and insurance. As a mail client it is mostly dead, but the apps live on.
- hkmurakami 6y ago
  Used Lotus notes as recently as 2010, I am pretty sure it's going strong in my megacorp former employer.
scarface74 6y ago
Novell, Word Perfect
- bawolff 6y ago
  Wordperfect was used in certain industries (legal especially i think) long after it started dying everywhere else. I don't think its an exception to this rule.
  - scarface74 6y ago
    Yes but it’s dead now.

boh 6y ago

I would think it wouldn't be the same product in 5-10 years.

sjg007 6y ago

Lots of companies have built on top of snowflake.