5 ms·

Almost all the big companies I worked for had a "database gang" -- a database group which, in the name of centralization, forced you to bow to them to get anyth

Almost all the big companies I worked for had a "database gang" -- a database group which, in the name of centralization, forced you to bow to them to get anything. New DB? bow to them. More nodes? bow to them. Reboot? bow to them. The internal budget "prices" would be off the charts unbelievable. It makes sense to centralize, but only at a certain cost. Beyond that cost, it is better to just de-centralize because not every project can spend 4-5 months of meetings to spin up a DB. The cloud changed this because it became an OpEx discussion and something you could spin up on your own. For non-production workloads, it comes especially obvious to do this.

stingraycharles 6y ago
This still in no way answers why Snowflake is so valuable, though. I completely understand our argument, and I agree with it; I just don’t think the article’s arguments are anything else than ex post facto rationalization. When they mentioned NPS I almost snorted my coffee, that metric can be gamified in any way you want it.
- stcredzero 6y ago
  >> Almost all the big companies I worked for had a "database gang" -- a database group which, in the name of centralization, forced you to bow to them to get anything. New DB? bow to them. More nodes? bow to them. Reboot? bow to them. The internal budget "prices" would be off the charts unbelievable. > This still in no way answers why Snowflake is so valuable, though. It explains it to a T! You have something you want, but internal company politics and territoriality keep you from getting it the way you want. An outside provider lets everyone get it for a bit of cash. It's basically the same play as Salesforce. It's not some kind of technical moonshot. It has to do with a modicum of technology delivered by a 3rd party who can avoid all of the internal friction. The next founder who can think of this kind of play, then execute on it, will be the next Salesforce/Snowflake, and will probably have the ear of the same investors!
  - stingraycharles 6y ago
    What I meant was why Snowflake specifically is so valuable. BigQuery, Redshift or any other cloud db would fill this gap as well. Why Snowflake?
    - mmm_grayons 6y ago
      It's a chance for investors to get in on the next big thing and invest specifically in data warehousing. No one can put his money directly into BigQuery or Redshift. Edit: why are so many people downvoting this? Is there some other reason for Snowflake's valuation (aside from tech bubble playing a role)?
    - TuringNYC 6y ago
      I find Snowflake much easier to use than BigQuery, Redshift. It is also cloud-service-provider agnostic. So your only hook at that point is ingest + any snowflake-specific SQL (and obviously security migration etc.) So for retention, they compete on UX rather than walls. W/r/t value, the idea is a disproportionate of egress from Oracle, Teradata, etc will end up at Snowflake, hence huge TAM, SAM, and SOM.
    - chrisjc 6y ago
      Snowflake fanboy here who can't really answer your question about why it's so valuable. Not sure I can rationalize the current value. Not sure I think it should be valued this much. But I can probably answer why Snowflake instead of Redshift (sorry, not too familiar with BigQuery)... First of all it's cloud-provider agnostic so you can set up Snowflake on any or all of the 3 major cloud providers as well as set up replication between them directly or indirectly through their data exchange. Probably the most powerful feature is the way that Snowflake has the ability to scale (up or down) compute (vertically and horizontal) and storage independently of each other. Furthermore, you have the ability to scale compute down to nothing, and spin up "instantly" when the demand arrives. On top of all of this there is an incredible selection of functionality that i could go on and on about.
      - texasbigdata 6y ago
hitekker 6y ago
> The internal budget "prices" would be off the charts unbelievable. I find that this occurs when an infrastructure team considers itself a "platform". The only supplier of an asset that everyone else demands can set the "price" of the asset as high as they want.
- P00RL3N0 6y ago
  This isn't actually how monopolies work. Theoretically, monopolies cannot just charge what they want for a product, unless demand is perfectly inelastic. i.e. At some point, people (consumers) will stop paying or move to a substitute.
  - TuringNYC 6y ago
    I've seen multiple corporate re-orgs (Fortune 100) and you are right -- internal monopolies cannot just charge what they want. Eventually, internal customers get fed up and "revlot" -- then you have a corporate re-org where IT infra/services get distributed across business lines or lower. Eventually that blows up too -- re-composing data at the parent level (e.g., for quarterly financial reporting) becomes too exhausting and the company decides to revert to centralized services.
chrisjc 6y ago
> it is better to just de-centralize Until you want to join and then all of sudden it's not your problem. Then you end up with a "gang" of cowboy analysts running ad-hoc data-dumps against operational datastores affecting production uptime and stability only so that they can do a lookup between the multiple (source_table_column_count * source_table_row_count) sheets in their uber excel document. I'm all for decomposing the monolith as long as you have a plan for recomposing when it's necessary.
- TuringNYC 6y ago
  Yep, thats why I said centralizing makes sense -- up until some price point. Beyond that point, you might as well just spend the money on re-composing when you need.
  - chrisjc 6y ago
    Fair point! Sorry I missed that.
  - emn13 6y ago
    It's not just the technical costs of recomposing to achieve a jaoin; it's also diversity of this kind makes long-term maintenance a nightmare, and moving people/functionality/whatever between apps almost impossible. If everybody cowboys their own storage, you're risking building up a legacy cruft that's very hard to work your way out of later. The costs if recomposing later can be prohibitive, if a few particularly poor choices were made early on, and the hidden costs of inflexibility can bite too. There's nothing wrong with outsourcing storage; that's not the issue - the issue is the culture in which it's easier to just not talk to the rest of the company (assuming the company is small enough to have any kind of cohesion in the firs place). If it's too expensive to talk (or even pick from a few common defaults) beforehand, how are you ever going to interop later on? You're getting the downsides of a large organization without the upsides.
- deleted 6y ago
  [deleted]
closeparen 6y ago
The database gang at my company offers most things through self-service tooling a sub-24-hour Slack queue for most others. Meanwhile the "spend the company's money" gang is way off in some ivory tower, and I'm pretty sure a purchase order would take 4-5 months of meetings (heavily involving the database gang). My director has a budget for headcount and budget for travel & entertainment which is divided among the Sr. Managers and Managers, but I think we'd have to reach the CTO or CTO + 1 to get the authority to spend money on products and services. Baffled both by how bad your internal infrastructure is and by how easy it is for you to buy stuff.

stingraycharles 6y ago

This still in no way answers why Snowflake is so valuable, though. I completely understand our argument, and I agree with it; I just don’t think the article’s arguments are anything else than ex post facto rationalization. When they mentioned NPS I almost snorted my coffee, that metric can be gamified in any way you want it.

stcredzero 6y ago
>> Almost all the big companies I worked for had a "database gang" -- a database group which, in the name of centralization, forced you to bow to them to get anything. New DB? bow to them. More nodes? bow to them. Reboot? bow to them. The internal budget "prices" would be off the charts unbelievable. > This still in no way answers why Snowflake is so valuable, though. It explains it to a T! You have something you want, but internal company politics and territoriality keep you from getting it the way you want. An outside provider lets everyone get it for a bit of cash. It's basically the same play as Salesforce. It's not some kind of technical moonshot. It has to do with a modicum of technology delivered by a 3rd party who can avoid all of the internal friction. The next founder who can think of this kind of play, then execute on it, will be the next Salesforce/Snowflake, and will probably have the ear of the same investors!
- stingraycharles 6y ago
  What I meant was why Snowflake specifically is so valuable. BigQuery, Redshift or any other cloud db would fill this gap as well. Why Snowflake?
  - mmm_grayons 6y ago
    It's a chance for investors to get in on the next big thing and invest specifically in data warehousing. No one can put his money directly into BigQuery or Redshift. Edit: why are so many people downvoting this? Is there some other reason for Snowflake's valuation (aside from tech bubble playing a role)?
  - TuringNYC 6y ago
    I find Snowflake much easier to use than BigQuery, Redshift. It is also cloud-service-provider agnostic. So your only hook at that point is ingest + any snowflake-specific SQL (and obviously security migration etc.) So for retention, they compete on UX rather than walls. W/r/t value, the idea is a disproportionate of egress from Oracle, Teradata, etc will end up at Snowflake, hence huge TAM, SAM, and SOM.
  - chrisjc 6y ago
    Snowflake fanboy here who can't really answer your question about why it's so valuable. Not sure I can rationalize the current value. Not sure I think it should be valued this much. But I can probably answer why Snowflake instead of Redshift (sorry, not too familiar with BigQuery)... First of all it's cloud-provider agnostic so you can set up Snowflake on any or all of the 3 major cloud providers as well as set up replication between them directly or indirectly through their data exchange. Probably the most powerful feature is the way that Snowflake has the ability to scale (up or down) compute (vertically and horizontal) and storage independently of each other. Furthermore, you have the ability to scale compute down to nothing, and spin up "instantly" when the demand arrives. On top of all of this there is an incredible selection of functionality that i could go on and on about.
    - texasbigdata 6y ago

hitekker 6y ago

> The internal budget "prices" would be off the charts unbelievable. I find that this occurs when an infrastructure team considers itself a "platform". The only supplier of an asset that everyone else demands can set the "price" of the asset as high as they want.

P00RL3N0 6y ago
This isn't actually how monopolies work. Theoretically, monopolies cannot just charge what they want for a product, unless demand is perfectly inelastic. i.e. At some point, people (consumers) will stop paying or move to a substitute.
- TuringNYC 6y ago
  I've seen multiple corporate re-orgs (Fortune 100) and you are right -- internal monopolies cannot just charge what they want. Eventually, internal customers get fed up and "revlot" -- then you have a corporate re-org where IT infra/services get distributed across business lines or lower. Eventually that blows up too -- re-composing data at the parent level (e.g., for quarterly financial reporting) becomes too exhausting and the company decides to revert to centralized services.

chrisjc 6y ago

> it is better to just de-centralize Until you want to join and then all of sudden it's not your problem. Then you end up with a "gang" of cowboy analysts running ad-hoc data-dumps against operational datastores affecting production uptime and stability only so that they can do a lookup between the multiple (source_table_column_count * source_table_row_count) sheets in their uber excel document. I'm all for decomposing the monolith as long as you have a plan for recomposing when it's necessary.

TuringNYC 6y ago
Yep, thats why I said centralizing makes sense -- up until some price point. Beyond that point, you might as well just spend the money on re-composing when you need.
- chrisjc 6y ago
  Fair point! Sorry I missed that.
- emn13 6y ago
  It's not just the technical costs of recomposing to achieve a jaoin; it's also diversity of this kind makes long-term maintenance a nightmare, and moving people/functionality/whatever between apps almost impossible. If everybody cowboys their own storage, you're risking building up a legacy cruft that's very hard to work your way out of later. The costs if recomposing later can be prohibitive, if a few particularly poor choices were made early on, and the hidden costs of inflexibility can bite too. There's nothing wrong with outsourcing storage; that's not the issue - the issue is the culture in which it's easier to just not talk to the rest of the company (assuming the company is small enough to have any kind of cohesion in the firs place). If it's too expensive to talk (or even pick from a few common defaults) beforehand, how are you ever going to interop later on? You're getting the downsides of a large organization without the upsides.
deleted 6y ago
[deleted]

closeparen 6y ago

The database gang at my company offers most things through self-service tooling a sub-24-hour Slack queue for most others. Meanwhile the "spend the company's money" gang is way off in some ivory tower, and I'm pretty sure a purchase order would take 4-5 months of meetings (heavily involving the database gang). My director has a budget for headcount and budget for travel & entertainment which is divided among the Sr. Managers and Managers, but I think we'd have to reach the CTO or CTO + 1 to get the authority to spend money on products and services. Baffled both by how bad your internal infrastructure is and by how easy it is for you to buy stuff.