7 ms·
Great article. On the surface, it's about Snowflake. At a deeper level, the article is about the perverse incentives motivating SaaS businesses to do seemingly
by cs702 4y ago
Great article. On the surface, it's about Snowflake. At a deeper level, the article is about the perverse incentives motivating SaaS businesses to do seemingly dumb, inefficient things and avoid seemingly obvious optimizations by default.
Many SaaS businesses are perfectly happy to let customers shoot themselves in the foot if it generates more revenue. The BigQuery example (presently, by default, `select * from table limit 10` obediently scans the entire table at your expense!) is spot-on.
As the article so well puts it, every SaaS company has a vested financial interest "to leave optimization gremlins in."
- soheil 4y agoIf that lowers the barrier to entry without having expert level knowledge to know what a full table scan even means why not? Instead of hiring a dba maybe you could hire an intern instead and happily eat the cost of Snowflake.
- kalimoxto 4y agoI think the point of the article is that an optimizer doesn't affect the barrier to entry at all, but adding it would save end users quite a bit of money. So they don't do it because end users' money is revenue for Snowflake/Alphabet
- soheil 4y agoIf you could just add an optimizer why doesn't the db engine just do that?
- whimsicalism 4y agoTake a step back and reread the article and the comments you are replying to.
- florbo 4y agoIt doesn't lower barriers to entry, it's contrary to logical expectations for someone unfamiliar with how BQ works. If the query is limited to 10 results you wouldn't expect it to scan all 2 trillion of your records. Granted there are numerous warnings in the GUI for these types of things but make this mistake in Python and you're none the wiser.
- soheil 4y agoWait are you saying the BQ db engine is not following logical expectations? You do realize a "limit" clause doesn't prevent a full table scan in all cases, right?
- horsawlarway 4y agoand that db expert you just recommended against hiring could surely tell you that... The intern won't.
- Aulig 4y agoIt feels like these companies haven't found the right value metric to price along. Ideally it should align with the value the customer receives.
- altdataseller 4y agoBut that's almost impossible to measure by Snowflake. How would they know how much more revenue you earned because you use Snowflake?
- Rastonbury 4y agoI don't think their customers could quantify it if they tried (and i'm not implying Snowflake doesn't give value, it probably does but how does a company attribute it)
- polskibus 4y agoOnly competition can enforce this. The article ideally demonstrates the problems with monopolies and vendor lock-in.
- danielmarkbruce 4y agoSnowflake is nowhere near a monopoly, and plenty of customers have moved from other vendors (Teradata, Netezza, etc) to Snowflake - showing that vendor lock-in is not as strong as it might seem.
- polskibus 4y agoIf that's the case, then why aren't Snowlake and Google targeting query optimiziation with higher priority to lower end-user costs? There's no incentive in the market for them to do so - once you switch to them, you'll eat up the cost of quirks and learn how to avoid them the hard way.
- danielmarkbruce 4y ago
- deepGem 4y agoWow their statement about not participating in benchmarking wars is alarming. In this day and age, when benchmarking tools are so inexpensive and almost everything is very transparent, why not participate. Or even better engage with a neutral third party such as Jepsen to get on an even playing field and duke it out.
- datavirtue 4y agoBecause their business is providing a solution that IT failed to. Despite the large cost, which the business was already accustomed to from previous IT attempts, pales in comparison to the additional costs of doing it themselves. It's like the cloud in general, the cost is high but so is the hype. When all that dust settles over the coming years the business will start shopping on price. They will then realize they have been locked in to some extent and will need to start wriggling loose of the lock-in.
- deleted 4y ago[deleted]
- danielmarkbruce 4y agoBecause their value prop isn't being #1 on benchmarks. It's about * being easy to manage * being able to scale up and down compute so you can get good performance without having to keep a bunch of machines running.
- lokar 4y agoBenchmark results rarely predict actual application perf. You need to run your own queries against your own data. Do a real POC.
- hodgesrm 4y ago> Wow their statement about not participating in benchmarking wars is alarming. I found the Snowflake statement pretty reasonable. [0] Vendor benchmarks are largely propaganda. What actually counts is performance on real-world workloads, starting with your own. Plus good bencharks are costly to do well. If vendors are going to invest in load testing, it's way better to do it as part of the QA process, which directly benefits users. The other thing for vendors to do is to drop DeWitt clauses so others can run benchmarks and share the results. Snowflake announced this in the statement and also changed their acceptable use policy accordingly. [1] [0] https://www.snowflake.com/blog/industry-benchmarks-and-competing-with-integrity/ https://www.snowflake.com/blog/industry-benchmarks-and-compe... [1] https://www.snowflake.com/legal/acceptable-use-policy/ https://www.snowflake.com/legal/acceptable-use-policy/ Disclaimer: My company runs a cloud service for ClickHouse that competes against Snowflake.
- spmurrayzzz 4y agoWell said. I'd also add a cynical note that the recurring revenue model is incentivized to keep the gremlins around not just because of the impact to metered costs, but also because off-ramping is that much more difficult once engineers implement workaround/solutions to mitigate the impact of those smells. Just another way that vendor lock-in occurs (intentionally or otherwise).
- kolinko 4y agoin case of BigQuery it makes sense though - they use map reduce on distributed clusters, so there is no easy way to stop after 10 results are found
- JimmyAustin 4y agoIt's pretty easy to limit the number of results returned by each partition to by limited to 10, then have that further reduced to 10 total during the reduce step.
- dcow 4y agoI was thinking about this too. Why don’t SaaS companies just force price increases to offset their broken pricing model? Nobody would care, you’re paying the same you were paying yesterday. If you’re still the best in class product with sticky features people will stay. If not and you’re competing, then you have the opportunity to reduce the price in the future or simply not increase it and let users see lower bills which might also retain them.
- twistedpair 4y agoFWIW, BigQuery tables can be configured to require a partition filter clause [0] in the SQL query, so that you cannot shoot yourself in the foot like that. Now if they'd just make an Organization Policy to let you turn it on by default for all new tables. [0] https://cloud.google.com/bigquery/docs/querying-partitioned-tables#require_a_partition_filter_in_queries https://cloud.google.com/bigquery/docs/querying-partitioned-...
- cs702 4y agoYes. That's exactly the OP's point: It's up to you to remember to do the extra work necessary to avoid shooting yourself in the foot by default.
- twistedpair 4y agoDepends who the "you" is; someone just getting started with Cloud, or a savvy enterprise operator? GCP has sided with an "easy out of the box experience". For example, a new project has a "default" network with some permissive firewall rules. A savvy operator wouldn't build things this way, but for a first time user, the cloud is daunting and a JustWorks™ experience gets them moving quickly (e.g. so they can SSH into their VMs easily). Now, once you've gotten your feet under you, and want to build a solid cloud setup, you'll add Organization Policies [1] like "Skip default network creation", and all new Projects will be completely closed off from the web by default, at the cost of all networking being more complex. Once you're ready for this, turn it on. So, how should a SaaS database work? Should you have to learn all the intricacies of sharding, partitioning, indexing, SELECT, FROM, HAVING, FULL/INNER/OUTER JOIN, WHERE, GROUP BY, LIMIT, before you write your first query? This is a long standing yin/yang question of product UX. What user persona and UX do you design for on the experience and complexity spectrum. [1] https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints https://cloud.google.com/resource-manager/docs/organization-...
- itsdrewmiller 4y agoSo they have a system for enforcing rules but still haven't built the rule that would reduce their revenue - seems like an example in favor of the article.
- tluyben2 4y agoFunny that most people here advocate aws while they have tons and tons of foot shooting tools that cost people 1000s of usd all the time. And we just accept it. Like if you want to kill a complex cluster with one api call or button click, it won’t let you for xyz; that’s not because they cannot, it’s because you will just let it be and that makes money.
- makk 4y ago> As the article so well puts it, every SaaS company has a vested financial interest "to leave optimization gremlins in." It depends on the time scale. A SaaS optimizing for, say, a 1-3 year financial return will see their interests through a different lens than one optimizing for a multi-decade return. Leaving optimization gremlins in isn't aligned with customers' interests in the long run, so the customers will eventually find alternatives if the SaaS doesn't eventually align itself with customers.
- smugma 4y ago"As an investor, I expect Snowflake to show amazing profitability and record-breaking revenue numbers. As an Engineer, if Snowflake continues on the current path of ignoring performance, I expect them to lose share to the open-source community or some other competitor, eventually walking down the path of Oracle and Teradata. Here are a few things I think they can do to stay relevant in five years."
- danielmarkbruce 4y agoThe point is incentives.
- danielmarkbruce 4y agoIt's a terrible article. The author misunderstands competition and how much it drives products in this area. Snowflake is incentivized to make their product better on every dimension. If Snowflake don't improve, customers will leave in droves - like when they moved to Snowflake. In practice, as has been pointed out in other comments, they do improve their performance (for competitive reasons) and it does cost them money when they do it.... They did it a couple qtrs ago and left $97 mill on the table. https://www.fool.com/earnings/call-transcripts/2022/03/02/snowflake-inc-snow-q4-2022-earnings-call-transcrip/ https://www.fool.com/earnings/call-transcripts/2022/03/02/sn...
- uoaei 4y agoI don't think it misunderstands business competition. In fact it understands the concept of competition very well, and develops an insightful critique into the perverse incentives that are borne from competition. It benefits no one except for a couple thousand people to so blatantly play their customers in this way. In fact, it's worse, as it incentivizes that same behavior of other market actors in the space.
- danielmarkbruce 4y agoWhat exactly in the article suggest the author understands the pressure of competition on incentives? The author states that Snowflake are not incentivized to increase performance due to short term revenue concerns but doesn't mention they are also incentivized to do the opposite from a competitive perspective. The result is incomplete enough that it ends up being flat wrong with respect to the behavior that the company actually engages in. The author missed the fact Snowflake did the very thing he/she suggested they were incentivized not to do, recently, at a cost of $97 million. The CEO explained why they are doing it and how they are actually incentivized. I don't know how the article could miss the mark by more than it has. The company literally does the opposite of what he/she suggested.It's not like they are the only one either, AWS has a history of reducing prices. Why? Once again, competition.
- morelisp 4y ago
- scarface74 4y agoStandard disclaimer: I work at AWS in consulting and could easily be accused of drinking the Kool Aid. Everyone from consultants, SAs, Sales, support etc is constantly working toward getting customers to “optimize” their spend. Of course any business wants you to give them more money. But, none of us are pushed to get them to spend money on services or methods to do things inefficiently. I specifically work in consulting specializing in “application modernization”. That means most of my implementations are cheap and I’m constantly spending time making sure my implementation is cheap as possible and still meet the requirements. I first noticed this attitude from AWS when I was working for a startup. This isn’t just with AWS. I spent years working in enterprise shops and saw the same attitude working with Microsoft. I can’t speak for any other large organizations - AWS and Microsoft are the only two I’ve worked with as either a customer or employee where there was huge spending on infrastructure or software. Now I could easily get started about my opinion of Oracle from the customer standpoint. But I won’t.
- thehappypm 4y agoI worked at a BiqQuery shop and they have a terrific feature where right next to the “Run query” button there is an estimate of the cost of the query, in bytes. It becomes extremely obvious when a query is a full table scan.
- philjohn 4y agoHa! I wonder if we worked at the same place ... it's in the travel space, because when I worked there someone wrote a plugin that did this, and it was a real eye opener at times!
- thehappypm 4y agoNope, not travel! For us it was a built in feature, not a plugin. This was earlier this year also.
- ed25519FUUU 4y ago> The BigQuery example (presently, by default, `select * from table limit 10` obediently scans the entire table at your expense!) is spot-on. This bit me on big queries Public patent search, which I was just noodling with for fun. Each query was $4. Ow!
- cyanydeez 4y agoOne more deeper level: almost all consulting exists in a world of consultants driven to limit efficiency lest their billables decline. I know a few people who seemed to aggregate their entire personality to "hard worker" when they refused to progress.
- aiisjustanif 4y agoThat’s one massively vague take on the whole industry of consulting, including on-prem software, open-source solutions. My billable hours do fine while making operations more efficient and cost less.
- scarface74 4y agoIt depends. Many large companies have internal “Professional Services” departments with “consultants” who are full time employees. Standard disclaimer: I work in ProServe at AWS. When you “consult” and are employed by the company selling the software, billable hours and utilization is not the be all end all. Consulting is just the “nose of the camel in the tent”. They want you to be as efficient as possible so they can make ongoing revenue. Trust me, AWS is not going to complain if it only took me 20 hours to do work that was estimated for 40 and brings in half as much consulting revenue if it means ongoing revenue from the customer. There isn’t just a singular focus on utilization rates.