4 ms·

I’ve had a similar experience where I could email PMs and we even had Google forward deployed engineers on-site. The trouble is that their customer service is

I’ve had a similar experience where I could email PMs and we even had Google forward deployed engineers on-site. The trouble is that their customer service is terrible. We reported numerous issues and there was no tracking at all. For one UI bug in their webapp, a PM completely disregarded the reproduction instructions I gave him and made me have a half-hour kong Hangout with a remote engineer to prove the bug existed. The engineer even found more bugs while watching my screen. The sales engineers (solution consultants?) are rather slimey, especially compared to the PMs who often admit when they don’t know something. This one eng kept pushing Dataflow even after we said no 100 times. It appears Google uses Dataflow / Beam internally now (instead of MapReduce) and so I guess at Google you’re dumb if you’re not doing the “correct” thing. Also frustrating was how much money we wasted due to things like multi-gpu jobs taking 10-15 minutes to start and their GPU hypervisor pausing the whole system for 10-20 seconds incessantly. If you use GCE, make sure Google is paying for the bloody edges. Get a ton of credits.

slovenlyrobot 7y ago
I had a 4 roundtrip email thread with their support over a speech API bug. Eventually sent them a narrated step-by-step screencast, and the guy is finally like "oh, it's not supposed to work that way." (Yes, that's what I said in e-mail 1!)
- stingraycharles 7y ago
  To be perfectly fair, I have had these types of experiences (plus more) with AWS as well. Their solution engineers are pretty useless, bugs in products never get solved and you get endless forwards and escalations on their issue tracker. I think in general, scaling support for such complex a product that’s constantly changing is really difficult.
- dilyevsky 7y ago
  4 rt? You got off scot-free! I had several multi-week threads with daily posts for something that should’ve been a p1 bug fix on their end.
faizshah 7y ago
Cloud Dataflow is also vastly more expensive than using pyspark and Apache Beam is 10 times slower for most operations than Flink or Spark. I have to try the new flexible scheduling but DataFlow the last time I tried it 2 years ago was a massive rip off. Also DataPrep was amazing until you run the DataFlow pipeline it produces and a < 1TB dataset cleaning costs $25... Have had good experiences once the data is in bigquery and bq keeps costs low. But lately Ive been trying to figure out a semi-managed way of doing exactly-once stream processing for cheaper than dataflow. Possibly an architecture around cloud run might work.
- choppaface 7y ago
  Oh wow thanks for the info! Had no idea Beam was that slow.
  - faizshah 7y ago
    Check this reference out: https://arxiv.org/pdf/1907.08302.pdf https://arxiv.org/pdf/1907.08302.pdf “ Our benchmark results show that Apache Beam has a noticeable impact on the performance of DSPSs in almost all cases. Programs developed using Apache Beam suffered from a slowdown of up to a factor of 58 in the worst case. At the same time, there is one scenario where the query developed using Apache Beam is about as fast as its counterparts using the APIs of the corresponding DSPS. However, for most scenarios we observed a slowdown of at least a factor three.”
- ramraj07 7y ago
  $25 for cleaning a relatively small dataset does sound a bit high but not outrageous.. given the process abstracts out all the infra tweaking and setting up various systems. $25 an hour is probably a good estimate of how much a data warehouse like snowflake costs for a reasonable cluster as well, so unless you spin up your own spot ec2 instances and set up a spark cluster and perform the operations the cost savings are going to be marginal at best? Do you need to run this pipeline more than once a day? Is it sufficiently important for your business case? I feel like 25 is a very small sum if any of them are remotely true. On a related note, how practically reliable is the DataPrep->DataFlow workflow? Could a reasonably smart analyst with zero programming experience set them up and run them? Feel like that's who this workflow is built for, but I've not had the best of experiences with Trifacta demos in AWS with reliability.

slovenlyrobot 7y ago

I had a 4 roundtrip email thread with their support over a speech API bug. Eventually sent them a narrated step-by-step screencast, and the guy is finally like "oh, it's not supposed to work that way." (Yes, that's what I said in e-mail 1!)

stingraycharles 7y ago
To be perfectly fair, I have had these types of experiences (plus more) with AWS as well. Their solution engineers are pretty useless, bugs in products never get solved and you get endless forwards and escalations on their issue tracker. I think in general, scaling support for such complex a product that’s constantly changing is really difficult.
dilyevsky 7y ago
4 rt? You got off scot-free! I had several multi-week threads with daily posts for something that should’ve been a p1 bug fix on their end.

faizshah 7y ago

Cloud Dataflow is also vastly more expensive than using pyspark and Apache Beam is 10 times slower for most operations than Flink or Spark. I have to try the new flexible scheduling but DataFlow the last time I tried it 2 years ago was a massive rip off. Also DataPrep was amazing until you run the DataFlow pipeline it produces and a < 1TB dataset cleaning costs $25... Have had good experiences once the data is in bigquery and bq keeps costs low. But lately Ive been trying to figure out a semi-managed way of doing exactly-once stream processing for cheaper than dataflow. Possibly an architecture around cloud run might work.

choppaface 7y ago
Oh wow thanks for the info! Had no idea Beam was that slow.
- faizshah 7y ago
  Check this reference out: https://arxiv.org/pdf/1907.08302.pdf https://arxiv.org/pdf/1907.08302.pdf “ Our benchmark results show that Apache Beam has a noticeable impact on the performance of DSPSs in almost all cases. Programs developed using Apache Beam suffered from a slowdown of up to a factor of 58 in the worst case. At the same time, there is one scenario where the query developed using Apache Beam is about as fast as its counterparts using the APIs of the corresponding DSPS. However, for most scenarios we observed a slowdown of at least a factor three.”
ramraj07 7y ago
$25 for cleaning a relatively small dataset does sound a bit high but not outrageous.. given the process abstracts out all the infra tweaking and setting up various systems. $25 an hour is probably a good estimate of how much a data warehouse like snowflake costs for a reasonable cluster as well, so unless you spin up your own spot ec2 instances and set up a spark cluster and perform the operations the cost savings are going to be marginal at best? Do you need to run this pipeline more than once a day? Is it sufficiently important for your business case? I feel like 25 is a very small sum if any of them are remotely true. On a related note, how practically reliable is the DataPrep->DataFlow workflow? Could a reasonably smart analyst with zero programming experience set them up and run them? Feel like that's who this workflow is built for, but I've not had the best of experiences with Trifacta demos in AWS with reliability.