chimerasaurus

8 ms·

31.
by chimerasaurus 7y ago
Would love to know the pain points so we can make it better - Composer, Airflow, or both.
32.
by chimerasaurus 8y ago
I am currently waiting for a return refund for > 62k and have been told it will take over a month. Pretty sure most people would never consider that an acceptable return policy. Of note, I am a big fan and want to order a new one, but ..
33.
by chimerasaurus 8y ago
So, I took delivery last week the day before the price changes. I went back the next day asking how I could work with them to find a solution since I was in the return window and now the product was $2.9k less expensive. That was a bummer
34.
by chimerasaurus 8y ago
I also highly recommend using labels[1] with billing data to get a better understanding of spend. 1: https://cloud.google.com/resource-manager/docs/creating-mana...
35.
Google Data Studio and Cloud Dataprep Moved to GA(cloud.google.com)
33 points by chimerasaurus 8y ago|0 comments
36.
WaPo: Justice Department to consider allegations of censorship on social media(washingtonpost.com)
2 points by chimerasaurus 8y ago|0 comments
37.
by chimerasaurus 8y ago
My key advice for any new PM: Every Dilbert cartoon about naming a product is true.
38.
by chimerasaurus 8y ago
Disclaimer: I'm the PM for Composer. :) If the cost of Composer is an issue, ping me. Running a static environment _does_ have a cost, but for serious ETL it should be pretty inexpensive all things considered. You _should_ be able to u
39.
Google Cloud launches Cloud Composer - managed Apache Airflow(techcrunch.com)
3 points by chimerasaurus 8y ago|0 comments
40.
by chimerasaurus 9y ago
Disclaimer - I'm the Cloud Dataproc PM. :) Super good question. For most use cases, GCS is going to give you better performance than using PD. GCS removes some headaches, like replication. In so doing, when you read from GCS you can of
41.
by chimerasaurus 9y ago
We're also looking at making this a bit easier with some API changes (additions, really) and work with other OSS projects, like Apache Airflow. Streaming is a very interesting use case; a focus for us as well. Keeping clusters running
42.
by chimerasaurus 9y ago
This is still not an uncommon problem. Interestingly, we (Cloud Dataproc team) have been trying to work in the opposite direction. A few months ago we launched single-node clusters so people can use Spark on one VM instead of creating crazy
43.
Spark 2.0-preview Now on Google Cloud Dataproc(cloud.google.com)
2 points by chimerasaurus 10y ago|0 comments
44.
by chimerasaurus 11y ago
+1 All the more reason that 15 minutes * 100 nodes * x days can add up very quickly. :)
45.
by chimerasaurus 11y ago
+ YARN :)
46.
by chimerasaurus 11y ago
I think it depends on the use case and service. As a disclaimer, I work for Google on Cloud Dataproc - a managed Spark and Hadoop service. So, I am passionate and focused on Spark clusters from 3 CPUs/3GB ram to 5k+ CPUs and TBs+ of RA
47.
by chimerasaurus 11y ago
In addition, you can also use preemptibles and custom VM machine types (specify the exact ratio of CPU/RAM you want for master + workers) for your clusters. This gives even more control for cost vs resources. Disclaimer - I work at Goo
48.
Google Dataflow (soon Apache Beam) Python SDK Released(beam.incubator.apache.org)
5 points by chimerasaurus 11y ago|0 comments
49.
by chimerasaurus 11y ago
Since Cloud Dataproc is ephemeral it's going to be a (probably) better idea to use GCS over HDFS so there is no data loss. Technically, Cloud Dataproc clusters have both HDFS (on PD) for write/read-intensive operations (and scratc
50.
by chimerasaurus 11y ago
There are several organizations included on the proposal, including Google, who will still be actively involved in the project, if accepted.
51.
by chimerasaurus 11y ago
Going to start backwards on this one. :) The ASF proposal contains a few different components (SDK and runners) all of which have lived on GitHub for awhile (the proposal has links if you're interested.) If accepted as an incubating pr