4 ms·
Since Cloud Dataproc is ephemeral it's going to be a (probably) better idea to use GCS over HDFS so there is no data loss. Technically, Cloud Dataproc clusters
by chimerasaurus 11y ago
Since Cloud Dataproc is ephemeral it's going to be a (probably) better idea to use GCS over HDFS so there is no data loss.
Technically, Cloud Dataproc clusters have both HDFS (on PD) for write/read-intensive operations (and scratch space) along with the GCS connector. GCS is not the default file system, however.