3 ms·

How does the data get efficiently moved to the spun up cloud nodes? The article mentions parceling out jobs to nodes with independent storage, but how does tha

by hcrisp 10y ago

How does the data get efficiently moved to the spun up cloud nodes? The article mentions parceling out jobs to nodes with independent storage, but how does that help data-intensive work? Are they doing something besides just copying data to the local storage of the cloud nodes? It seems like that would be a bottleneck.

godmodus 10y ago
copying and moving data is the bottleneck. but considering the magnitude of traffic and the type of infrastructure they're using to pass this data.. it's extremely negligible. this is big data and the type of analysis of such high-dim information is the real issue. (took me and my team 21-hours to clean/train on wikipedia data) - that is, each iteration was 21 hours long. it took us weeks to deliver results, using 3.7ghz/8gb i5 machines. performance gain is achieved by carefully designing the data stack so that it's split-able, and making sure your routines implement sane message passing with little wastage. the rest of it is having football-sized computer clusters crunch the hell of it so it saves you a fuck-ton of time eventually.
boulos 10y ago
See my note above, but we have great peering with Fermilab and I failed to mention this above: us-central1 in Council Bluffs, Iowa is also quite close to Fermilab (less than 500 miles) so the latency was pretty good too. [Edit: And I should add that by copying into GCS, they were able to grab compute anywhere in us-central1 without worrying about it. Similarly, on preemption, you don't need to transfer all the data from home, you just grab it from GCS again]. Disclosure: I work on Google Cloud and helped with this effort.
dekhn 10y ago
They used gcsfuse. It lets you mount GCS buckets as filesystem mounts and do reads/writes on objects, from GCE hosts. gcsfuse is quite fast. It also caches files locally, if you ask it to. So, presumably, subsequent jobs access input files via a cache hit.

by hcrisp 10y ago

godmodus 10y ago

copying and moving data is the bottleneck. but considering the magnitude of traffic and the type of infrastructure they're using to pass this data.. it's extremely negligible. this is big data and the type of analysis of such high-dim information is the real issue. (took me and my team 21-hours to clean/train on wikipedia data) - that is, each iteration was 21 hours long. it took us weeks to deliver results, using 3.7ghz/8gb i5 machines. performance gain is achieved by carefully designing the data stack so that it's split-able, and making sure your routines implement sane message passing with little wastage. the rest of it is having football-sized computer clusters crunch the hell of it so it saves you a fuck-ton of time eventually.

boulos 10y ago

See my note above, but we have great peering with Fermilab and I failed to mention this above: us-central1 in Council Bluffs, Iowa is also quite close to Fermilab (less than 500 miles) so the latency was pretty good too. [Edit: And I should add that by copying into GCS, they were able to grab compute anywhere in us-central1 without worrying about it. Similarly, on preemption, you don't need to transfer all the data from home, you just grab it from GCS again]. Disclosure: I work on Google Cloud and helped with this effort.

dekhn 10y ago

They used gcsfuse. It lets you mount GCS buckets as filesystem mounts and do reads/writes on objects, from GCE hosts. gcsfuse is quite fast. It also caches files locally, if you ask it to. So, presumably, subsequent jobs access input files via a cache hit.