copying and moving data is the bottleneck. but considering the magnitude of traffic and the type of infrastructure they're using to pass this data.. it's extrem

copying and moving data is the bottleneck. but considering the magnitude of traffic and the type of infrastructure they're using to pass this data.. it's extremely negligible. this is big data and the type of analysis of such high-dim information is the real issue. (took me and my team 21-hours to clean/train on wikipedia data) - that is, each iteration was 21 hours long. it took us weeks to deliver results, using 3.7ghz/8gb i5 machines. performance gain is achieved by carefully designing the data stack so that it's split-able, and making sure your routines implement sane message passing with little wastage. the rest of it is having football-sized computer clusters crunch the hell of it so it saves you a fuck-ton of time eventually.