3 ms·
Here's a few pointers on big data: * You obviously need more than one machine to do computation. Things become more difficult as you go from one machine to man
by jfasi 8y ago
Here's a few pointers on big data:
* You obviously need more than one machine to do computation. Things become more difficult as you go from one machine to many, so learn about parallel programming tactics like MapReduce[1] and Flume[2].
* Storing stuff is also much more difficult. You'll either be working with distributed filesystems like GFS [3] or Colossus [4], NoSQL databases like bigtable [5], or SQL-like databases like Spanner [6] and F1 [7]
* Algorithms are tricky, because things that take no time at all on a single machine get to be very difficult when done on multiple machines. I don't actually know of any resources on this one, but if you know the algorithms and learn distribution tactics like I outlined above, it should make more and more sense with practice.
* Finally, you need to know your operating systems and networks. What operations are cheap (RAM access, local disk access, most local computation, same-cluster network operations, streaming remote disk access) and what operations are expensive (longer network hops, random-access remote disk operations). You also should be able to ballpark (to an order of magnitude) things like latencies of operations, dataset sizes, QPS's per machine, etc.
Also, when reading the papers, pay more attention to the interface than the implementation. One of the hardest things is transitioning from being spoiled by the local OS interface, where everything is cheap, to the distributed world, where seemingly random things are expensive or impossible.
[1] https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf https://static.googleusercontent.com/media/research.google.c...
[2] https://ai.google/research/pubs/pub35650 https://ai.google/research/pubs/pub35650
[3] https://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf https://static.googleusercontent.com/media/research.google.c...
[4] https://www.systutorials.com/3202/colossus-successor-to-google-file-system-gfs/ https://www.systutorials.com/3202/colossus-successor-to-goog...
[5] https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf https://static.googleusercontent.com/media/research.google.c...
[6] https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c...
[7] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf https://static.googleusercontent.com/media/research.google.c...
- franzwong 8y agoThanks. Appreciate for the links!