In chapter 2, you describe writing the master dataset to a distributed filesystem (presumably HDFS). If you create a new file with a new record for each update

In chapter 2, you describe writing the master dataset to a distributed filesystem (presumably HDFS). If you create a new file with a new record for each update to the dataset, wouldn't this result in lots of small files on the filesystem, resulting of fragmentation. HDFS seems well suited for large files, not small (http://hadoop.apache.org/common/docs/r1.0.0/hdfs_design.html#Data+Organization http://hadoop.apache.org/common/docs/r1.0.0/hdfs_design.html...).