4 ms·

Very neat. Btrfs is a practical choice. Looks like the Go http server listens for both changes and jobs? Could you tell me about the advantages git semantics

• jdoliner 12y ago
  > Very neat. Btrfs is a practical choice. Looks like the Go http server listens for both changes and jobs? Right now pfs is just a file system so servers only listen for changes. However longterm this is going to be the case because submitting a job is actually going to be implemented as making a commit in the file system that tells pfs which data to run the job over and the container to use to do so. > Could you tell me about the advantages git semantics add to MapReduce? One is that it gives you an audit log which is very useful for debugging. I often find myself with data in HDFS that doesn't look like what I expect it to and wish that I know what process changed it. Another killer feature is that it removes the distinction between stream processing and static MapReduce. Since our file system is commit based we think of MapReduce as an operation over a stream of commits rather than a static set of files. When more commits come in you can recompute your job for only the changes. > Last, how does Pfs choose to distribute data across containers? Right now pfs uses a simple static hashing algorithm based on filename. This probably isn't going to work for production deployments. We're planning to improve it soon. > You've inspired me to put up an Unbounce page. Let's talk ZFS soon! Yes please.