We used to do just that, logging as CSV, but switched to JSON. At a certain scale you don't use Unix tool chain anymore except for a tail and that's for pure d

We used to do just that, logging as CSV, but switched to JSON. At a certain scale you don't use Unix tool chain anymore except for a tail and that's for pure debugging. We log >10TB per day and they go to Hadoop for processing. JSON is crazy verbose. But you pay for flexibility, we want to remove and add fields at will everyday with new business requirements. Its a pain to maintain csv log versions or use the "never remove a column" rule. Compression rocks on JSON you get 90% gzip compression easily. We would have considered logging to thrift if we didn't have this huge flexibility requirement.