3 ms·

CSV Same way most web servers log traffic.

• ju-st 10y ago
  Considering the use case ("archiving and offline analysis") this is likely the best choice. If you need some buzzwords or want to get fancy, try HDFS
• IanCal 10y ago
  Strongly recommend this. Your disks are not going to struggle with 1G/day, it's easy to setup & manage, easy to move data around and easy to process later. Worst case is it's quick to try and see if it suits.
• Mahn 10y ago
  Or perhaps SQLite, since it should perform similar to writing to a CSV file and you get querying out of the box afterwards.
  • saosebastiao 10y ago
    SQLite is a major powerhouse for stuff like this, and if your analysis is on a single computer with a single disk, analysis is almost always faster than any sort of highly parallel big data setup. I've loaded, analyzed, and reported on 2TB of data using SQLite in less time than it took for an 8 machine spark cluster took to load the data.