3 ms·
This looks very interesting. I've got multiple PB of log files that I'd like to be able to do real-time and batch processing on. Log files are being generated a
by jgrahamc 14y ago
This looks very interesting. I've got multiple PB of log files that I'd like to be able to do real-time and batch processing on. Log files are being generated at a small number of Gbps.
Anyone else got that much data and have a good solution?
- untitledwiz 14y agoI don't think anyone has managed to get interactive speeds (~seconds to max couple of minutes) at the PB scale.
- benbjohnson 14y agoWhat kind of analysis are you trying to do on that dataset? Log files can be verbose as well. Do you need all the data in the logs or can you cut down the data in each row to only what you need?