2 ms·
I've done with a hundred billion rows. It's not too difficult but it takes a lot of time. Note that elasticsearch scales really easy and really well, I have an
by user5994461 6y ago
I've done with a hundred billion rows. It's not too difficult but it takes a lot of time. Note that elasticsearch scales really easy and really well, I have an article on a small setup many years ago for 12 TB in a small company https://thehftguy.com/2016/09/12/250-gbday-of-logs-with-graylog-lessons-learned/ https://thehftguy.com/2016/09/12/250-gbday-of-logs-with-gray...
The challenge is, a migration would require a lot of hardware resources (storage mainly) to setup a new cluster and the poster couldn't get one damn hard drive to work with, so any migration would have been a death march from the start.
Knowing that and that the original system was out of disk and had no backup of any sort. I would personally question the effort to try to keep all these old logs.