3 ms·
I'm in the process of migrating a ~5.4TB time series database. This is also slow/painful. I wonder if there are any general-purpose/open source tools that could
by thisisbrians 6y ago
I'm in the process of migrating a ~5.4TB time series database. This is also slow/painful. I wonder if there are any general-purpose/open source tools that could help with these sorts of tasks (I've had to roll my own job queue/ETL worker script infrastructure to manage this). In my case, it looks like the full migration will take around 10 calendar days assuming no issues come up.
- user5994461 6y agoThe trick is to find a common baseline format, like CSV or JSON, then you can leverage the bazillion of existing tools. The second trick is to split the data in manageable chunks (a day of logs for example), then you can transfer one unit at a time to the new system. Dump a chunk on one side, load on the other side, verify it's good. It will still take a while but you have a good indication of progress and don't need to start all over if one chunk fails.