Best I can offer is for data structure think about working backwards if possible. Every time I've been involved in projects using 10TB+ data it's the processin

Best I can offer is for data structure think about working backwards if possible. Every time I've been involved in projects using 10TB+ data it's the processing that tends to expose the most bottlenecks. Nothing competes with making a pseudo dataset about 5%-10% of the size and running benchmarks which roughly match your expected workflow. Good luck