3 ms·
He's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
by MichaelGG 11y ago
He's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
- parasubvert 11y agoHe's talking about writing it for Apache Spark. Which really isn't intended for 100 MB (I bet I could write a unix pipe & filter script that's faster than Spark), but is intended for 10 TB through several PB.