We need to handle data in a similar level as you mention and also use plain text files as only reliable medium to store data. A recent blog: http://nunobrito198

We need to handle data in a similar level as you mention and also use plain text files as only reliable medium to store data. A recent blog: http://nunobrito1981.blogspot.de/2016/02/how-big-was-triplecheck-tech-in-2015.html http://nunobrito1981.blogspot.de/2016/02/how-big-was-triplec... My advice is to step away from AWS (because of price as you noted). Bare metal servers are the best startup friend for large data in regards to performance and storage. This way you avoid virtualized CPU or distributed file systems that are more of a bottleneck than advantage. Look for GorillaServers at https://www.gorillaservers.com/ https://www.gorillaservers.com/ You get 40Tb storage with 8~16 cores per server, along with 30Tb of bandwidth included for roughly 200 USD/month. This should remove the IOPS limitation and provide enough working space to transform the data. Hope this helps.