Before you start trying to scale with Scala/Spark, consider if my preferred progression makes more sense: * < 50 million rows: Can your data fit into a standar

Before you start trying to scale with Scala/Spark, consider if my preferred progression makes more sense: * < 50 million rows: Can your data fit into a standard RDBMS? Writing all your business logic in SQL might make more sense. * < 1 billion rows: Can your data fit into a few flat files on disk, with some Python/Perl scripts to manipulate them? * < 1 trillion rows: Can you use an off-the-shelf solution like Google BigQuery or Amazon Redshift? You can also use this rather than flat files depending on the skillset of your team. * < 1 trillion rows: Can you engineer smart data structures & algorithms, write some custom C++, tune the compiler flags, inspect the resulting assembly, cache intermediate results on disk, etc. for a small number of core queries? For anything larger, good luck. You seem to genuinely need a dedicated data engineering team, and similar tools may very well help you scale up. Even so, try to keep things simple. A single binary with well-defined inputs & outputs(even poorly written in C++) is intrinsically simpler than a distributed network(even if the Scala code itself is simpler and well-written), just due to deployment considerations. If you need things to be accessible in real-time from a website, that may also require a sophisticated solution. Consider if you can queue the query and use one of the above approaches, however. It's even harder to deploy a real-time distributed query system with worst-case performance guaranteed to be queryable from the front-end of your website.