4 ms·
You can spin up a standalone Spark cluster in about an hour and it's pretty easy. And yes you will need to rewrite a small amount of your code but you're doing
by threeseed 7y ago
You can spin up a standalone Spark cluster in about an hour and it's pretty easy.
And yes you will need to rewrite a small amount of your code but you're doing so here as well.
At least you will be able to scale out to much larger volumes in a consistent way.
- zaphar 7y agoFor many tasks a spark cluster is overkill. Streaming will get you a really long way before you have to tackle running a while spark cluster. And you'll probably be able to still do it on your laptop.
- threeseed 7y agoYou can't just apply a streaming paradigm to a batch style workload. And 90% of all tasks are batch orientated.
- zaphar 7y agoBatch is just streaming in really large chunks. And besides if you are messing around with data on your laptop you probably aren't at the stage of working out a batch workload yet. Jumping immediately to this needs to be a batch workload before you have had a chance to play with it is probably overkill.