2 ms·
Ask HN: I've recently started using Spark for some simple stream processing amenable to embarrassing parallelism (that also involves making a REST call per row)
by SonOfLilit 3y ago
Ask HN: I've recently started using Spark for some simple stream processing amenable to embarrassing parallelism (that also involves making a REST call per row). It's been a horrible, horrible experience, troubleshooting IT issues and debugging code are extremely time consuming.
Am I missing something or abusing the system or is this the norm?
- 392 3y agoSounds like my experience when tinkering as well. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- appplication 3y agoThis is normal for spark. We have very little expectation even our senior engineers can optimize their own jobs. It’s a difficult system to learn but can be very powerful. That said, making REST calls inside spark sounds like a recipe to accidentally DDoS your target endpoint.