3 ms·
Either you work with data that’s not really big data or you have some super awesome engineers who know spark very well, because spark is pretty much not declara
by ramraj07 4y ago
Either you work with data that’s not really big data or you have some super awesome engineers who know spark very well, because spark is pretty much not declarative as much as it says so. You have to constantly battle crap like GC errors, data skew, etc. Even on supposedly managed solutions like databricks. As opposed to solutions like snowflake which are a lot more declarative (nothing is perfect of course).
- hobs 4y agoAnd the vendors are happy to tell you to throw more cores at it instead of grapple with the core issues. One of the biggest problems with spark imo is that people think that the magic cloud is going to make it go fast, when it actually means that the REPL is just muuuuch slower to iterate on.