4 ms·

Last time this was discussed Zstandard didn't have a splittable mode and by the looks of it[1] they still don't. That doesn't make it a bad algorithm it just me

Last time this was discussed Zstandard didn't have a splittable mode and by the looks of it[1] they still don't. That doesn't make it a bad algorithm it just means that it's not a good choice yet for Hadoop et al. As far as I know no container format has implemented Zstandard yet. Does anyone know any better? It seems like we could use a better alternative to Snappy. [1] https://github.com/facebook/zstd/issues/395 https://github.com/facebook/zstd/issues/395

gopalv 9y ago
> As far as I know no container format has implemented Zstandard yet. ORC has a reserved future enum for ZSTD [1] (my guess is that FB already uses it). The original headache was the patent licensing scheme[2] for the actual codebase, but now that hadoop has a ZStandard built into libhadoop.so, this is much more easy to ship to Apache. My branch is so far behind now that I probably would implement it faster if I started over & used the libhadoop.so impl as the basis - https://github.com/t3rmin4t0r/orc/tree/zstd https://github.com/t3rmin4t0r/orc/tree/zstd [1] - https://issues.apache.org/jira/browse/ORC-46 https://issues.apache.org/jira/browse/ORC-46 [2] - https://github.com/facebook/zstd/issues/775 https://github.com/facebook/zstd/issues/775