4 ms·
I migrated off apache parquet to a very simple columnar format. Cut processing times in half, reduced RAM usage by almost 90%, and (as it turns out) dodged thi
by marginalia_nu 2y ago
I migrated off apache parquet to a very simple columnar format. Cut processing times in half, reduced RAM usage by almost 90%, and (as it turns out) dodged this security vulnerability.
I don't want to make too harsh remarks about the project, as it may simply not have been the right tool for my use case, though it sure gave me a lot of issues.
- ryan-duve 2y agoWhat "very simple columnar format" did you switch to?
- marginalia_nu 2y agohttps://github.com/MarginaliaSearch/SlopData https://github.com/MarginaliaSearch/SlopData Writeup about some of the ideas that went into it: https://www.marginalia.nu/log/a_112_slop_ideas/ https://www.marginalia.nu/log/a_112_slop_ideas/