4 ms·
I definitely understand reluctance to trust new data formats. I will say though that we have been using Zstandard in production at Facebook since well before 1
by ctur 10y ago
I definitely understand reluctance to trust new data formats. I will say though that we have been using Zstandard in production at Facebook since well before 1.0 and trust it completely for both data at rest and data in flight. Since 1.0, we've widened that even further to many different parts of the company.
To that end, every release (and pretty much every commit) undergoes significant testing. We also fuzz test heavily and have significant regression tests in place constantly looking at corner cases and verifying known prior bugs stay fixed. It's impossible to prove there are no issues, but so far we've put a lot of effort into being as reliable as possible and seen that play out well in our production environments. We trust it.
- Twirrim 10y agoI feel pretty safe in trusting it. I just like to be careful with a mental "new tech" budget for a project/platform. The more new tech is involved, the more risk there is. The balancing point there being the value the new tech brings. With the platform I'm working on at the moment, the gains are relatively minor: shaving a few seconds off here and there, when things are in the scale of minutes; and saving 20-30MB when things are in the scale of gigabytes, but also not lots of them. As the project grows, and as the new tech we've introduced proves itself, there's always scope to introduce other new tech like Zstandard and start gaining in the storage savings.