3 ms·

Yann will be giving a talk on Zstandard at today's @Scale 2016 conference, and the video will be posted. He can answer the most technical questions about Zstan

by ctur 10y ago

Yann will be giving a talk on Zstandard at today's @Scale 2016 conference, and the video will be posted. He can answer the most technical questions about Zstandard, but I may be able to answer some as well; we both work on compression at Facebook.

tmd83 10y ago
I am really looking forward to this. I usually like to read more than vidoes but for complicated topic with a good presenter it can actually be a comprehensive starting point. Would the video also be posted today or we will have to wait? One thing I haven't figured out from either today's post or Yann's blog is whether Zstandard is switching between huff0 and FSE depending on compression level or is it somehow using both together? Also the post says its both OoO friendly and multi-core friendly but the speed benchmarks are those in a single core context or multi-core? Does only the format/algorithm multi-core friendly or the standard cli can run multi-threaded.
- ctur 10y ago
  I'm not sure when videos will be posted -- hopefully soon. All benchmarks today are single threaded. The algorithm itself is single threaded, but can be parallelized across cores. We will soon release a pzstd command line utility to demonstrate this, similar to pigz, which accelerates both compression and decompression. Zstandard uses both huff0 and FSE together when it compresses -- it doesn't switch between them based on the input.
- twitch_checksum 10y ago
  Video is now available here : https://atscaleconference.com/videos/real-time-data-compression-at-scale/ https://atscaleconference.com/videos/real-time-data-compress...
deleted 10y ago
[deleted]
socmag 10y ago
Hey quick question, and sounds awesome. If I wanted to use this in the pipeline of my servers Journaling system, is there any requirement that I restart the stream periodically. That is to say, should I use it per journal entry (probably not a good idea for short messages), for the entire uptime of the writer, or with periodic restarts? Obviously can measure and find out for myself, but wondered if you had any thoughts. Thanks!

by ctur 10y ago

tmd83 10y ago

I am really looking forward to this. I usually like to read more than vidoes but for complicated topic with a good presenter it can actually be a comprehensive starting point. Would the video also be posted today or we will have to wait? One thing I haven't figured out from either today's post or Yann's blog is whether Zstandard is switching between huff0 and FSE depending on compression level or is it somehow using both together? Also the post says its both OoO friendly and multi-core friendly but the speed benchmarks are those in a single core context or multi-core? Does only the format/algorithm multi-core friendly or the standard cli can run multi-threaded.

ctur 10y ago
I'm not sure when videos will be posted -- hopefully soon. All benchmarks today are single threaded. The algorithm itself is single threaded, but can be parallelized across cores. We will soon release a pzstd command line utility to demonstrate this, similar to pigz, which accelerates both compression and decompression. Zstandard uses both huff0 and FSE together when it compresses -- it doesn't switch between them based on the input.
twitch_checksum 10y ago
Video is now available here : https://atscaleconference.com/videos/real-time-data-compression-at-scale/ https://atscaleconference.com/videos/real-time-data-compress...

deleted 10y ago

[deleted]

socmag 10y ago

Hey quick question, and sounds awesome. If I wanted to use this in the pipeline of my servers Journaling system, is there any requirement that I restart the stream periodically. That is to say, should I use it per journal entry (probably not a good idea for short messages), for the entire uptime of the writer, or with periodic restarts? Obviously can measure and find out for myself, but wondered if you had any thoughts. Thanks!