terrelln

13 ms·

91.
by terrelln 8y ago
I've generally found that thread-local storage is easily misused. My general rule of thumb is to avoid thread-locals wherever possible, but if they are required keep thread-local state constant sized, not one per object. The memory usa
92.
by terrelln 8y ago
A lot of compression ends up being compress once, decompress one to a few times, so the faster end of the spectrum fits well. Additionally, compression has to fit into the existing system, and existing systems already have tight constraints
93.
by terrelln 8y ago
Looking at the brotli API, I believe I spotted the source of our disagreements. Brotli separates out the window size from the quality, but in zstd the level implies the window size, unless you use advanced parameters. For example, level 3 h
94.
by terrelln 8y ago
Levels above 19, which has a 8 MB window size, may only be used with the --ultra flag, whose documentation says "requires more memory".
95.
by terrelln 8y ago
We recently added negative compression levels that extends the fast end of the spectrum significantly. We are also working on incrementally improving the strong end of the compression ratio spectrum. We don't expect plain zstd to compr
96.
by terrelln 8y ago
There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd di
97.
by terrelln 8y ago
Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have heade
98.
by terrelln 8y ago
The x axis is compression speed, and the y axis is compression ratio. Zstandard outperforms zlib in compression ratio, compression speed, and decompression speed (not shown). The only reason to stick with zlib is for compatibility with syst
99.
by terrelln 8y ago
GPLv2 was added before the patents grant was removed, so zstd could be included in the linux kernel.
100.
by terrelln 8y ago
If you're using lrzip, you should also check out zstd long range mode [0]. It uses a long window (128 MB by default, up to 2 GB), together with an efficient search strategy, and multithreading. For example, a 2 GB window, with 4 thread
101.
by terrelln 8y ago
Btrfs already supports multithreaded compression and decompression. Each 128 KB block is (de)compressed with a single thread, but multiple blocks can be (de)compressed in parallel. The zstd CLI supports multithreaded compression with the fl
102.
by terrelln 8y ago
I implemented zstd compression in btrfs/squashfs, and work on upstream zstd. * Btrfs compression is multithreaded, and can use up to the number of cores available on the system. * Compression might not help speed on SSDs, but it should
103.
by terrelln 8y ago
Both PEXs and XARs package a python script and its dependencies in single hermetic file. PEX is a self-extracting zip file which has to be fully extracted before being run. The extracted files could potentially be modified. XAR is a self-mo
104.
by terrelln 8y ago
I spent some time today investigating what exactly is causing the difference between native and XAR start times. I confirmed the culprit is `pkg_resources.load_entry_point()`. Modern installations using wheels should avoid this overhead, an
105.
by terrelln 8y ago
Facebook's PAR is a self-extracting zip file, I assume Google's is similar. XARs are self-mounting SquashFS archives (a compressed read only filesystem). This means that XARs don't have to be extracted to a temporary director
106.
by terrelln 8y ago
We currently don't have a nice open source API for building node apps, but would welcome PRs that get us in this direction! There are two ways to build a node app using the XAR builder tools. 1. Use the `make_xar` tool which will creat
107.
by terrelln 8y ago
The current Python XARs rely on Python being on the system path. But it would be easy to build a custom Python XAR with the XAR builder library that includes the Python executable and makes sure to use the packaged executable.
108.
by terrelln 8y ago
Admittedly I haven’t profiled this yet, but my guess is it is a constant overhead of setting up pkg_resources that the native code uses to load the entry point. The test against native start speed was hot, so the pages required were already
109.
by terrelln 8y ago
XARs are just self mounting compressed readonly filesystems with an executable inside. We get hermitic dependencies by setting the PYTHONPATH, LD_LIBRARY_PATH and such in the bootstrapping script. One big benefit is that the filesystem only
110.
XARs: An efficient system for self-contained executables(code.fb.com)
246 points by terrelln 8y ago|120 comments
111.
Zstdandard v1.3.5 “Dictionary Edition”(github.com)
1 points by terrelln 8y ago|0 comments
112.
by terrelln 9y ago
Zstandard version 0.5 (and all other versions < 0.8.1) were pre-releases where forward compatibility was never intended. However, Zstandard is still backward compatible with versions down to version 0.4. All releases since August 2016 h
113.
by terrelln 9y ago
It is mentioned at the bottom of the README, but it isn't very findable. I've opened a PR https://github.com/facebook/zstd/pull/1085 .
114.
by terrelln 9y ago
The GPLv2 license was added for inclusion in the kernel, you may choose either GPLv2 or BSD. See the header file https://github.com/facebook/zstd/blob/dev/lib/zstd.h .
115.
by terrelln 9y ago
The library is dual licensed under plain BSD [1] and GPLv2 [2]. [1] https://github.com/facebook/zstd/blob/dev/LICENSE [2] https://github.com/facebook/zstd/blob/dev/COP
116.
by terrelln 9y ago
Zstandard also maintain ABI stability for a portion of the API, and require a macro definition to access the unstable parts.
117.
by terrelln 9y ago
Newer versions are forwards and backwards compatible with older versions. The compression format stabilized starting with version 0.8.1.
118.
Zstandard v1.3.4 – faster everything(github.com)
202 points by terrelln 9y ago|33 comments
119.
by terrelln 9y ago
Generally, pages in long ranges will be accessed less than those in a short ranges, and pages beyond the window size will never be accessed. You can construct data that doesn't fit this pattern, but its probably a good bet to make. Wit
120.
by terrelln 9y ago
Upstream SquashFUSE also has zstd support. https://github.com/vasi/squashfuse

More ›