janwas

33 ms·

421.
by janwas 5y ago
> Maybe modern codecs are seeing the writing on the wall and are increasing macroblock size for better parallel processing 10 years into the future We did indeed do this for JPEG XL - the future is now :) 256x256 pixel groups are indepen
422.
by janwas 5y ago
Yes, JPEG XL lossy and lossless formats both enable parallel and region of interest decoding by splitting the image into "groups"/tiles. The parallel speedup for lossless mode depends on the speed setting.
423.
by janwas 5y ago
> If your goal is to understand how hardware SIMD works, you're probably better off sticking to C code with intrinsics Agreed, and we're also using intrinsics in time-critical places. I am confident we will be able to hide both
424.
by janwas 5y ago
OK, that's fair. Godbolt is a great tool and autovectorization gets initial results quickly, but the price is eternal vigilance. Any compiler update/code maintenance risks falling on the wrong side of some compiler heuristic.
425.
by janwas 5y ago
Nice writeup with helpful diagrams, thanks for sharing! Readers might also find this short intro [1] helpful, including tips on porting. (Disclosure: author) 1: https://github.com/google/highway/blob/master&#x
426.
by janwas 5y ago
Nice. For anyone interested in ARM, they also have a guide with diagrams [1] and a searchable reference [2]. 1: https://developer.arm.com/documentation/102159/0400/Permutat... 2: https://developer.
427.
by janwas 5y ago
+1 to intrinsics or wrappers giving us more control over performance. > Plus, when you get the hang of it, writing your own SIMD library is fairly simple hm.. it's indeed easy to start, but maintaining https://github.com&
428.
by janwas 5y ago
+1 for restrict, that certainly helps. Out of curiosity, what's your use case where autovectorization works well? Personally, I have often been disappointed. Not much progress in 2 years: http://www.0x80.pl/notesen&#x2F
429.
by janwas 6y ago
Important to note that was a Xeon Silver, which is much more affected than Gold. More in-depth discussion here: https://lemire.me/blog/2018/08/13/the-dangers-of-avx-512-thr...
430.
by janwas 7y ago
:) There are updates every ~3 months at https://jpeg.org/news.html .
431.
by janwas 7y ago
Yes, quantization constraint was removed post-CD and Annex O was recently merged into the modular mode to simplify the standard. JXL's page count is 1/2 to 1/3 of contemporary video codecs.
432.
by janwas 7y ago
Lossless transcoding of JPEG bitstreams is done by Brunsli, there is also lossless storage of pixels, based on tech from FUIF plus an adaptive predictor by Alex Rhatushnyak.
433.
by janwas 7y ago
> I'm not criticising the JPEG XL code, it seems to be quite clean at first glance! Thanks, I understood your point about each project including its own definitions. At one point there was http://www.bookofhook.com/p
434.
by janwas 7y ago
IIRC J2K is about 40 MPixels/s per core. 4K60p is 498 MPixel/s :)
435.
by janwas 7y ago
JXL dev here. Thanks for raising these points. Indeed regrettable we have to have a bunch of boilerplate. FYI we actually do support hooks for malloc (JpegxlMemoryManagerStruct) and threading (JpegxlParallelRunner) precisely in order to be
436.
by janwas 7y ago
Yes indeed, for the lossless mode (based on tech by the same author) we're seeing about 45-82% of GIF size, and 60-80% of (A)PNG depending on content.
437.
by janwas 7y ago
Good news: the reference software ( http://gitlab.com/wg1/jpeg-xl ) can decode about 50 Megapixels/s on a single Skylake core, with 3.1-3.7x speedup on 4 cores. Encoding throughput is configurable from ~1 MP/s
438.
by janwas 7y ago
Hi, we indeed support byte for byte reconstruction of the original JPEG codestream.
439.
by janwas 8y ago
HighwayHash uses PSHUFB to spread entropy from twin 32x32=64 muls across 128-bit vectors [or vector halves, on AVX2]. See ZipperMerge in https://github.com/google/highwayhash/blob/master/highwayhas... .
440.
by janwas 9y ago
PractRand is a useful test suite.
441.
by janwas 9y ago
The FourCC at the beginning is indeed your encoding-id. This is common in file formats (e.g. RIFF). Out of curiosity, how is the 32-bit space insufficient?
442.
by janwas 9y ago
If you're interested in this kind of thing, dc_predictor.cc might be fun to look at. It computes the predictors in parallel (rather than the usual 'do the same thing to 32 pixels') because here pixels depend on causal neighbo
443.
by janwas 10y ago
We did an experiment with comparisons of Guetzli and (slightly larger) libjpeg output: https://arxiv.org/abs/1703.04416 Turns out that 75% of the 614 ratings are in favor of the Guetzli version.
444.
by janwas 11y ago
Agreed :) We (and SipHash developers) do care about short strings, though. Scripting language hash table inputs are typically around 10 bytes, so we can't ignore finalization overhead.
445.
by janwas 11y ago
Interesting. How big do you need the hashes to be?
446.
by janwas 11y ago
I hadn't seen CLHASH yet, thank you for mentioning it. Looks even faster but only XOR universal (meaning XORed hashes are uniformly distributed, not the hashes themselves). In particular, it fails smhasher's avalanche test (becaus
447.
by janwas 11y ago
For the record, S. Gueron has some papers on similar tree hashes ( http://dx.doi.org/10.4236/jis.2014.53010 ) and there's more theory in http://sponge.noekeon.org/TreeHashing.pdf .
448.
by janwas 11y ago
You're welcome, thank you for suggesting these :) We do need a contributor agreement click-through first, please see CONTRIBUTING.
449.
by janwas 11y ago
With a 64-bit hash, you shouldn't be seeing collisions at all on small datasets. A test such as smhasher can verify this.
450.
by janwas 11y ago
Thanks for mentioning, I have removed the smhasher dependency. We have an enhanced version of it which cannot be yet be released due to a dependency on threading libraries.

More ›