nigeltao

7 ms·

31.
by nigeltao 3y ago
jj is faster than jq. However, jsonptr is even faster and also runs in a self-imposed SECCOMP_MODE_STRICT sandbox (very secure; also implies no dynamically allocated memory). $ time cat citylots.json | jq -cM .features[10000].properties
32.
by nigeltao 4y ago
I'd be curious to know if SIMD cares about the big-endian design decision. You can always do SIMD shuffles near where you load and store the bytes, but it'd be simpler and probably faster if you don't. Similarly, BSWAP /
33.
by nigeltao 4y ago
That's how WUFFS (Wrangling Untrusted File Formats Safely) works: https://github.com/google/wuffs#what-does-compile-time-check...
34.
by nigeltao 4y ago
> The point remains though that you need to somehow get both the encoder side and the decoder side upgraded to benefit from the advantage, Well, for PNG, Apple have shipped exactly that. It was something they could unilaterally do with
35.
by nigeltao 4y ago
> I think you must be doing something very wrong if you measure less than 1 Mpx/s for fjxl. It should reach at least 50 Mpx/s or so, In case it wasn't clear, the 0.630 fjxl encode speed number in the top level README.md in
36.
by nigeltao 4y ago
> Did you compile fjxl with or without SIMD? I had not. Thanks for the tip. Fjxl encoding speed is now about 20x faster than libpng, not 10x, but still mid-table. Decoding speed is unaffected. https://github.com/nigeltao&
37.
by nigeltao 4y ago
I agree that compression ratio is the primary metric (but not the only important metric). I just found "[better] in all ways", explicitly calling out encode speed, incongruous with simply ignoring decode speed. As for which is mor
38.
by nigeltao 4y ago
fjxl is the "JXL_Lossless/f" lines in my full_benchmarks.txt link. It's about mid-table in terms of encode speed. I clocked it at about 10x faster than libpng, not 100x faster. If you look solely at encode speed, fjxl lo
39.
by nigeltao 4y ago
IIUC, the quality knob (not the effort knob) isn't relevant for lossless encoding. Knobs are also less relevant for lossless decoding , where there's only one correct output for any given input. In any case, my code ( https:&#x
40.
by nigeltao 4y ago
From the "Lossless Compression Performance" section of the article: > JPEG XL can do lossless image compression in a way that beats existing formats (in particular PNG) in all ways: it can be faster to encode, produces smaller
41.
by nigeltao 4y ago
Here's another SIMD implementation, with commentary: https://github.com/google/wuffs/blob/main/std/adler32/common... Like the fpng implementation, it's SSE (128-bit registers), but th
42.
by nigeltao 4y ago
https://github.com/facebook/zstd/tree/dev/doc/educational_de... is a self-contained zstd decoder. I get a 64 KB dynamically linked executable after running "make" in that directory. $
43.
by nigeltao 5y ago
I agree that it's similar but ultimately different problems. Copy/pasting from https://github.com/google/wuffs/blob/main/doc/related-work.m... gives: > Kaitai Struct is in a similar spa
44.
by nigeltao 5y ago
IconVG author here. Happy to discuss. Animation is issue #2 on https://github.com/google/iconvg and I have some ideas but no code yet. I'm also midway through changing the current "version 0" format into
45.
by nigeltao 5y ago
> Several times the compression/decompression speed of gzip Just be careful that you're comparing against the best implementation of gzip. One recent re-implementation of zcat was 3.1x faster than /bin/zcat (and the C
46.
by nigeltao 5y ago
JWCC (JSON With Commas and Comments) gives you exactly that, without other things that are arguably a mistake. https://nigeltao.github.io/blog/2021/json-with-commas-commen...
47.
by nigeltao 5y ago
See JWCC (JSON With Commas and Comments) https://nigeltao.github.io/blog/2021/json-with-commas-commen...
48.
by nigeltao 5y ago
> Only two levels of detail per file There are multiple levels of detail per file. Each LoD is specified by two numbers: a lower and upper bound on the height-in-pixels that enables the subsequent drawings.
49.
by nigeltao 5y ago
Hi. IconVG author here. Thanks for the attention. It's still a work-in-progress and I'm actually considering some major file format changes in the coming weeks. As some of you have noticed, the docs aren't complete. I wasn&#x
50.
by nigeltao 6y ago
> Do you accept PRs for the Rust benchmarks? Sure! I'll take a PR that adds script/bench-rust-png-decoder to the Wuffs repo. I'll also take patches to script/bench-rust-png. I'm not as fluent in Rust as I am in C
51.
by nigeltao 6y ago
> I'm curious if it is at all competitive with the Wuffs decoder You could try adapting https://github.com/google/wuffs/tree/main/script/bench-rust-... > https://github.com&#x2
52.
by nigeltao 6y ago
Thanks for the kind words!
53.
by nigeltao 6y ago
> smhasher implements both Ah, I'm based on aappleby/smasher, which is what the original location ( http://code.google.com/p/smhasher ) redirects to. I didn't realize that you have a more active fork. &
54.
by nigeltao 6y ago
Sorry, that part is relatively new and poorly documented. Choosy is sort of vtable-ish. It selects between method implementations (which all have the same signature), typically based on cpuid (e.g. if the CPU is SSE4.2+PCLMUL capable, set f
55.
by nigeltao 6y ago
That could get to parity with "Fastest", but it wouldn't match it with "Safest".
56.
by nigeltao 6y ago
> Looks like there isn't an encoder for PNG There isn't one yet, but I'd like to have one at some point. https://github.com/google/wuffs/blob/main/doc/roadmap.md
57.
by nigeltao 6y ago
> programmers must manage pointer offsets to each heap structure I'm sorry but I don't understand this statement. Can you re-phrase it? > I would be more interested how they plan to handle IO effects or if they want to omit
58.
by nigeltao 6y ago
> Are there any compiletime benchmarks? It used to take a couple of seconds to compile Wuffs' standard library. It's back down to a few hundred milliseconds. https://nigeltao.github.io/blog/2020/dumbin
59.
by nigeltao 6y ago
Yeah, that's a renderer for the IconVG file format. As I've said previously: IconVG is a compact, binary format for simple vector graphics: icons, logos, glyphs and emoji. IconVG is similar in concept to SVG (Scalable Vector Graph
60.
by nigeltao 6y ago
simdjson is indeed good stuff, but its focus is CPU efficiency, not memory efficiency. For example, it requires the entire input to be in memory. It does not support streaming input.

More ›