powturbo

13 ms·

61.
by powturbo 7y ago
There is new AVX512 instruction (_mm512_conflict_epi32) supposed to solve this, but it can't make the histogram construction faster than the scalar functions.
62.
by powturbo 7y ago
I agree and according to a steam hardware survey only 1% are using AVX2. AVX-512 is practically unused. And don't forget the billions ARM CPU's.
63.
by powturbo 7y ago
You can do better, without SIMD: https://github.com/powturbo/TurboBase64 The simple base64 scalar version is also faster the chromium implementation.
64.
by powturbo 7y ago
Not memory speed but extremely fast: https://github.com/powturbo/TurboHist
65.
by powturbo 7y ago
40 GB/s in L1 cache. The TurboBase64 benchmark is more practical. It is impossible to have the speed of a good memcpy when you're copying 33% more like in base64.
66.
by powturbo 7y ago
There is no claim about memcpy. The memcpy is used as indication in the benchmark.
67.
by powturbo 7y ago
Turbobase64: A portable scalar implementation can beat SIMD and saturates the fastest networks and fastest SSDs. It is faster than SSE on Intel/AMD and SIMD NEON on ARM see benchmark at Turbo Base64: https://github.com/
68.
by powturbo 7y ago
Yeah, for benchmarking it is important to use real data, because the compression ratio can significantly differ. You'll have better compression with lossy compression and better precision with user controlled and bounded relative error
69.
by powturbo 7y ago
- Timestamps are very easy to compress down to 0.01% - Floats are hard to compress without quantization or lossy compression. The Gorilla algorithm is not helping so much. You can do your own benchmarks with your data with TurboPFor : http
70.
Show HN: HN: Scalar Beats SIMD: Turbo Base64 Faster Than Other SSE/Neon Libs(github.com)
2 points by powturbo 7y ago|0 comments
71.
by powturbo 7y ago
1 - [TurboPFor]( https://github.com/powturbo/TurboPFor ) 2 - [TurboPFor: an analysis]( https://michael.stapelberg.ch/posts/2019-02-05-turbopfor-ana... ) 3 - [Debian Code Search:Website]( https:/&
72.
Show HN: Debian Code Search: Positional Index, TurboPFor-Compressed(michael.stapelberg.ch)
2 points by powturbo 7y ago|1 comments
73.
Show HN: TurboPFor ARM-Bringing SIMD Integer+FP Compression to the Smartphone(github.com)
1 points by powturbo 7y ago|0 comments
74.
by powturbo 8y ago
See [Static/Dynamic web content compression benchmark] One of the best benchmark about web compression including brotli + different gzip compatible libraries: https://sites.google.com/site/powturbo/home/w
75.
by powturbo 8y ago
Already done. The claims are not based on speculations, but on numbers coming from verifiable benchmarks with " https://github.com/powturbo/TurboBench" including the Daala_EC and the AOM AV1 entropy coder.
76.
Facebook Hit with Patent Suit Over Data Compression Tech(news.bloomberglaw.com)
5 points by powturbo 8y ago|1 comments
77.
by powturbo 8y ago
> "the performance claims just don't hold up" see now the fatcs in https://sites.google.com/site/powturbo/entropy-coder and how rANS based entropy coder outperforms the current AOMedia AV1 entrop
78.
by powturbo 9y ago
see Compression Benchmark at ( https://github.com/powturbo/TurboBench ) reply
79.
by powturbo 9y ago
see Compression Benchmark at ( https://github.com/powturbo/TurboBench )
80.
by powturbo 9y ago
In my benchmarks only the AVX2 is faster than the portable scalar "turbobase64" https://github.com/powturbo/TurboBase64 SIMD does not always mean faster.
81.
by powturbo 10y ago
It may be interesting to look at a web content benchmark with the best gzip compatible compressors and peak memory usage: https://sites.google.com/site/powturbo/home/web-compression Only facts and numbers, no
82.
by powturbo 10y ago
- Byte/Nibble transpose/shuffle for improving compression of binary data (ex. floating point data) - Scalar/SIMD Transpose/Shuffle 8,16,32,64,... bits Dynamic CPU detection and JIT scalar/sse/avx2 switching - 1
83.
Show HN: Turbo Transpose compressor filter for binary/integer/floating point data(github.com)
2 points by powturbo 10y ago|1 comments
84.
by powturbo 10y ago
Yes, you can simply mmap the index file. The index server have not to read the index at the start. The goal of query processing is to decrease the query latency, this is why it is better/necessary to do partitioning. Each query can be
85.
by powturbo 10y ago
Thanks for your nice feedback. 3. About "parallel query processing".... I mean, each thread (index server) is responsible for only one shard/segment/partition. You need a local broker (1 per PC/node) to read client
86.
by powturbo 10y ago
1. What's the license of TurboPFor,TurboRLE ? GPL (see license in the source files). 2. Have you tried comparing TurboPFor that with roaring bitmaps ? No, but because of the size overhead of roaring bitmaps they can be only used for fe
87.
Show HN: TurboHist: Turbo Histogram Construction(github.com)
5 points by powturbo 10y ago|6 comments
88.
Show HN: Turbo Base64 Encoding/Decoding library(github.com)
1 points by powturbo 10y ago|0 comments
89.
Dynamic web content compression benchmark(sites.google.com)
2 points by powturbo 10y ago|0 comments
90.
Show HN: Dynamic Web Content Compression Benchmark(encode.ru)
1 points by powturbo 10y ago|0 comments

More ›