8 ms·
Kraken: 3x faster decompression than zlib
- muizelaar 10y agoUnfortunately not open source.
- vvanders 10y agoFeel free to downvote me but some people need to eat. Game development doesn't really match well with the consultancy model of most commercial Open Source companies. Also the RAD guys are awesome, they've been putting out solid tools for years.
- lazyjones 10y agoAll valid points, but think of all the open source projects, even browsers, that could be improved with this compression method if it wasn't proprietary. Perhaps it's time to attempt to "free" such software via a cooperative effort, e.g. by talking to RAD about a one-time payment for an open source release and organizing that as a Kickstarter project.
- vvanders 10y agoThat's the rub though, you're talking about a one-time payment. That isn't a sustainable way to keep a company running. The amount of work that RAD puts into stuff like this, BINK, and their other tools if you wanted to do a OTP you'd be looking at something in the neighborhood of 50-60M. Everyone uses their stuff in games since it's that good. You'd have to replace licensing costs not only for each game but each platform those games ship on.
- lambertsimnel 10y agoI wonder whether it would be viable GPLd with a proprietary licences available for a fee. I suppose that might still be costly and would be incompatible with the licences used by some FOSS projects.
- brigade 10y agoI can't find the post now, but one of the RAD developers behind this codec (it's their 3rd ANS based codec I believe) has said that they would likely be willing to open source their ANS family for a one-time payment of iirc around $5 million. No need for the entire RAD stack.
- brigade 10y agoOops, his ballpark was $50 million. I obviously have no idea how much things cost. Still, I think many of the key ideas behind Kraken are out there and not patented. Zstd and lzfse already use ANS variants, for instance. But they're targeting different speed/compression tradeoffs, and aren't implemented as well as Kraken.
- eln1 10y agoGoogle VP10 and WebP are also planned to be ANS-based: https://chromium.googlesource.com/webm/libvpx/+/fb9186d68d498919b290d7f444560f44a1d8d1df https://chromium.googlesource.com/webm/libvpx/+/fb9186d68d49... https://chromium-review.googlesource.com/#/c/338781/ https://chromium-review.googlesource.com/#/c/338781/
- _3u10 10y agoThink about how browsers could be improved if people were willing to pay $5 for them. Perhaps it's time to attempt to "pay" for such software via cooperative effort, eg. by the publisher setting a price for it, and then people paying $5. Why people will pay $5 for coffee they could make at home for 'free' but not for software is beyond me.
- voltagex_ 10y agoA lot of the time people don't understand how expensive software is to develop.
- overcast 10y agoIt's a tangible thing. People STILL have issues paying for something that isn't physically in their hands.
- dilap 10y agoNah, I don't think it's that -- people spend crazy amounts of $ for non-tangible assets in F2P games.
- ferbivore 10y agoI don't think you're talking about the same subset of people here.
- dilap 10y agowhat, people who use browsers and play games? there's got to be a pretty big overlap :) (though it's true that a large fraction of the people that play f2p games don't pay at all.) i know for me personally, one reason i've been reluctant to pay for software is just that they paying-for-software experience has been so terrible -- you have to clumsily reenter your payment information, you get emailed, sometimes slowly, a registration code, which you inevitably lose and breaks your software when you have to reinstall on a new computer, etc.
- 10y ago
- feduzi 10y agoAnd this world needs better algorithms. It depends on what you put first: yourself or others.
- tossaway1 10y agoSeems like a false choice. Plenty of people make a living doing things that help the world.
- deleted 10y ago[deleted]
- feduzi 10y agoI didn't suggest just 2 options. It is a range of choices you get, starting from "you", ending with "others", pick any point. And sure, you're right "Plenty of people make a living doing things that help the world.", but why this "Kraken" is not Open Source then? Disclaimer: it is a discussion, I don't try to blame RAD Game Tools for not releasing the source of Kraken. Just curios of what others think.
- jagtesh 10y agoPutting yourself first isn't a bad thing, especially where bread and butter is involved. Might I suggest dual licensing with a provision for free-to-use in non-commercial projects? Something similar to QT.
- feduzi 10y agoWhere did you get the idea that putting yourself first is a bad thing in my comment? I didn't mean that. What I was trying to say that you can pick any of this. At the end of the day you're the one who did the job, so it is up to you to decide what to do with it.
- tossaway1 10y agoQt is also free to use in commercial projects. I believe fftw had the sort of dual license you have in mind.
- deleted 10y ago[deleted]
- microcolonel 10y agoI get what you mean, but it is still a downside for the format. "contact us" pricing is fine and all, but it means it's irrelevant to me most of the time. Their software could be the coolest; but the chances of them taking an individual seriously are probably low.
- loup-vaillant 10y ago> some people need to eat. That'll need to be addressed some day: jobs are being automated away (yesterday typists, today cashiers, tomorrow drivers…), and those people need to eat too. Then there's artificial scarcity. Unlike food, software can be copied at negligible costs. Proprietary software puts arbitrary (and harmful) restrictions to that copying so that its developer can eat. When we get our act together and finally have a form of guaranteed income, we won't need copyright or patents any more, because there won't be any starving artist or author or developer. Proprietary software is harmful and needs to die. The need for proprietary software is harmful and needs to die. I long for the day they will.
- unsigner 10y agoThere's nothing artificial about the scarcity of people like cbloom and ryg, competent enough to produce something like Kraken. "getting your act together" sounds scary to someone who grew up in Communist Eastern Europe; I surely hope you don't get your act together.
- loup-vaillant 10y agoI was talking about the marginal cost of software. Of course the first copy is expensive. Of course talent is scarce. Of course time is scarce. Still, once it's done, copying it cost near nothing. If you raise that marginal cost above this natural "near zero", that creates an incentive to reinvent the wheel. Your actually scarce resource, talent, is being wasted on duplicated effort. Proprietary software causes duplicated efforts. Not an ideal use of our resources, don't you think? --- > "getting your act together" sounds scary to someone who grew up in Communist Eastern Europe Come on, that was a small group of people seizing power to the detriment of most everyone else. You sound like it's the only alternative to our corporate capitalism. Seriously?
- deepnet 10y agoRAD own the IP completely so they can dual-license GPL and Commercial. No non-Free commercial product or game would touch the GPL version so license revenues would continue. RAD-tech could then be used in Free software too, gaining publicity, adoption and developer mind-share. If innovation comes downstream in the Free version then RAD could license that from the coders for their commercial product. The good will from their initial gift of Freeing their tech may make upstream patch re-licensing on good terms. The social benefits of dual licensing should be encouraged by a GPL tax-relief akin to charitable write-downs.
- hhjkjhkjhhlkj 10y agoYou know that open source and free (gratis) are different. Money can change hands.
- MichaelGG 10y agoWhat about LZHAM; I thought that was the top "slow compress, good ratio, fast decompress" contender for games and such?
- panic 10y agoThis article has some comparisons between Oodle's LZNA and LZHAM (at least as of a year ago): http://cbloomrants.blogspot.com/2015/05/05-09-15-oodle-lzna.html http://cbloomrants.blogspot.com/2015/05/05-09-15-oodle-lzna....
- Johnie 10y agoThere's too many projects named Kraken: * http://krakenjs.com/ http://krakenjs.com/ - Javascript framework * https://www.kraken.com/ https://www.kraken.com/ - BTC exchange * https://kraken.io/ https://kraken.io/ - Image Optimizer
- dikaiosune 10y agohttps://ccb.jhu.edu/software/kraken/ https://ccb.jhu.edu/software/kraken/ <- taxonomic ID assignment for metagenomic sequence samples
- oso2k 10y agoHahaha...that's ironic.
- Osiris 10y agoAgreed. Another: GitKraken - https://www.gitkraken.com https://www.gitkraken.com
- codersbrew 10y agolove gitkracken
- dwc 10y agoThere are too many projects of any name (most never see the light of day), but somehow all these Krakens get released.
- loup-vaillant 10y agoWell, what is dead may never die… is what the Krakens say…
- josteink 10y agoI guess someone was ordered to release the Kraken(s), and as Eddie Izzard famously observed[1], nobody ever issued an order to retrieve the Kraken. To the grumpy crowd: Sorry in adance for the reddit-like comment. I feel a bit light-spirited today :) [1] https://www.youtube.com/watch?v=Cyq4zT0h0yk https://www.youtube.com/watch?v=Cyq4zT0h0yk
- mmozeiko 10y agoFew benchmarks from @rygorous: https://gist.github.com/rygorous/67c78fde615e078117ea187047291913 https://gist.github.com/rygorous/67c78fde615e078117ea1870472... (on PC) https://gist.github.com/rygorous/9563760e24463a37984c117b5728638d https://gist.github.com/rygorous/9563760e24463a37984c117b572... (on PS4)
- Gratsby 10y agoBut what is it's Weissman score?
- gjkood 10y agoI was just watching Silicon Valley yesterday and this is exactly what I was thinking when I read the title. 6.0+?
- noir_lord 10y agoIs the speed difference algorithmic, implementation optimization or a mix of both?
- mistercow 10y agoThey say it's a mix of both.
- EvgeniyZh 10y agoHow is it compared to Zstandard https://github.com/Cyan4973/zstd https://github.com/Cyan4973/zstd ?
- wolf550e 10y agoEspecially with level 22 with --ultra, I found the compression to be good and decompression very fast. Can't wait for 1.0 for format freeze (currently decompressor contains code for previous versions, but that's a hack).
- kazinator 10y agoIs this 3x faster than zlib at decompressing on zlib-compressed data, or 3x faster on samples compressed with its own format versus the same samples compressed with zlib?
- mmozeiko 10y agoIt is custom compression algorithm. It has its own format.
- rb2k_ 10y agoAnother interesting compression project is "Zstandard" Open source, high performance, decent compression and a "dictionary mode" that's especially fun when it comes to compressing a lot of small data. http://www.zstd.net http://www.zstd.net Previous discussion over at https://news.ycombinator.com/item?id=8941955 https://news.ycombinator.com/item?id=8941955
- richdougherty 10y agoHere's a nice graph from the zstd site, showing the trade off between compression and speed for several different algorithms: https://raw.githubusercontent.com/Cyan4973/zstd/master/images/DCspeed5.png https://raw.githubusercontent.com/Cyan4973/zstd/master/image...
- eln1 10y agoAlso lzturbo is very interesting, closed source but you one can download exec: https://sites.google.com/site/powturbo/ https://sites.google.com/site/powturbo/ https://github.com/powturbo/TurboBench https://github.com/powturbo/TurboBench
- Jerry2 10y agoDoes anyone know how Apple's lzfse [0] compares to some of these other compression algos? >Beginning with iOS 9 and OSX 10.11 El Capitan, we provide Apple’s proprietary compression algorithm, LZFSE. LZFSE is a new algorithm, matching the compression ratio of ZLIB level 5, but with much higher energy efficiency and speed (between 2x and 3x) for both encode and decode operations. I'm interested because of the low energy characteristics... [0] : https://developer.apple.com/library/tvos/documentation/Performance/Reference/Compression/ https://developer.apple.com/library/tvos/documentation/Perfo...
- userbinator 10y agoI guess it's yet another optimised LZ variant, and it's no surprise that LZ-based/inspired algorithms are fast, because their main loop is very simple and does not involve the intense bit manipulation of Huffman/Arithmetic algorithms. That's how very carefully optimised LZ can end up being faster than memcpy() in some cases: http://www.oldskool.org/pc/lz4_8088 http://www.oldskool.org/pc/lz4_8088
- stellar2 10y agoLooking at the figures, it seems to have roughly the same performance as Zstandard in high compression mode.
- iso-8859-1 10y agoIt is no secret that vanilla zlib is slow. Would have been interesting if they compared with https://github.com/Dead2/zlib-ng https://github.com/Dead2/zlib-ng