5 ms·
Rars: a Rust RAR implementation, mostly written by LLMs
I spent a fortnight using Claude to create specs for every version of RAR, then another using gpt-5.5 to write compressors in Rust.
It's not fast and it's not pretty, but it works.
- deleted 5mo ago[deleted]
- cactusplant7374 5mo ago> and it almost earned me an OpenAI ban Were you flagged for a cybersecurity violation?
- gibspaulding 5mo ago> Well, it turned out that at some time during spec investigation, Claude needed to understand authenticity verification which is a paid feature. With a context full of reverse engineering tools it cracked WinRAR and bypassed product registration, then dutifully documented its crimes in the spec. The docs, when viewed, triggered OpenAI’s alarms and stopped it dead in its tracks. I squashed this out of the git history, and decided not to implement the feature at all. You can draw your own conclusions as to what this says about the state of agentic development.
- themafia 5mo ago> But, it works, and the world now has a free software RAR implementation. Does it? How are you legally intending to use copyright to license this machine output? How would you know it's not encumbered in any way?
- perching_aix 5mo agoReally unsure why this is getting downvoted, to my understanding this is a massive, unsettled concern. It wasn't even a disasm/pseudocode to formal spec flow, and then a separate human implementation. The same human has been in the loop throughout, and large parts of it were generated directly. It's basically guaranteed tainted. Edit: I should have skimmed a bit more patiently, there was in fact no "disasm/pseudocode + the human getting tainted" part to this apparently.
- charcircuit 5mo agoThe human wasn't looking at the copyrighted code and was giving high level steering instructions. If you look at the spec generated it doesn't look like a derivative work of the copyrighted material. The program was generated from the spec. It seems mostly fine from my perspective.
- 0cf8612b2e1e 5mo agoIf I use a decompiler on existing binaries, then some machine translation utility to turn that into a different language, that still feels like a derivative work, even if no human were reviewing the specifics.
- ameliaquining 5mo agoThe idea is to make it so that the parts of the output that are derived from the existing binary are not themselves eligible for copyright protection. I.e., factual descriptions of the file format, without any implementation details from the binary.
- ameliaquining 5mo agoI read the post you're replying to as saying "this is copyright-encumbered and nonfree because it's a derivative work of everything in Claude's and GPT-5.5's training corpus", which is an argument I find fairly tiresome. (Realistically, if courts actually rule that this is the case, this tiny little project will be the least of anyone's concerns.) "This is copyright-encumbered and nonfree because it's a derivative work of the legacy RAR binaries" is a different argument (and seems like it depends on details of the setup that were somewhat glossed over in the post).
- deleted 5mo ago[deleted]
- themafia 5mo agoThe point is, excepting current legal standards which are already very murky, how can _you_ claim copyright, if you don't _know_ it isn't encumbered? You can get these LLMs to generate copyrighted outputs both intentionally and accidentally. This is a known fact; therefore, if you're not checking the output to see if this has occurred then you're potentially generating legal risks for yourself and anyone who uses your code. To not only ignore this for your own use case but to then release the code under a proclaimed license seems legally problematic if not ethically concerning. If you did get sued for infringement I can't imagine that your defense would be that you find the argument tiresome? Honestly, do you think this would never happen, or how would you go about defending your actions here?
- falcor84 5mo agoIn all seriousness, why should anyone care? I always found software IP to be absurd, but this is a particularly absurd situation. We're talking here about a small utility tool implemented from scratch and open sourced, with no apparent intent to make any money from it. Are you concerned about the "encumberence" of using "unlicensed" tools to manipulate .doc, or .pdf, or .mp3 files?! Well I'm not, and if anyone ever tried to sue me for improper access to their proprietary formats, I'll show them some old testament impropriety.
- Georgelemental 5mo agoJudges tend to frown on old testament impropriety. And corporations tend to frown on employees who draw the ire of judges
- davidsong 5mo agoI generally don't anyway. Since the WTFPL came out I've been licensing under that with a warranty clause (don't blame me). My main goal here was an experiment to see how far I could push the technique, and learn things along the way. Regardless of whether people dare to use it commercially or not, we have interoperability for the foreseeable future. As an archivist/computing historian I think that's important.
- esafak 5mo ago> It’s sloppy, it’s slow, it’s almost two megabytes in size and somewhat worse than WinRAR on compression. As mathematicians say, optimization is left as an exercise to the reader. You did the hard part.
- 59nadir 5mo agoI mean, not really...? A vibecoded mess that runs badly, that's not really the hard part for something like compression/decompression tools.
- esafak 5mo agoWhat's your easy way of reverse engineering every previous version of this file format, if you don't think that was the hard part?
- no-name-here 5mo agoWouldn’t previous human-coded implementations already be in LLMs’ training sets? OP article even mentions a number of previous implementations. And even recent LLM “clean room” designs seem to gloss over how the final LLM implementer still has access to previous implementations’ code. Still, OP claims to have done the best job to date at creating (via AI) specs, and the non-optimal Rust implementation, so a net gain?
- 542458 5mo agoI don’t believe any of the extant open source rar implementations cover the range of features and versions OP’s does. I think that’s the point - OP’s isn’t the cleanest or fastest implementation, but it is the most broad open source version available.
- Imustaskforhelp 5mo agoKudos, this is a really cool project (even if it might be AI generated), I have starred the repo, (3rd starrer here) One thing I have been curious at is are there any ways to stop a rar compression mid way and then continue it later? Like suppose I have a compression happening for a large file, then would there be a possibility with this project to shut down the computer mid compression and continue it after starting it again? I would really love it if you can add this functionality!
- davidsong 5mo agoThanks! I guess you could save the state to a file on SIGINT, flush what's been written and pick it back up again if the state file exists when you restart, and use the CRCs of the files to abort if things have changed. I don't fancy doing that for so many versions of RAR, but it would be a cool feature to add it to an `xz` fork. I like the idea.
- Imustaskforhelp 5mo agoThanks it would still be interesting to see this added to xz but supposing the fact that LLM's were able to create the rars project, I suppose it might not be that difficult to add that to rar format eventually. Starting to do it from xz might make the most sense if you like the idea right though. Another idea for rar format that I have which I would love to hear your opinion on is that there are sometimes multiple .part01 .part02 .part03 and so on I have found that when you try to unrar it, it requires all the files at the particular. It would be really beneficial imo if it was possible to have the ability if there was some ability to somehow just unrar .part01 without requiring all the contents of .part02,03 etc. but from my very limited understanding, you also need some (I think last contents) of all files for the de-compression to work. Would it be possible to do something of this endeavour so that you don't require all the parts themselves but just something of a patch of an end, I am not sure about compression algorithms if that might be possible though but it felt like something which was a bit possible albeit hard/difficult to do with rar format. I would be curious to hear your opinions on it, and thanks for responding and I would be really interested in seeing the xz fork that you mentioned!
- slopinthebag 5mo agoHow do we know it's actually correct?
- perching_aix 5mo agoBy using it.
- repelsteeltje 5mo agoIt works == it's correct?
- mjr00 5mo agoThis is Rust we're talking about. It doesn't even need to work; as long as it compiles, it's correct.
- speedgoose 5mo agouse std::fs::File; use std::io::prelude::*; fn main() -> std::io::Result<()> { let mut file = File::create("content.txt")?; file.write_all(b"3!")?; Ok(()) }
- rakel_rakel 5mo ago; cat content.txt 3!;
- dataflow 5mo ago> This is Rust we're talking about. It doesn't even need to work; as long as it compiles, it's correct. No, it doesn't even need to compile. The mere fact that it's in Rust means it's correct.
- perching_aix 5mo agoYes? What do you think fuzzing, unit testing, integration testing is for? It's an empirical evaluation of correctness. Literally just try and see. For actual correctness verification in the strong sense, you'd need to start from a specification written in a formal language so that it's machine checkable, which if I had to guess not even win.rar GmbH has.
- xphos 5mo agoWould it really take 5 years to develop rare compress and decompression that seems an extreme overestimate in time. I don't know of the compressor decompression but that seems really high
- q3k 5mo agoYeah, sounds closer to a 5 week thing, if you know what you're doing.
- self_awareness 5mo ago5 week is a decompressor for 1 version. If this supports multiple versions of RAR, then writing decompressors alone for all of them is probably a year effort of work.
- davidsong 5mo agoWell, it is every version of RAR. Documenting the quirks of rar 1.4, 1.5, 2.0, 2.9, 3.0, 4.0, 5.0 and 7.0, multiple compression strategies, PPMd, RARVM, compression levels, encryption, multi volume support, a huge test corpus, round trips for compatibility... The spec docs are linked.
- xphos 5mo agoYour probably right I should read the spec but I've worked with so many brillant engineers and I am still pretty young so their must be even more. I just think the overselling of complexity is usually what makes things enterprise grade :) (if you know you know)
- unixhero 5mo agoRar means weird in Norwegian and adorable in Swedish. Just for an anecdote.
- mhitza 5mo agoRare, in Romanian.
- vedaba 5mo agoThose almost sound like antonyms which is ironic given how closely related the two languages are
- hackyhacky 5mo agoIt means hello in dinosaur
- npn 5mo agoRar is proprietary. Good luck.
- hayd 5mo agohttps://law.stackexchange.com/a/83552 https://law.stackexchange.com/a/83552 I suppose the question is whether the author had ever entered into a contract limiting reverse engineering...
- npn 5mo agoIf they read the source code of unrar, or in this case, using genai (which obviously included unrar source code in its training set) then yes. You can check the agreement for unrar source code release.
- snvzz 5mo agoClean room is a thing. First use some model to make a specification. Then another to implement it from the specification.
- rebolek 5mo ago> "For the last 15 months or so my hobby has been shouting at Claude" How can you shout at Claude when it’s 1) foobaring, bamblabooing and fghrtawing all the time without telling you what’s going on 2) when it finally interacts, it’s asking for a permission you told it 30 seconds ago "yes and do not ever ask me again until heat death of the Universe" 3) and after all of that, it just spits out: "you’re out of tokens, give up your liver or wait until next Trump’s war"
- wolttam 5mo ago> foobaring, bamblabooing and fghrtawing all the time without telling you what’s going on Oh man, now I have to plug my tool[0]... it doesn't hide anything, but by default tries to provide a pleasant interface (ctrl+o to toggle details similar to CC, but less janky?) Disclaimer: It's way simpler than Claude Code or even pi (on purpose) [0]: https://codeberg.org/mlow/lmcli https://codeberg.org/mlow/lmcli
- davidsong 5mo agoThe proper way to work with Claude or Codex is, IMO, to load up the context with a discussion about what you're doing and why. You go back and forth, pushing back on its opinions and shaping the context until the tokens are ready to flow into the right shape. Every angle you miss is an opportunity for them to slop out all over the place, and, until Codex was mature, the longer you ran the task for, the more it'd spread out and lose shape. Re-shaping the context sometimes involves severe pressures like "wtf is this ugly crap?" or "did I just spot you laying a turd in my codebase again?" and other strong forms of disapproval, mixed with "hmm not sure I like the sound of that"s, to "yea that's much better" to pull it back in the other direction. The trick is to shape the flow before the tide comes in and you end up like King Canute
- periodjet 5mo ago[flagged]
- RIMR 5mo agoFor everyone out there whining about AI, there's one of you whining about being anti-AI. Maybe just cut the unprompted whining?
- deleted 5mo ago[deleted]
- perching_aix 5mo agoWould be great, but then it's a saturation game, and the other side doesn't have any compelling reason to hold back the same way. So it's contingent on how fair the platform is, and what nonverbal, out of band options remain. HN is better than most in this regard thanks to community flagging, but even then there's a lot of it. Ultimately, it'd seem that the ratio you're describing skews a whole lot more towards the anti-ai sentiment side, than towards the anti-anti-ai one (or towards a stalemate). Or rather, that the latter sentiment is not common enough necessarily to thwart such comments. And so you see it reflected verbally instead.
- dclavijo 5mo agoIt wonders me that a coupe of days ago I did the same with Unique and a single skill.md, repo: https://github.com/daedalus/uniq-reconstruction https://github.com/daedalus/uniq-reconstruction, on succes I tried with rar but failed. Kudos
- spprashant 5mo agoI have never attempted something so ambitious with AI, but this feels spot on in terms of experience. As you cede more control to the model, you will find yourself losing control on things like code quality and performance.
- luckystarr 5mo agoYou have to make performance part of the spec. It will then create benchmarks and plan differently. If you omit this, you get what you get.
- donbventures 5mo ago[flagged]
- sntran 5mo agoGood luck with keeping it online. Somebody built `rar-stream` with Rust, and its GitHub is no longer there.
- rvz 5mo ago> but it works. Are you sure "it works?"
- p0w3n3d 5mo agoFirst, isn't RAR compression algorithm proprietary somehow? Second, why compress to RAR if you can compress to 7z?
- WithinReason 5mo agoOne thing it has built in that other algorithms don't is optional redundancy so it can recover data from a damaged archive
- p0w3n3d 5mo agowow, yeah I remember even using it (when moving files on floppy). I'm all for it then, however I wonder if this won't be sued down.
- trembolram 5mo agoThe same guy reimplemented Amiga LZX in Rust. https://bitplane.net/dev/rust/amiga-lzx/ https://bitplane.net/dev/rust/amiga-lzx/ He seems to like doing this kind of stuff.
- Sembiance 5mo agoNeat. FYI your “home” links are broke. And here are 150,000 more unique RAR files if you need to test if your compressor produces the same byte-for-byte output if you unrar and then re-rar these: https://discmaster.textfiles.com/search?format=rar&dedup=dedup&sortBy=b3sum https://discmaster.textfiles.com/search?format=rar&dedup=ded...