9 ms·
I made an app that lets you split a file into horcruxes
- Tomino 6y agoVery cool! I built something similar for patent application about 5 years ago for proximity image encryption. Idea was that image was split into X number of encrypted pieces each still being a valid image (disorted or something custom). If you wanted to see the image again, you had to be in close proximity to other parties that have these parts. BLE beacon served as the proximity for the prototype.
- sudhirj 6y agoOne related scheme is fountain codes, where you can split a file into a pseudo infinite stream of blocks such that finding N blocks will almost certainly allow reconstruction. Very useful in UDP / satellite transmission, where you can keep broadcasting these blocks and clients can listen at their convenience. The state of the art codec is RaptorQ, I’ve got a Go library that uses the slightly older Raptor standard to do chunking https://github.com/sudhirj/pump https://github.com/sudhirj/pump
- nullc 6y agohttps://github.com/catid/wirehair https://github.com/catid/wirehair
- jcahill 6y agoFountain codes are seeing use in DNA storage encoding schemes. That's how I use them, at any rate.
- rocky1138 6y agoI know nothing about this but it sounds super interesting. Care to share more?
- jodrellblank 6y agoWhat happens if you can't get N blocks, but say N-1 or N-2? Can you get a partial reconstruction, or nothing at all?
- pkulak 6y agoIt's actually probabilistic, though the probabilities very quickly approach 0 and 1 on each side. But to answer your question, yes, you could recover some of the file if you didn't get as many pieces as you needed. Been a year or so since I did any work with fountain codes, but I believe most implementations send all the chunks, followed by n error correction chunks, so it would depend on how many real chunks you got. The error corrections wouldn't get you anywhere though.
- toolslive 6y agoonline codes [0] just generate random linear equations of the source chunks and send both generated right hand side and the seed that generated the equation. This way there's no state to keep (now, which equations did I send ?) Most generated equations are of degree 2 (so an equation with 2 ones) some (about 1% iirc) are of degree 1 (so pure data). Having less than the critical mass of equations will give you partial reconstruction. But all bets are off regarding how much you will get. They were used in storage by Amplidata in their Amplistor at some point. [0] https://en.wikipedia.org/wiki/Online_codes https://en.wikipedia.org/wiki/Online_codes
- dougmwne 6y agoIt would be a fun modification to require 6/7 or 5/7 files so that you needed to bring a certain number of pieces, but not every piece. Inspired by RAID 5 algorithm that has enough parity to allow one drive failure in a group of 3 or more.
- compsciphd 6y agowhat is the difference between reed solomon erasure codes and shamir secret sharing? i.e. if I just split the secret into a number of reed solomon error correcting blocks (where n blocks are sufficient to recover the full data), is that fundamentally different?
- ScottEvtuch 6y agoPresumably the secret sharing algorithm is more resistant to brute forcing if you have one less than the required pieces. They solve opposite problems
- myself248 6y agoCould you make one equivalent to the other, by R-S encoding a key? And if you have less than the whole key (assuming it's large enough that the missing portion is still bruteforce-resistant), then even a partial reconstruction of the key still gives you none of the data.
- lowercase1 6y agoYes you can! Apply an All or Nothing Transform before RS encoding so that you need all of the original data to decrypt. Then you can only decrypt if you have k of n pieces. You save by a factor of k in piece size over SSS. http://web.eecs.utk.edu/~plank/plank/papers/FAST-2011.pdf http://web.eecs.utk.edu/~plank/plank/papers/FAST-2011.pdf
- OskarS 6y agoYes, it's very different: if you split some error coded thing across several files, each individual file is going to give you some chunk of the data. It's not secret in any way, it's just that you've only got, like, 1/7th of it. With Shamir's secret sharing, having anything less than the required number of files is useless, you can't decrypt any of the data unless you reach the required number.
- compsciphd 6y ago
- nikeee 6y agoThere is also the CLI tool ssss that uses shamir secret sharing to split data: https://linux.die.net/man/1/ssss https://linux.die.net/man/1/ssss
- filoeleven 6y agoHuh, so the predecessor to Horcrux is also an oblique Harry Potter reference, in that it is named in Parceltongue.
- myself248 6y agoBoooo. Hiss.
- hirundo 6y ago> Q) This isn't really in line with how horcruxes work in the harry potter universe! > A) It's pretty close! You can't allow any one horcrux to be used to resurrect the original file (and why would you that would be useless) but you can allow two horcruxes to do it (so only off by one). Checkmate HP fans. Not buying it, and the fact that this is the first FAQ is evidence that the author doesn't really either. A better fit to Tom Riddle's horcrux would simply be a lossy compression copy of the file. Which would admittedly be pretty useless, maybe unless the copy contains a lossy copy of your soul. But then Virgin Galactic is also a pretty good name even though they haven't yet left the solar system. That should be his defense: it's just a cool name.
- cujo 6y agoWhat don't you buy here? I'd say "It's pretty close!" is pretty accurate!
- grawprog 6y agoIt'd be more accurate if the horcrux files were embedded into other files that had to be deleted using an arcane command from the horcrux program to resurrect the original file...while the original file remains in a half existing unreadable state until resurrected. ETA: For even more accuracy, increase the 'readability' of the file(for text only I guess) with each horcrux deleted. Allow them to be added one by one so the original file slowly 'increases it's power'.
- myself248 6y ago> a lossy copy of your soul Name of my next poetry anthology, right there.
- felbane 6y agoThis would be an absolutely perfect album name for an ambient music band called Digital Horcrux.
- chrismorgan 6y ago
- widforss 6y agoI was almost posting a snarky comment about how this is just Shamir's Secret Sharing, which is in no way new. But, hey, this is really cool. It was probably really fun to write, and luminates a cool scheme that too few know anything about.
- ur-whale 6y agoTurns out the only other shamir secret sharing app I know of that's actually usable (ssss or somesuch name) only does the deed for keys, while this does it for files. This is actually pretty useful, but sort of sad it drags the whole go ecosystem with it (who knows what go will look like in 20 years and if this app will still compile and work).
- amelius 6y ago> This is actually pretty useful, but sort of sad it drags the whole go ecosystem with it (who knows what go will look like in 20 years and if this app will still compile and work). This is something which the CS community needs to solve, imho. I.e. provide an executable language with a formal specification that is guaranteed to be available indefinitely. And to make this more useful, other programming languages should provide back-ends targeting this language.
- WorldMaker 6y agoFor better and worse, isn't that what WASM is/has become/will become?
- sneeuwpopsneeuw 6y agoA friend of mine has made something like this for a blockchain hackathon once, around 2 years ago. The technics he used where relatively simple. It stats with some Elliptic-curve cryptography math to split up a single main key into multiple keys. Every person would than have a full copy of the encrypted files and when enough people combine there keys on the blockchain they would get the main key to decrypt the files and from then on it would be public that the files have been opened and by who. This app seams to use Shamir's Secret Sharing, this is something where I am not familiar with, but from how far I understand the Wikipedia article about it. it works roughly the same but it is more general. I'm interested to see if people will actually use this. If anyone has some additional explanations about the differences between these algorithms then that would be very appreciated.
- enimodas 6y agoRecently similar: https://news.ycombinator.com/item?id=23541949 https://news.ycombinator.com/item?id=23541949
- k_sze 6y ago1. Write your will; 2. Split it into N + 1 horcruxes and distribute them to your N children; and a remaining piece to a lawyer; 3. Force them to all come together to decrypt the will for fairness.
- SoylentOrange 6y agoThen what happens if one child who knows they won’t get anything, refuses to provide their piece? BTW K of N cryptography is well known as Shamir’s Secret Sharing. https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
- gramakri 6y agoThis is what is used in hashicorp vault
- mprovost 6y agoAnd in TFA
- hyko 6y agoEven worse, what if one or more children lose their bits and can’t provide their piece?
- deleted 6y ago[deleted]
- acid__ 6y agoHave each child make horcuxes of their horcrux :)
- booleandilemma 6y agoThen what happens if one child who knows they won’t get anything, refuses to provide their piece? Don’t give that child a piece in the first place!
- nanomonkey 6y agoSee also Dark Crystal[https://darkcrystal.pw https://darkcrystal.pw], which uses Shamir's Secret Sharing to break your secret into "shards", allowing you to also set a number of friends that are needed to recreate the secret (less than the total number of shards). Sharing is done over your social network (currently Briar and Secure Scuttlebutt).
- Multicomp 6y agoso is this an alternative to multipar / quickpar / par2 from the usenet days? Looks good. I always try to build redundancy into my offline backups, as if the given backup in my hand is the last backup that hasn't been cooked / melted / flooded etc. ...because one day, it just might! Talking about worst-worst-case scenario, with triple redundancy of online-offsite (can be ransomwared), offline-onsite (can be flooded/burned), and online-onsite (1st line of defense, ie syncthing or a nas)
- mnw21cam 6y agoPar2 is current, still receiving bugfixes, and is used every day as part of my file backup system.
- alternatetwo 6y agorar files having a recovery record is also really helpful for archiving files. I add a 3% redundancy to any archive I create with it because I've had some byte errors in the past.
- verroq 6y agoHow efficient are these splits? If a 100mb file is split in 5, with 3 needed to recombine, we’d expect the pieces to be at least 333mb won’t we?
- dj_mc_merlin 6y agoEach share is (roughly) the same size as the original. In effect, all of them are just encrypted versions of the original.
- ur-whale 6y agoAh, this is slightly disappointing. For some reason, my gut was telling me each piece would be smaller than the original. I wonder if this (horcruxes of size ~ 1/N) is actually possible.
- hinkley 6y agoChunks of size 1/M are possible. 1/N would violate information theory. If your data is compressible, you should do that first.
- wjn0 6y agoWithout some form of compression, I don't think it is. In particular, consider your example (5 horcruxes with 3 needed to reconstruct). View the original file as the interval (0, N) and view it as a set covering problem. If each horcrux covers an interval of size N/3, then if any pair overlaps, there is no third horcrux that can complete the covering. This is a contradiction because 5 horcruxes of size N/3 must overlap somewhere.
- tenplusfive 6y agoIt would be possible to just make the encrypted file publicly available and only distribute the key shares to each "horcrux". The shares themselves should be the size of the key that is used for encryption.
- jesseduffield 6y ago
- bokwoon 6y agoheh, you seem to have messed up the bracket order for the markdown links. I memorise it as the mnemonic "square bracket": first the square [], then the bracket ().
- hinkley 6y ago[I have claimed something](here is my proof)
- jesseduffield 6y agoThose brackets get me every time. Fixed :)
- postit 6y agoI love when the universe throws my own ideas back at me but with a slightly better implementation The backstory on this was me freaking out when I had a newborn coming and I wanted my legacy to be handed to him at the right age if something happened to me.
- tyingq 6y agoInteresting. Feels like you could accomplish the same effect with regular encryption though. Base64 encode the key, pad it with random data that matches the size of splitting into N-1 parts. Then split the encrypted file into N-1 b64 encoded parts. For lowish values of 'N', you could then just decrypt with each "key" until something readable emerges. The key size, algo, etc, could be prepended to each part in plaintext. Or, if you want a variation where no parts are optional, a piece of the key in every split part, with a sufficiently long key.
- ur-whale 6y agoDoes your scheme support M-of-N recovery?
- deleted 6y ago[deleted]
- tyingq 6y agoNo, that's a good point. I was misled by some of the comments that assumed the posted scheme didn't either. However, I'm assuming "all parts present" is one of many desired use cases.
- nbadg 6y agoYou're looking for secret sharing algorithms, which is what this is using behind the hood. The classic (used in OP) is Shamir's secret sharing algorithm [1]. The general gist of it, simplified into the 2-of-N case, is: 1. the encryption key is the y-intercept of a line 2. each shared key ("horcrux" in the OP terminology) is a point along the line 3. more keys are simply more points on the line Once you have any two keys, you can fully define the line and recover the Y intercept. This gives you really strong guarantees because each point on its own reveals nothing about the y-intercept; without satisfying the M-of-N threshold, the value could still be anything. Generalizing this to the M-of-N case simply involves increasing the order of the curve (ie line -> parabola -> ...) [1] https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
- r0rshrk 6y agoDropbox does something similar for cold storage: https://dropbox.tech/infrastructure/how-we-optimized-magic-pocket-for-cold-storage https://dropbox.tech/infrastructure/how-we-optimized-magic-p...