8 ms·

Building your own secure storage space that mirrors Dropbox's functionality

• deleted 15y ago
  [deleted]
• willvarfar 15y ago
  I'd like to see compression, anonymity and encryption built into the storage. And de-dupe. This is not a contrary aim from the compression, anonymity and encryption bit.
  • drivebyacct2 15y ago
    >And de-dupe. This is not a contrary aim from the compression, anonymity and encryption bit. From a technical point, they largely are contrary aims.
    • willvarfar 15y ago
      The server need not be party to the key that is used to exchange the filenames themselves between the clients. The content can be compressed in blocks before upload; dynamic, anchored block sizes would increase de-dup. The compressed blocks can be individually encrypted with a key derived from the filename and block. There's still plenty of opportunity for whole-file de-dupe there.
    • keyist 15y ago
      Tarsnap gives deduplication, compression, and encryption. Some technical background on how it does this here: http://www.tarsnap.com/efficiency.html http://www.tarsnap.com/efficiency.html There is a detailed breakdown provided as well. Sample output: Total size Compressed size All archives 697 MB 323 MB (unique data) 215 MB 100 MB This archive 148 MB 67 MB New data 17 MB 5.2 MB Edit: Don't want to misrepresent Tarsnap's granularity. The print-stats option gives size in bytes. Above output was generated with the --humanize-numbers option.
      • nknight 15y ago
        I assume this is only deduplication within a single account? Because I don't see anything there on how one would provide dedupe across crypto domains (which, to the best of my knowledge, should be theoretically impossible if the crypto is done right).
        • cperciva 15y ago
          Correct.
      • pbhjpbhj 15y ago
        Just been looking at Tarsnap. It looks good but I'm curious what happens if the author/creator/owner/maintainer (singular) gets run over by a bus?
        • cperciva 15y ago
          I do my best to avoid busses. (I've always been more of a fan of switched point-to-point connections than busses, to tell the truth.) In all seriousness, if I get hit by a bus Tarsnap probably won't live on without me -- but it runs itself smoothly enough that there should be plenty of time for people to download their data.
          • pbhjpbhj 15y ago
            Thanks for your reply. I'd been looking at the FAQ and it seemed like there wasn't much redundancy built in - not wishing ill of you at all.
            • cperciva 15y ago
              You're quite right, and it's a reasonable thing to ask. I don't mind.
      • cperciva 15y ago
        More sample output, from a heftier machine, in case anyone is curious: Total size Compressed size All archives 3321059166180 1113881916696 (unique data) 116996664943 27182261608 This archive 13831036740 4855451330 New data 336950399 62343192 (Yes, that's 3.3 TB deduplicated and compressed down to 27 GB.)
    • JoachimSchipper 15y ago
      You can deduplicate within accounts; cross-account deduplication is indeed a problem.
• samstave 15y ago
  THANK THE GODS Now let me tell you what is REALLY needed :) Teamwork. I work on a virtual team - we all work independently from our homes on client projects. It would be great to create the following for real enterprise version of this: Create an master account. /lipsync/ Then have project/team folders under /lipsync /lipsync/1 /lipsync/2 /lipsync/3 /lipsync/4 Then have users A, B, C, D Each user can have subscriptions to the project/team folders. But they also have their root /lipsync/ account Thus I can have my lipsync account - and in that I can be on team 1 and 4 with user D Users B and C have subscriptions to 2 and 3 and the master server is all four. This is how I have been wanting dropbox to be able to accomodate a virtual company. You can sort of accomplish this with "shared folders" in drop-box - but because I cant have a server of my own I have to pay .... EDIT: Wow - they jsut came out with Teams (or I have not seen this before... but it is still $13/month per user - which isnt that bad - but you still dont have a local server.) Anyway, good work - I will use lipsync.
• rdl 15y ago
  Good luck getting it to work for mobile apps built to Dropbox's API, which is my main use of dropbox.
• akent 15y ago
  TLDR: The code is at https://github.com/philcryer/lipsync https://github.com/philcryer/lipsync
• audionerd 15y ago
  Is this just a way to trigger rsync every time a change happens to a given folder? Looks like this command powers it: rsync -rav --stats --log-file=/home/$USER_NAME/.lipsyncd/lipsyncd.log -e "ssh -l $USER_NAME -p $SSH_PORT" --delete $REMOTE_HOST:$LOCAL_DIR $REMOTE_DIR Couldn't you just toss that in a Guardfile and get the same effect? https://github.com/guard/guard/ Or am I oversimplifying it?
  • mariocesar 15y ago
    while inotifywait -e modify -e create -e delete -e move -r $1; do rsync -vrae 'ssh -p 2299' --delete $1 noentrar@noentrar.net:files/$2 done Here you have → https://forrst.com/posts/Two_10_seconds_scripts_to_keep_your_data_synchro-BBg https://forrst.com/posts/Two_10_seconds_scripts_to_keep_your... I made this long time ago, the same concept just with inotify, to don't abuse the my small bandwith
• ericabiz 15y ago
  The real selling point of Dropbox, for me, is the ability to just drag and drop files to a folder in Windows and have them be shared. I can also create a subfolder--then I can hit the Dropbox website and type in an email address, and voila, my files in that folder are shared with that person. I agree with Dropbox having security issues and would like to see a replacement. I hope this project implements the drag-and-drop functionality cross-platform. Then I could really recommend it to non-technical users (or some enterprising soul on HN could use it to build a more secure Dropbox clone for non-technical end users.)
  • hackermom 15y ago
    Correct me if I'm wrong, but doesn't FUSE allow this as well?
• nikcub 15y ago
  co-incidentally, I wrote a dropbox replacement this week using AppEngine. It is just a webdav server with a simple web interface. I will release the code at some point this weekend
• roel_v 15y ago
  Or, use AeroFS and get something that actually works like Dropbox, and not what somebody who has only read a description of what Dropbox does thinks it does. (also, I lol'ed at "vetted by the community as being 'a good idea'" and backing that put with a screenshot of a Reddit post (of all places) where a couple of people say 'yeah that's the ticket')
  • urza 15y ago
    Any spare invites to AeroFS, anyone? I would like to give it a try.
  • kermitthehermit 15y ago
    Sure, use yet another proprietary thing to share your files. What could possibly go wrong?
    • roel_v 15y ago
      Sure, rely on a hacked together contraption of scripts that have no idea about conflict resolution and requires manual editing and configuration for each machine you want to install it on, and doesn't offer near the same amount of features than other options. What could possibly go wrong?
      • kermitthehermit 15y ago
        You would've known I don't like this scripted thingy either if you had read my other comment. Also, personal work, confidential data and other stuff should never go anywhere near something like dropbox, aerofs and anything like this.
• kermitthehermit 15y ago
  It's not even close to dropbox. It's supposed to have more features like detecting when a file was also changed locally and remote and create a new file from the remote file. It seems very fragile and I wouldn't trust it to backup my data somewhere.
• larelli 15y ago
  Every time I read about the X-th "clone with Dropbox's functionality" I wonder where it will store its files. How many of the potential users have access to online storage they really control themselves? Instead of having the files on Dropbox they end up on AWS or the like. Thats because they try to copy Dropbox down to the flaw, that it doesn't encrypt files on the serverside. For me, any solution would have to include the capability to upload only encrypted files to the server. I know of the duplicity project, which does that for simple, manually triggered backups, but which once caused me headaches to get the data back from partially corrupted files.
  • jpdoctor 15y ago
    > Every time I read about the X-th "clone with Dropbox's functionality" I wonder where it will store its files. I have a different reaction: I keep wondering how long Dropbox can lead in a market with such low barriers to entry.
    • shabble 15y ago
      I presume that the de-duplication saves them enough bandwidth and storage to allow a much lower price point (as well as the free plans) A secure system can't do either of those things, as far as I know. Assuming the data is encrypted per-user, on their local devices, you can't easily compress it (because the files are now very high entropy), and you can't scan for preexisting files, because you'd be giving an attacker an oracle about which files (or rather, which file hashes) exist already. Even then, you could probably use some traffic analysis to deduce something about the data. Having data securely stored also removes some of the dropbox unique(ish?) features, like being able to share content with people via the web. The pretty much diametrical opposite of dropbox is tarsnap. Everything else sits somewhere in the middle.
• sidman 15y ago
  I dont really know what the big deal is. If you have a file that is top secret dont have it on the internet whether it be your email, in dropbox or even on your own aws/hosted server. Keep it on a flash drive and have it stapled to your arm if you want to transport it. For images that you want to share, or files that you kinda dont give a rats about if they were to get compromised or disappear, place them in places on the internet that match their confidentiality requirements (email - for secure, dropbox - for kinda secure or your own private server - for very secure) I think arguing whether a new solution is required because dropbox can't do the job or whether dropbox can keep your files safe is a moot point. If your files are on the internet they are never 100% safe. Just keep your most private files on an external hard disk and have that disk detached from your computer and the only risk you run is if someone robs your house. I personally think dropbox is great for what i use it for, i put images that i want to share, i transfer files to friends overseas and we all happily share our stuff easily and seamlessly and if i wanna access it i login through the browser or install a new client. WIN. No other service at the moment does it this easily for me. If any of my files where compromised, well whatever, a few holiday pictures or some itineraries. The legal responsibility is put on Dropbox to keep your stuff safe, but I think the "common sense" responsibility is put on the user to decide whether s/he thinks dropbox is the place for a specific file.
  • mike-cardwell 15y ago
    I don't get your argument. It seems that you're saying that if something can't be made 100% secure, then there is no point trying to make it more secure than it already is.
    • mattme 15y ago
      All security is compromise–Bruce Schneier.
    • sidman 15y ago
      Its not an argument, its a state of mind in keeping your information that you value safe. Im saying, as things get more secure you can start to move your information there (as i said with email, dropbox or your own servers). Use the online medium that matches your documents safety requirements. If your a security guru then out of all those your own servers that you harden yourself would probably be the most secure. For non security guru's maybe email or dropbox do a better job. No one ever said dont make it more secure, but as the sophistication of the security methods goes up over time so will the sophistication of the crackers/hackers. This means your judgement not matter how much better security gets will always play a apart. Im stating that you shouldnt rely on others for security. Whether dropbox, or a clone. Stating that dropbox as a vendor cant be trusted isnt a valid as it will apply to all the clones. Bottom line you should take responsibility of your own files. Any new system that tries to be more secure will "most probably" still have security issues with it anyway.
• nico_h 15y ago
  the site is at http://lipsync.it/ http://lipsync.it/
• nico_h 15y ago
  The site looks exctly like http://jekyllrb.com/ http://jekyllrb.com/ . Are the project related ?
• jeza 15y ago
  I used rsync to sync files I was working on to a server >10 years ago. Only difference is that it can detect changes to file locally then update the server. To retrieve updates from the server, it relies on cron to poll the server, rather than implement some kind of push notifications. That said, it's always nice to ensure people are aware of what alternatives exist.
• andreasvc 15y ago
  This approach seems to overlook the bigger picture which is that Dropbox makes sure it doesn't lose your data. Aside from that the web interface and dealing with conflicts is an important issue. Just imitating the syncing is not going to cut it, and if you're not outsourcing the storage it rather defeats the purpose of not having to worry about the data anymore.
• jgfoot 15y ago
  What I'd like to see is an easy-to-set up way to run my own version of Dropbox on a small server in my own home. It needs to have a web interface, not just rsync, because a lot of people (like me) don't have access to anything remotely like rsync on our work machines.
  • deleted 15y ago
    [deleted]
  • mtogo 15y ago
    > don't have access to anything remotely like rsync What...? Why not?
    • jgfoot 15y ago
      I am an employee in a non-technical position working in a locked-down environment with (appropriately) paranoid IT staff. The only applications on my work computer that can talk to the Internet are a mail client and a web browser. There are millions of me.
      • reemrevnivek 15y ago
        So you're saying that you expect an application in your browser to reach down and monitor, upload, and download gigabytes of data to your filesystem? Your appropriately paranoid IT staff would collapse in convulsions of terror if this were possible. Fortunately, it's not. The reason Dropbox (and rsync, and lipsync) are native apps isn't because the developers are unaware that there are people in locked down environment and need a browser-based tool, it's because the apps need to be native.
        • ceejayoz 15y ago
          I think he's saying that it'd be nice to have a web interface for getting/putting one or two files on machines that can't have the client installed. Dropbox has this.
• hackermom 15y ago
  I do this myself on my OpenBSD server by simply using an SFTP-only account that has its home directory set to a virtual filesystem that uses encryption through the svnd(4) node. If I want to move the entire encrypted filesystem to another server or so I just copy the file its contained in, and mount it there. Setting this up took about 5 minutes.
  • strags 15y ago
    If I'm understanding you correctly (and my apologies if I'm not), this isn't really quite the same thing - it sounds like a remote, encrypted filesystem. Dropbox and lipsync are file-syncing mechanisms, not filesystems. So, if you're offline, you can continue to work - your changes will be pushed to the server next time you're online.
• gks 15y ago
  People should check out Strongspace (https://www.strongspace.com/ https://www.strongspace.com/) Combined with their OS X app it makes a reasonably compelling alternative for mac users. http://blog.strongspace.com/announcing-strongspace-app-for-the-mac http://blog.strongspace.com/announcing-strongspace-app-for-t...
• evantahler 15y ago
  Try one in Ruby? https://github.com/evantahler/synchzor https://github.com/evantahler/synchzor
• executive 15y ago
  Why would I want a login page that lets anyone login to any account without a password?
• strags 15y ago
  While it's cool to see people replacing Dropbox with OS alternatives, this only seems to capture a fragment of Dropbox's functionality. How, for instance, does it deal with conflicts where files have been changed on two machines independently prior to sync? Dropbox is (somewhat) clever about it, and renames the conflicted versions - and IIRC you can resolve the conflicts manually. At least both files are available on both machines - not sure how this situation is dealt with by lipsync. Dropbox push-updates are immediate. Lipsync relies on a cron job to kick off a receiving file sync. Dropbox will sync directly between clients on a LAN - great for when teams are sometimes working in an office, and sometimes at home. Dropbox maintains revision history - does this? Dropbox has a web interface, mobile apps, etc... I realize that Rome wasn't built in a day, but until at least the first two points are addressed, this isn't much of a replacement.
• beagle3 15y ago
  sparkleshare is much closer to dropbox. Mac and Linux only for now, git back-end.
• Ingaz 15y ago
  Thanks. I will propose do something like this on my work. Even if it's not practical, it will be fun to implement it.