6 ms·
Here's an option you might not have considered, but might be applicable, depending on the read/write speeds you need from those hundreds of gigabytes of data. A
by AnkhMorporkian 10y ago
Here's an option you might not have considered, but might be applicable, depending on the read/write speeds you need from those hundreds of gigabytes of data. Amazon has cloud drive, which costs something like 50 bucks a year, and has unlimited storage.
The neat thing is someone created a FUSE file system called acd_cli that allows you to use the cloud drive as if it were a normal hard drive. The speeds aren't too shabby either; I easily get 150MB/s up and down on my dedicated server, and response times are snappy. Additionally, you can create a unionfs mount so that writes are instant, and you sync new files on a regular schedule.
Of course, that might not be applicable to your use case, but I use mine for many things. I have a plex server running using that with over 13TB of videos, and it works flawlessly. It allows me to run a full plex server with unlimited storage for 20 bucks a month.
- dhruvkar 10y agoI have the same issue. Running a Plex server (~6TB) at home, and thinking I need to put my media in a more accessible location. Are you using Amazon Drive as the one true source and syncing it locally to use with your Plex Server? Or are you pulling/streaming directly from ACD?
- AnkhMorporkian 10y agoMy setup is a little convoluted. I've found that using the write mount really slows me down, so I use a unionfs mount with the writes going to the local hard drive, and the cloud drive being the primary read source. Given that it's unionfs though, anything that I haven't synced will be read from the hard drive. So the order goes: Sonarr/CouchPotato -> Local Hard Drive -> Amazon Cloud Drive -> Plex. I stream directly off the cloud drive for any file older than an hour, as that's my upload interval.
- dhruvkar 10y agonice. so you read from ACD, because you're mainly writing to local? Or is there another reason for NOT reading from local? also, does local completely mirror ACD, or have you found a way around that?
- AnkhMorporkian 10y agoIt doesn't mirror it at all. The local stuff is only a temporary holding place for it to sit until it gets uploaded to ACD, at which point it is unceremoniously deleted. I have >10TB on my cloud drive, but I'm using <10GB on my hard drive.
- dhruvkar 10y agoGreat! Going to give this a try.
- AnkhMorporkian 10y agoGive me a shout if you need any help. I know HN doesn't have private messaging, but you can reach me on IRC on Freenode at AMorpork.
- dhruvkar 10y agowill do, thanks for the offer!
- deleted 10y ago[deleted]
- yurymik 10y agoPlease keep in mind that Amazon is running a variant of content-id and may mistakenly think that your files are infringing someones rights: https://www.reddit.com/r/PleX/comments/4j7wex/any_plex_users_utilizing_amazon_cloud_drive_may/ https://www.reddit.com/r/PleX/comments/4j7wex/any_plex_users... https://www.reddit.com/r/DataHoarder/comments/4j5fsi/after_storing_over_8tb_of_my_films_and_series_on/ https://www.reddit.com/r/DataHoarder/comments/4j5fsi/after_s...
- AnkhMorporkian 10y agoAs I mentioned in another comment, you can avoid all that by using encfs, which encrypts the files before they ever reach amazon. Edit: Just read the first reddit post. I call BS on anyone using encfs consistently and receiving a legitimate email from Amazon. Unless they've broken strong encryption, there's just no way.
- skrowl 10y agoOr they could just be using it as an excuse to get rid of high usage (non profitable) users. Encrypted files + you use a lot of storage and bandwidth = get kicked off.
- DavideNL 10y agoFyi, you might want to check this out (scroll to the bottom and start with the first post): https://amc.ovh/ https://amc.ovh/
- dhruvkar 10y agoexactly what i needed.
- AnkhMorporkian 10y agoI can +1 that. It's definitely not an A->B->C, but it's enough to get you 95% of the way there.
- some-guy 10y agoI've tried this, but I've had sync issues in the past. Also have they gotten rid of their 2GB file upload limit?
- AnkhMorporkian 10y agoRecent versions of acd_cli have largely fixed all the issues I've had in the past. I've found that essentially all issues can be solved by not writing directly to the cloud drive, but using a unionfs and uploading on a cron job. There is no 2gb limit that I have run into. I just checked and I have several >30GB videos uploaded to my cloud drive, so I can safely say there is no issue there. Edit: For some completeness, here's how I have everything mounted and syncing. acdcli mount /mnt/amazon_real --interval 30000000 -ao --uid 112 --gid 118 unionfs-fuse -o cow /mnt/cache=RW:/mnt/amazon_real=RO /mnt/amazon -o allow_other 0 * * * * acdcli upload --remove-source-files /mnt/cache/* /TV/ --max-connections 10; acdcli sync And that's it. The last one is a cronjob that runs every hour, if that wasn't obvious.
- amdavidson 10y agoI get that you probably tweaked this a bit for posting, but I see a disconnect that is relevant to how I want to set this up. In the first line you mounted your whole Cloud Drive at /mnt/amazon_real. On the 3rd line you upload from /mnt/cache to /TV/. Do you actually have a subset of the cloud drive mounted at /mnt/amazon_real? I tried to do that some time ago with acd_cli but wasn't able.
- AnkhMorporkian 10y agoWhoops. The truth is I have 2 cron jobs, one running on the hour to upload TV, and another 30 after to upload movies. I meant to simplify it to just syncing the entire thing for the post, but I neglected to do that properly. I have not looked into only mounting a section of it.
- josefdlange 10y agoFormer Cloud-Driver here. We launched a significant upgrade to the back-end just before I left, which technically speaking does not have the 2GB limit. That being said, I don't know if there aren't any artificial constraints. I was just responsible for the bits :D
- jdc0589 10y agomaybe you guys are using Plex for nothing but family vacation videos and publicly licensed content; but, no way in hell I would use 3rd party storage for my plex setup. On the otherhand, that's a kickass storage setup though.
- AnkhMorporkian 10y agoI use encfs on top of all of that, which means there's no possible way for Amazon to see my files.
- civilian 10y agoDo you have any thoughts on encfs vs. duplicity? http://duplicity.nongnu.org/ http://duplicity.nongnu.org/
- AnkhMorporkian 10y agoThey serve different purposes, and I actually use both for different reasons. Encfs is seamless, it's encryption that doesn't impede normal operations. You can just mount it and you're off to the races. Duplicity on the other hand is a backup solution. I use it to back up my mail server to S3. It's super handy for that, but it really couldn't be used with plex seamlessly like encfs could be.
- dhruvkar 10y agoJust family home videos for me ;)