14 ms·

A Storage Crisis

127 points by vikrum 5y ago

tmaly 5y ago
I am faced with this issue of dealing with hundreds of GB of family video and pictures everytime I upgrade my wife’s phone. The newer phones always have a higher resolution camera so file sizes keep increasing. I would love a simple easy to use tape drive that just works. Something I could backup files to for 20-25 years.
ksec 5y ago
I think there is another point worth mentioning. The Data we own or create are increasing. Taking more photos than ever, higher resolution, Video in 4K or 60 frames. Friends sending their media over. So not only does the usage increases, the actual size of those unit increases as well. And yet Hardrive price / GB hasn't fallen much at all. From 2013 to 2021 now, when it was $0.035 to $0.04 /GB. Mostly because much HDD patter density has stalled.
Gravityloss 5y ago
How much of the actual information in the high resolution photos is just noise? So we buy sensors with more megapixels that just save more noise. Then we need to buy more storage capacity, more bandwidth, more processing, more battery etc. Since the cameras have been hitting closer and closer to physical limits but something still needs to be sold... It's like people buying clothes from the mall and taking them straight to a storage building. They don't have enough space at home. And they won't have time to wear so many clothes anyway.
orasis 5y ago
I’m still happy with my setup of Drobo + 2 rotating time machine backup HDs. I’ve probably been using it close to 10 years now.
PaulHoule 5y ago
I think $6 a month for "unlimited data" is going to end badly. The AWS or Azure price is high, but it's a scalable price, a real price.
- nateroling 5y ago
  Yup, I'm currently experimenting with using an AWS bucket alongside [CloudMounter](https://cloudmounter.net https://cloudmounter.net) to create an Archive disk for things I don't really need day-to-day, but would like to hang onto. I'm currently not backing that data up, though. It's not critical, and the chances of AWS losing it are low enough that I'm not too worried. The biggest risk would be myself accidentally deleting it. One thing I have decided for sure, though, is that archiving lots of small files is awful. Much better to wrap them up in a tarfile, uncompressed.
  - terpimost 5y ago
    Amazon Drive is fine. No need to have a mirror of all that data on my computer. With that I could have selected sync or no sync at all. + unlimited photo storage. Amazon Photo and Amazon Drive it’s like a single product.
    - ValentineC 5y ago
      I've used Amazon Drive back when they offered their unlimited product, and they've lost/corrupted some of my files before. I personally wouldn't recommend Amazon Drive as a backup destination to anyone.
- wgjordan 5y ago
  > I think $6 a month for "unlimited data" is going to end badly. Backblaze has been offering their 'unlimited data' for over a decade [1], and it hasn't ended badly for them yet. It's fully sustainable because they only lose money on a few customers (like that one customer storing 430TB for $6/month [2]), while most customers use much less storage than that, so the service remains profitable overall. The soft limit of its 'unlimited' comes from it being a personal-backup mirroring service and not a fully-external cloud storage, so the service only fits certain use cases. [1] https://www.backblaze.com/blog/all-in-on-unlimited-backup/ https://www.backblaze.com/blog/all-in-on-unlimited-backup/ [2] https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backblaze_cloud_team_managing_750/ejlbsqo https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backb...
bethecloud 5y ago
I recommend looking into decentralized cloud storage solutions, which are really the next generation of AWS S3-like services. Check out storj.io
- acdha 5y ago
  Decentralization adds a good deal risk and operational overhead but it doesn't really change the core problem that people are generating more data than many want to pay for. If you want to store a non-trivial amount of data, someone needs to get paid to maintain storage pools and validate multiple copies.
  - sliken 5y ago
    I've always wondered why nobody does p2p backups. Take whatever the backups are, say 1TB, split it into 12 pieces, add as many pieces as the user wants (say 6 on average), then you can recover with any 12. When you add another 1TB, add another 18 peers. Monitor your peers, only trust the ones with a track record of successful challenges, and of course let you white list peers you trust (like friends and family). Of course the perfect challenge is just a restore, but some checksome of a range of a blob could be useful as well, and consume less bandwidth. Drop peers that are unreliable, or ask for too much bandwidth for their restores. Encrypt the files before adding reed solomon, use a unique encryption key for each peer that stores 1/Nth of your backup set. That way you can "pay" for your storage by just adding that much more local disk space to trade with your peers.
    - wmf 5y ago
      It's been tried plenty of times. P2P software is really complex but the use case is people who aren't willing to pay, so maintaining the software isn't sustainable.
    - acdha 5y ago
      A couple problems off the top of my head: 1. Everything is harder to work with: you have to deal with less reliable networks and storage, computers which aren't on all of the time, very slow uplinks, etc. Since these aren't professionally managed systems, too, you have less visibility — did that node just drop offline because the hard drive failed, losing everything, Comcast is having a bad day, the owner just bought a new one and wiped the old one without unenrolling it, or because it rebooted and is almost back up? 2. People are selfish: I don't want Netflix getting slow because you decided to retrieve your data, I'll complain if I hit a storage limit on my computer due to your stored backup data, etc. This forces you do deal with things like traffic shaping and storage rebalancing more aggressively and those are hard problems to get a popular balance on. Consider, for example, what happens when someone uses your service and it goes well but then they run out of space and need to clear some up in a hurry (_especially_ if they put on their cowboy hat and just delete a bunch of large files because they know they aren't the only ones with a copy). 3. The solutions to the previous problems make the cost problem worse: storing more copies can avoid some of the problems but then you need to figure out how to get the network to support, say, 5 copies instead of 2-3. 4. Consider what happens the first time the police bust someone for a major crime and their data is backed up on your computer. Not many people are enthusiastic about going into court to prove the negative assertion that they didn't have a decryption key. Only peering with trusted systems avoids some of these issues but not all and the _big_ problem is that the upper bound for how much this service is worth is basically the cost of iCloud/Dropbox/S3/Backblaze/etc. The savings you can get between the fixed operational costs and what those services charge is probably not enough to support development.
deleted 5y ago
[deleted]
TheDudeMan 5y ago
If you don't want to pay for the cloud, then there is only one sane option: You keep upgrading your local storage so that you never have more than a few devices. Every six months, buy a new giant disk and decommission as many smaller disks as you can.
- arthurcolle 5y ago
  This sounds extremely expensive
  - amarshall 5y ago
    Cloud storage is really expensive, though. Putting 4 TB in S3 costs US$94.21/month before any transfer costs. A 4 TB CMR HDD is ~$140. That doesn’t include power costs, but over a year you get $990 to spend on that and other things from the cost difference vs. S3 (plus you can sell your old drives when you’re done with them). (I use B2 for cloud storage backup since it’s a lot cheaper than S3 at US$20 for 4 TB, but that still is ~$105/year more than local)
    - dragontamer 5y ago
      > 4 TB CMR HDD is ~$140 And that's with bad pricing! Last year, 4TB was $120 and 6TB was ~$150. Building out a NAS for $1000 ($600 in 4x hard drives, $400 for other components) is very reasonable. Last year that was 4x6TB == 12TB storage + 12TB redundancy, but this year prices are worse so you "only" get 8TB + 8TB redundancy. $400 can afford a Synology or various NAS devices. It can also afford a new desktop that you can install FreeNAS or whatever onto. ----------- Eventually, when the 8TB is not enough, just buy a new HBA card and shove 4x more hard drives in there for a 2nd storage on the same NAS. Maybe 8TB x 4 == 16TB usable + 16TB redundancy. Except no need to copy everything over, just keep the old 8TB cluster working, and just start writing to the new 16TB storage.
      - rhacker 5y ago
        This is the true answer to this entire thread.
the_third_wave 5y ago
Assuming you have family living elsewhere but reachable through a fast internet connection you can do what I do by making a deal with them: they hang your backup box off their net and you will do the same for them. The backup box is some piece of computing equipment with storage media attached, e.g. a single board computer hooked up to a JBOD tower. Depending on the level of trust between you and your family you can use the thing as an rsnapshot target - giving you fine-grained direct access to time-based snapshots (I use 4-hour intervals for my rsnapshot targets which are located on-premises in different buildings spread over the farm) or as a repository of encrypted tarballs, or something in between. Allow the drives to spin down to save power, they'll be active only a fraction of the day. The average power consumption of the whole contraption does not need to exceed 10-15W making electricity costs negligible. You can have as much storage capacity as you want/can afford at the moment, keep for for as long as you want or until it breaks without having to pay any fees (other than hosting their contraption on your network - possibly including building it for them if they're not that computer-savvy).
- deleted 5y ago
  [deleted]
- yjftsjthsd-h 5y ago
  > Depending on the level of trust between you and your family you can use the thing as an rsnapshot target - giving you fine-grained direct access to time-based snapshots (I use 4-hour intervals for my rsnapshot targets which are located on-premises in different buildings spread over the farm) or as a repository of encrypted tarballs, or something in between. These days, openzfs native encryption is the best of all worlds, I think.
  - the_third_wave 5y ago
    In that case, use that. ZFS seems to be a somewhat touchy subject with some people using it as widely as possible while others - myself included - prefer more modular storage systems where the tasks of volume manager, striping/slicing/raid management, encryption layers and file systems are performed by discrete software layers. Both systems work, both have their pros and cons, in the end it comes down what you value the most.
lordnacho 5y ago
What are the requirements? Large capacity, redundancy, reasonable access speed, always on but maintenance downtime of even a few hours a year tolerable? Easy but your data isn't yours: sync your data to GDrive or Apple or whatever, and sync a NAS to that. A little harder but still doable: get a Hetzner and set that up as your storage, set up your own access, sync to local NAS. A Hetz is also really useful for running a load of other services, so for 50 bucks or so it seems pretty reasonable. You could just buy a huge disk and run a server in your house, but it gets annoying in various ways. Kids unplug the power, it creates heat, maybe noise, multiple disks end up needing management, that kind of thing.
darklighter3 5y ago
Amazon Snowball? https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/
wmf 5y ago
The solution is either NAS or cloud archive (not backup) such as SmugMug or ExpanDrive.
JohnJamesRambo 5y ago
Almost everything I create doesn’t really need to be saved and I could never find it if I needed to. I just let Google Photos keep them. It’s good enough.
ur-whale 5y ago
The real question being, of course, given that your life is finite ... how often will you actually look at any of these pics. The more pix there are, the less likely it is you'll ever even open any of these.
- tablespoon 5y ago
  > The real question being, of course, given that your life is finite ... how often will you actually look at any of these pics. > The more pix there are, the less likely it is you'll ever even open any of these. Well, there are also future generations to think about, but their interest will fall off too (until you reach the genealogical profile level, which maxes out at a few portrait-type pictures of any regular individual). It's pretty essential to aggressively curate and organize data like this.
  - sliken 5y ago
    Dunno, tagging seems worth it, deleting not so much. Storage is getting cheaper ever year. I've looked through a many albums from parents, grand parents, and great grand parents. I hope that my pictures make it that far, but seems like the world needs a "free" (except for storage/bandwidth) photo backup service to have any chance of surviving multiple generations.
    - tablespoon 5y ago
      > Dunno, tagging seems worth it, deleting not so much. Storage is getting cheaper ever year. The way I think about it is that you need relatively small "SAVE THIS" drive/disc. Sure, storage is getting cheaper, but you don't want to burden someone with some massive storage array, and you can't count on whatever software you used to tag to still work in someone else's hands.
      - sliken 5y ago
        Dunno, can you guess what will be important to the next generation? 2 generations? 3? You might play a small part of important events, people, etc that will not be realized later. Even silly things like the first robin of the season, or a picture at the pool on the hottest day of the year.
D13Fd 5y ago
I back up about 15tb of photos with Arq to Wasabi cloud storage. It has been running with zero maintenance (other than occasional partial restores) since late 2018. I transitioned off of AWS cloud storage when they raised the prices. I’m not sure if Wasabi is still the cheapest and fastest, but they have been great to deal with. And Arq is an excellent set-it-and-forget-it encrypted cloud backup app. I also run a server with a 16tb RAID 1 array and a set of local backup drives. Sadly it is almost full, and the volume of data makes it a hassle to upgrade (not to mention the cost). I’ve found standard 1Gig Ethernet to be just barely fast enough for editing photos over the local network. However, for my own sanity, I usually do the initial editing on a local drive before sending the files to the server (and from there to the cloud backup).
- wmf 5y ago
  Note that he's looking for primary storage not backup.
- chillage 5y ago
  How do you deal with merging Lightroom catalog files if they are all indexed remote? Sounds like a tricky workflow
wyager 5y ago
Spend a few hundred to a few thousand buck and build a chunky NAS. Get a bunch of 12TB SSDs and put them in a RAIDZn. Also, what shitty phone takes 108MP photos? That's guaranteed to be some stupid Android phone gimmick. There's no way having that many pixels with a teeny optical path and a teeny sensor is useful. I'd only want above 100MP on a medium format sensor.
- kondro 5y ago
  And then what happens WHEN the NAS fails? And if the answer is to buy 2 NASes, what happens if something physically destroys the location. Fire & theft isn't all that uncommon over a lifetime. Or if you (probably a more likely) accidentally format or delete its contents.
  - wyager 5y ago
    It’s quite unlikely that a RAIDZ2-3 system fails in a way that you lose your data (short of catching on fire), but I also have offsite snapshot backups at rsync.net. 10mbps up is plenty to keep that synced, even if you write hundreds of GB per day.
jcoq 5y ago
A few people have raised this question in the thread already, but why are we compelled to store such unreasonable numbers of images? I'm definitely not immune and I find that the satisfaction I get from my photo collection is inversely proportional to the size of my collection. At this point, I'm just lugging around this huge mass of data "just in case". There's no way I will ever have time to sort my images. There are probably 2000 wedding images alone, let alone the tens of thousands thousands of random snapshots that may or may not be something I ever care to see again. At this point, I would almost prefer a smart solution opt-in solution similar to what Google photos provides for smart albums: "We found these images that would be good long-term. Save them indefinitely?"
- sliken 5y ago
  Sounds like you are not organized. Are all your photos tagged with location, long/lat, date, and subject matter? Can you in 5 seconds or less pull up every waterfall? Photo of a loved one? Every rainbow? Every trailhead? I reference mine quite often, usually for a purpose I didn't originally think of. The name of that cool restaurant in a city you visit rarely? How about how many double rainbows did you get last year in June? Pinning down dates from a previous pet. Or the funny pet photo with a squirrel in it? Where were you on March 2nd when your credit card got charged for $1202.20? When did you meet your new significant other 3 years ago or so? What trail head was it that you saw a bear? What is the serial number, VIN, or similar for just about anything valuable you own... or owned. How old where you when you won that race? The cost of a store a photo for life isn't much, seems silly to spend hours and hours trying to delete them, doubly so because mistakes will be made. Just tag them so you can find what you want. I was surprised how much tagging helped. Grand mothers poured over ever damn photo I tagged with the grand kids. Starting collections of wild animals we've seen by state. etc. When bored I find it rather fun to relive a vacation or hike. Even my kid seems to quite enjoy "visiting" places I've been. Sure 95-99% of my photos don't get viewed, ever. So? Why waste man hours on useless photos, just make sure you are organized enough to find the 1-5% that you do care about.
  - blooalien 5y ago
    I personally use https://www.digikam.org/ https://www.digikam.org/ to keep my own photo/image library organized and tagged. It's also easy to embed a bunch of that data directly into the image file metadata, too. Like you, I enjoy the ability to easily call up entire categories of images at will/need with a simple keyword/location search. Really handy when I'm doin' graphics stuff (Blender 3D, Inkscape, GIMP, whatever) and want a particular type of image for some purpose.
ilamont 5y ago
Isn’t this the business model that Dropbox, iCloud and Google Drive, etc. are based upon?
splittingTimes 5y ago
Maybe the data retention policy of OP is the real problem. Sound like a data hoarder/virtual messy to me. When ever I take pictures on a trip or vacation, i go through them at the end of each day, delete most of them and keep maybe 2 or 3 max, beautify them and the rest goes into the bin. No matter how long the trip, i try to only keep at max the 15 best Pictures. All filler no killer. That way I am comfortable to show others pictures of a trip without boring them and I also like to look at them from time to time as I know those are the best moments. At the end of each year we create a calendar with photo collages of 4 to 6 pictures per month. The calendar goes to relatives and we create a photo book from the print outs. That is what gets archived.
- arch-ninja 5y ago
  There's a ton of easy low-hanging fruit along these lines that could be improved. Good example: I have a python script that routinely scans my ~/downloads directory and quarantines anything more than 24 hours old. It then scans the quarantine directory and deletes anything more than a week old. If the data is important, I'll have 2 chances to save it someplace meaningful.
  - jnieminen 5y ago
    I use the downloads folder to store the interesting pieces. It is fascinating to jump back in time and see what you have came across.
    - Arrath 5y ago
      Same. For example my downloads folder has near every major release version of Starsector, Dwarf Fortress, and Factorio.
  - pfranz 5y ago
    This may not be directly related to your workflow, but in macOS there's a preference to delete items that have been in Trash after 30 days. If you're more proactive about putting stuff in the Trash (and less proactive about emptying it), that (plus backups) gives a similar behavior. My wife doesn't ever empty her Trash and previously I had written a script that did it.
fermentation 5y ago
What do folks typically use for storing media you don't access very often, but when you do access it you want it relatively quickly and at high transfer speeds? I like to backup my UHD blurays losslessly and stream them from my PC via plex (~100mbps). I'm thinking local storage via a NAS is probably the cheapest option here, right?
- geerlingguy 5y ago
  NAS, definitely. Even hard drives are a ton faster on a local network than most people's internet connection over a cloud service.
phaedrus 5y ago
I tried to back up a 60% full Ubuntu Linux RAID to a 4TB USB3 hard drive. I set the backup going and left it a few hours. I was shocked to come back and find the RAID 100% full and only a few GB copied. It took me a few minutes to figure out what happened. Maddeningly, the USB drive has a feature that it goes to sleep after 15 minutes or so, even if Ubuntu is actively using it to write files! Maybe the drivers on Windows do something that keep it awake, but on Linux it just goes to sleep in the middle of being used. Now here's the thing, the drive DOES realize it has more to do and wake back up. But this sleep-wake causes a USB disconnect and reconnect. Which causes Ubuntu to unmount and remount the drive. Now here's the problem, perhaps because the backup program is still making the the original mount point "busy", Ubuntu doesn't re-mount the media to the path. Instead it, gets mounted at "/media/path-1". Since Linux uses regular folders as mount points, the old mount point at "/media/path" becomes a valid folder on the local disk. So the backup program keeps going, but now it's filling up the local disk. I haven't found a solution for this problem that will allow me to complete a backup (or even complete a long-duration manual copy).
- nightfly 5y ago
  Most linux tools have an option that keeps you from crossing filesystem boundaries
  - Filligree 5y ago
    In this case he needs the opposite -- an option keeping it from not crossing filesystem boundaries.
  - ta988 5y ago
    I knew that for reads how do you do that for writes?
    - p1mrx 5y ago
      chattr +i /mnt/foo This makes the mount point immutable when nothing is mounted there.
throwaway290232 5y ago
I recently did a comparison of cloud storage options: - If you want "personal storage" (no programmatic access) the options are generally around $5 per TB/month - but that's a lot of drag-and-dropping and praying the connection doesn't die during transfer. - If you want object storage (think S3) the cheapest is $5 per TB/month, the average is $10 per TB/month, and the "high end" is $20 per TB/month, with extra costs for bandwidth ranging from $10 to $120 per TB/month. Honestly, just store less crap. Marie Condo your digital life. Does it not spark joy? Have you not looked at it in the last 6 months? Does it not serve a useful purpose, such as tax records? Ditch it. Even if you want to keep some pictures/video, either print a copy in original-quality, or compress/downscale it. I took a 3 minute video on my phone and it's 425MB. And it was still grainy! I used to download two-hour movies that were 725MB and looked like a DVD! errrrrrr... I mean, I heard about a guy that did that.
- SilverRed 5y ago
  >And it was still grainy! Because the grain is not from video compression but because the camera sensor wasn't receiving enough light so it boosted the ISO to compensate which causes grain. Also the fact that phones have to live encode video which is very non optimal while movies are shot uncompressed and then a powerful CPU can spend as long as it needs for a perfect compression.
  - deathanatos 5y ago
    The CPU in a real machine makes a huge difference: the last video I shot on my phone went from 3.9M → 1.6M, ~59% compressed. (With no noticable quality loss, but the original isn't winning any Oscars…)
    - SilverRed 5y ago
      And your phone is already pretty well off. Cheap dashcams spit out awful quality video with large file sizes since they don't have the benefit of expensive encoder chips.
edrxty 5y ago
There's a huge market here for something that professional photographers could use. I know several and they all tell each other to keep buying more external 1tb hard drives then back everything up to random cloud image hosts. Most of them end up with a bunch of zip lock bags laying around full of disks and thumb drives. I made the mistake of building a NAS for one at one point but it turns out that you need to be really diligent about documenting the setup as rebuilding a raid5 array when it inevitably fails can be very difficult if you don't remember what the actual configuration was. The crux of the issue is that they need massive storage capacity but whatever the solution is needs to be able to manage merging disjoint Adobe Lightroom catalogs while de-duplicating and without the possibility of data loss, all while needing basically no maintenance because these people tend not to be incredibly computer literate.
- notyourday 5y ago
  You are confusing the market of people who say that want something with market of people who will pay for it.
- ksec 5y ago
  This has been a problem for what? Nearly a decade now? I thought NAS vendor would one day figure it out. Nope. No one seems to give a damn. Not to mention some tools to help with all the duplication we had on different HDD or USB Sticks.
- hardwaresofton 5y ago
  Would you mind sharing why these photographers can't use Dropbox/Photoshelter/etc? Photoshelter looks to have a $50/month unlimited cloud storage plan[0], which seems pretty cheap if they're going to really take advantage of the "unlimited" aspect. For a professional photographer $50/month to not have to worry about carrying around this stuff seems like a deal (and less if they don't have that much to store). [0]: https://www.photoshelter.com/signup/subscriber https://www.photoshelter.com/signup/subscriber
  - pfranz 5y ago
    I haven't done serious photography in years, but my guess is that the market isn't quite as big as the parent suspects (the people who want it aren't willing the pay). There's a giant graveyard of push-button photo backup sites targeting photographers. Sadly, I would be suspicious of using any service (since it would mean uploading 100s of gigs of images for the kinds of people who would use it) since they shut down so often. I've never heard of Photoshelter, but it sounds like you upload high-resolution final images that are ready to view and print--it doesn't seem all that different from Smugmug or Flickr. Most photographers want to archive the source files and edits (project files). I see you can technically upload RAW files, but unless that integrates very well with your photo management software, I'm not sure if it's of much use. Lightroom databases can get relatively large since it includes caches and thumbnails. So you'd need a separate system for backing up those that ties it back to the RAW files. Dropbox might work, but I'm not sure how approachable "smart sync" is and their pricing tiers don't seem like a great fit. My guess is a well-loved open source project that wraps around b2 or AWS might be the best fit. But, honestly, a pile of hard drives will probably win out. You make your investment upfront and it's a simple enough mental-model. Unfortunately, they tend to die and with large data sizes photographers don't always go for redundancy.
snowwrestler 5y ago
10mbps up will upload about 30GB in 8 hours overnight while you sleep. Most people are not taking 30GB of new photos every day. Or using their Internet connection all day when they are awake either. So cloud storage is totally viable for most people with 10mbps upload speeds. Now, a person might have a big backlog of data. This guy has 7TB of photos. If you start uploading, you’ll get caught up eventually! If you want to get caught up faster, you just need to find a faster pipe and park your computer/HDs on it for a little while. For me, it was a friend with FiOS. For many professionals, it might be their office. The point is, once you’re through the backlog, 10mbps is probably enough to maintain. Again: for most people.
- boring_twenties 5y ago
  You might be forgetting about data caps. Comcast for instance has a 1.2TB/mo cap, graciously upped from 1.0TB last year.
  - blooalien 5y ago
    Indeed. And upload speed caps, as well. My 100 megabit downstream only allows 5 megabits upload speed for example.
    - boring_twenties 5y ago
      Indeed! Mine is advertised as 16Mbps which isn't great to begin with, but I can only burst to that for a few seconds after which it quickly drops to something like 3-4.
crossroadsguy 5y ago
> The best new phones come with the ability to shoot 108 megapixel photos, record 4K video with stereo sound, and pack the results into a terabyte of onboard storage. But what do you do when that storage fills up? A good starting point for me is to not keep everything I shoot. I have often noticed that every 50-100 photos I shoot I only want to keep 5-10. Rest are bad/weird shots, pics I’m not interested in etc. I never faced this when I used film.
newscracker 5y ago
If you want more reliable backups, Backblaze personal backup should never be on your list. The client takes anywhere between 1-8 hours to upload the index of files to the server after uploading the actual data for the backup, but it will say that the backup is complete before this happens and give a false assurance. So if your machine dies in that span (data uploaded but index not uploaded), then it’s as good as the backup not happening because the server has data blobs but no metadata, and cannot provide you a way to restore that. This is an intentionally designed behavior (for some other reasons) that adds risks that I belief nobody should take. See this blog post and comment from nearly four years ago for a better description. [1] And here’s Backblaze’s support page on the same, last updated a month ago (June 2021) indicating that this behavior remains. [2] Any backup solution that doesn’t, at a minimum, follow the 3-2-1 rule will cause more instances of regret in the future. If the data is of value, it needs better and constant care. [1]: https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-back-up/#what-backblaze-doesnt-back-up-update-2017-08-24 https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-bac... [2]: https://help.backblaze.com/hc/en-us/articles/217665498-Why-hasn-t-Backblaze-backed-up-my-new-files-yet- https://help.backblaze.com/hc/en-us/articles/217665498-Why-h...
nixpulvis 5y ago
The short answer to "can I replace physical storage with cloud storage?" is no.
deleted 5y ago
[deleted]
SlowBall 5y ago
I use borgbase (https://www.borgbase.com https://www.borgbase.com) for my online backups. It took a few days to make the initial upload, but now it's it only takes a couple of minute every hour. And you can select the pruning strategy you want. I haven't had to use it yet, but I check the files from time to time, and so far I'm satisfied with the service.
adreamingsoul 5y ago
I used Backblaze B2 as an offsite backup for RAW digital film. 1TB took about a month to completely backup, which is not ideal. B2 worked great and the pricing is indeed excellent, but the volume of raw data to backup over broadband is just not realistic. I think it would be better to physically mail a drive for archiving raw footage, or use tap drives.
gnicholas 5y ago
Lots of people talking about photo retention policies here, but for me the issue is videos. We have little kids, and that means lots of cute videos. It's harder to scan through them and see what's useful versus what isn't. I try to remember to delete videos that didn't turn out when I take them, but that doesn't help with the videos that I shoot that are 2 mins long but only have 20 seconds of something worth keeping. So far I've been able to get by with retail HDDs; I have a 5TB drive at home and another that we keep off-site and refresh a couple times a year. This seems to be a sustainable strategy for me, as affordable (~$100) portable HDDs are growing at a rate that is faster than my storage needs. I don't know if it would be cheaper to pay for iCloud, but for whatever reason I feel safer having two HDDs with my media than trusting Apple's (or anyone else's) cloud.
- ValentineC 5y ago
  > I don't know if it would be cheaper to pay for iCloud, but for whatever reason I feel safer having two HDDs with my media than trusting Apple's (or anyone else's) cloud. I'd suggest paying for iCloud just for the ability to automagically backup your iPhone(s). You can exclude Photos from being backed up if you prefer. I use iCloud Photo Library as well on a larger plan, but the backup feature alone is definitely well worth $0.99.
- fouc 5y ago
  Would be nice if AI could suggest what to cut, make it really quick to clean up a video
  - gnicholas 5y ago
    I can't imagine AI would be able to tell what constitutes the cute part of a video of a little kid, but it would sure be nice if I could just tell Siri to "delete all but the last 20 seconds of this video". It's not that manually cropping is that many steps, it's that on an iPhone, it's hard to get the crop tool to work reliably (or my fingers are just really fat).
foysaluix 5y ago
This storage talk abut hybrid drives sounds a lot like 2010. This isnt new and SSDs are really not much more expensive than your regular HDD and they offer much more benefits. Its just that Apple charges you for 1 drive the price of at least 4 same drives if you were buying it on your own.
- unionpivo 5y ago
  This is for long term storage. SSD has 0 benefits over spinning rust. and is still more expensive. I can buy decent spinning rust 10TB disk anywhere, if i want that in ssd's, id need 5 drives, or start hinting for enterprise grade equipment. So for me for raw storage (if you need more than a couple of TB's) spinningg rust is still the winner.
WalterBright 5y ago
I remember when my life's professional work fit on a CD. It was a strange feeling.
salmonlogs 5y ago
I previously had a 4-bay Synology and loved the UI but couldn't deal with the price jump to an 8-bay. I discovered xpenology.org which allows you to bootload Synology software on x86 hardware. I turned an old desktop PC into an 8-bay storage machine with a PCIe SATA card and now have all the flexibility I want, without the price penalty for Synology hardware. The best place to find hard drives is https://diskprices.com/ https://diskprices.com/ which scrapes Amazon and a few other sites to show TB per £/$. It's almost always cheaper to buy a retail packaged external disk and extract the hard drive. Note: Some hard drives once shucked from external cases won't turn on when attached to regular SATA power adapters due to mismatches in power connector specification. Easily solved by using a SATA power to Molex adapter, and connecting that to a Molex to SATA power adapter (SATA->Molex->SATA) as Molex does not support the 3.3V pin which is the issue.
nazgulnarsil 5y ago
Hoarding is a mental illness
- dragonwriter 5y ago
  > Hoarding is a mental illness No, hoarding is a behavior which may or may not, in any particular case, be a symptom of one or another mental illness.
devops000 5y ago
Never heard about cloud storage?
ricardobeat 5y ago
I can't fathom how you get to 7TB of photos without being a photographer, or recording video all the time. Granted I haven't been too prolific in the last few years, but my entire collection from 3 DSLRs + iPhones going back to ~2005 totals 300GB. All backed up in B2 + google cloud.
- ValentineC 5y ago
  > Granted I haven't been too prolific in the last few years, but my entire collection from 3 DSLRs + iPhones going back to ~2005 totals 300GB. You don't keep the RAW files from your DSLRs? That'll bring the total size up pretty quick.
nazrulmum10 5y ago
I want to replace those external drives with cloud storage. Is that possible?
hadlock 5y ago
I ended up buying a 4 disk synology NAS unit/appliance for about $400 a couple of years ago. It's been remarkably reliable, and has a built-in feature to backup to Glacier, although there's third party tools for most any online data backup system you can think of. Prior to that I had this monsterous, 4U server that I had to maintain, update, debug, not to mention build and move from house to house, for 8 years. At the time it seemed like a good move, but over time, as my time got shorter to work on personal products, I started to look for other ways to solve the storage problem. I much prefer the Synology NAS to the 4U custom built system. It's about as close to a toaster-style appliance as I can think of. If it died tomorrow, I'd buy another one (then restore from my Glacier backups).
svilen_dobrev 5y ago
i am using this jpeg recompressor: https://github.com/danielgtaylor/jpeg-archive https://github.com/danielgtaylor/jpeg-archive Tried a lot of combinations of quality, and settled for -q medium --max 75% --accurate Anything higher makes no sense/difference, even if you print it at A3. And my images are 8-18-24Mpix. i'm not a pro-foto but somewhat of an old pre-print school - Your level of pickiness may vary :/ Effectively it is like 4-8 times down - Total went from 150Gb into 30Gb - that's about 40,000 images. Which still doesn't solve the OP's or anyone's problem of too-easy-producible digital "assets".. but is more manageable.
- ValentineC 5y ago
  I guess it depends on how much one values "archiving", but WebP has a lossless option [1] that might be a good alternative to the above. [1] https://imagemagick.org/script/webp.php https://imagemagick.org/script/webp.php

127 points by vikrum 5y ago

I am faced with this issue of dealing with hundreds of GB of family video and pictures everytime I upgrade my wife’s phone. The newer phones always have a higher resolution camera so file sizes keep increasing. I would love a simple easy to use tape drive that just works. Something I could backup files to for 20-25 years.

ksec 5y ago

I think there is another point worth mentioning. The Data we own or create are increasing. Taking more photos than ever, higher resolution, Video in 4K or 60 frames. Friends sending their media over. So not only does the usage increases, the actual size of those unit increases as well. And yet Hardrive price / GB hasn't fallen much at all. From 2013 to 2021 now, when it was $0.035 to $0.04 /GB. Mostly because much HDD patter density has stalled.

Gravityloss 5y ago

How much of the actual information in the high resolution photos is just noise? So we buy sensors with more megapixels that just save more noise. Then we need to buy more storage capacity, more bandwidth, more processing, more battery etc. Since the cameras have been hitting closer and closer to physical limits but something still needs to be sold... It's like people buying clothes from the mall and taking them straight to a storage building. They don't have enough space at home. And they won't have time to wear so many clothes anyway.

orasis 5y ago

I’m still happy with my setup of Drobo + 2 rotating time machine backup HDs. I’ve probably been using it close to 10 years now.

PaulHoule 5y ago

I think $6 a month for "unlimited data" is going to end badly. The AWS or Azure price is high, but it's a scalable price, a real price.

nateroling 5y ago
Yup, I'm currently experimenting with using an AWS bucket alongside [CloudMounter](https://cloudmounter.net https://cloudmounter.net) to create an Archive disk for things I don't really need day-to-day, but would like to hang onto. I'm currently not backing that data up, though. It's not critical, and the chances of AWS losing it are low enough that I'm not too worried. The biggest risk would be myself accidentally deleting it. One thing I have decided for sure, though, is that archiving lots of small files is awful. Much better to wrap them up in a tarfile, uncompressed.
- terpimost 5y ago
  Amazon Drive is fine. No need to have a mirror of all that data on my computer. With that I could have selected sync or no sync at all. + unlimited photo storage. Amazon Photo and Amazon Drive it’s like a single product.
  - ValentineC 5y ago
    I've used Amazon Drive back when they offered their unlimited product, and they've lost/corrupted some of my files before. I personally wouldn't recommend Amazon Drive as a backup destination to anyone.
wgjordan 5y ago
> I think $6 a month for "unlimited data" is going to end badly. Backblaze has been offering their 'unlimited data' for over a decade [1], and it hasn't ended badly for them yet. It's fully sustainable because they only lose money on a few customers (like that one customer storing 430TB for $6/month [2]), while most customers use much less storage than that, so the service remains profitable overall. The soft limit of its 'unlimited' comes from it being a personal-backup mirroring service and not a fully-external cloud storage, so the service only fits certain use cases. [1] https://www.backblaze.com/blog/all-in-on-unlimited-backup/ https://www.backblaze.com/blog/all-in-on-unlimited-backup/ [2] https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backblaze_cloud_team_managing_750/ejlbsqo https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backb...

bethecloud 5y ago

I recommend looking into decentralized cloud storage solutions, which are really the next generation of AWS S3-like services. Check out storj.io

acdha 5y ago
Decentralization adds a good deal risk and operational overhead but it doesn't really change the core problem that people are generating more data than many want to pay for. If you want to store a non-trivial amount of data, someone needs to get paid to maintain storage pools and validate multiple copies.
- sliken 5y ago
  I've always wondered why nobody does p2p backups. Take whatever the backups are, say 1TB, split it into 12 pieces, add as many pieces as the user wants (say 6 on average), then you can recover with any 12. When you add another 1TB, add another 18 peers. Monitor your peers, only trust the ones with a track record of successful challenges, and of course let you white list peers you trust (like friends and family). Of course the perfect challenge is just a restore, but some checksome of a range of a blob could be useful as well, and consume less bandwidth. Drop peers that are unreliable, or ask for too much bandwidth for their restores. Encrypt the files before adding reed solomon, use a unique encryption key for each peer that stores 1/Nth of your backup set. That way you can "pay" for your storage by just adding that much more local disk space to trade with your peers.
  - wmf 5y ago
    It's been tried plenty of times. P2P software is really complex but the use case is people who aren't willing to pay, so maintaining the software isn't sustainable.
  - acdha 5y ago
    A couple problems off the top of my head: 1. Everything is harder to work with: you have to deal with less reliable networks and storage, computers which aren't on all of the time, very slow uplinks, etc. Since these aren't professionally managed systems, too, you have less visibility — did that node just drop offline because the hard drive failed, losing everything, Comcast is having a bad day, the owner just bought a new one and wiped the old one without unenrolling it, or because it rebooted and is almost back up? 2. People are selfish: I don't want Netflix getting slow because you decided to retrieve your data, I'll complain if I hit a storage limit on my computer due to your stored backup data, etc. This forces you do deal with things like traffic shaping and storage rebalancing more aggressively and those are hard problems to get a popular balance on. Consider, for example, what happens when someone uses your service and it goes well but then they run out of space and need to clear some up in a hurry (_especially_ if they put on their cowboy hat and just delete a bunch of large files because they know they aren't the only ones with a copy). 3. The solutions to the previous problems make the cost problem worse: storing more copies can avoid some of the problems but then you need to figure out how to get the network to support, say, 5 copies instead of 2-3. 4. Consider what happens the first time the police bust someone for a major crime and their data is backed up on your computer. Not many people are enthusiastic about going into court to prove the negative assertion that they didn't have a decryption key. Only peering with trusted systems avoids some of these issues but not all and the _big_ problem is that the upper bound for how much this service is worth is basically the cost of iCloud/Dropbox/S3/Backblaze/etc. The savings you can get between the fixed operational costs and what those services charge is probably not enough to support development.

deleted 5y ago

[deleted]

TheDudeMan 5y ago

If you don't want to pay for the cloud, then there is only one sane option: You keep upgrading your local storage so that you never have more than a few devices. Every six months, buy a new giant disk and decommission as many smaller disks as you can.

arthurcolle 5y ago
This sounds extremely expensive
- amarshall 5y ago
  Cloud storage is really expensive, though. Putting 4 TB in S3 costs US$94.21/month before any transfer costs. A 4 TB CMR HDD is ~$140. That doesn’t include power costs, but over a year you get $990 to spend on that and other things from the cost difference vs. S3 (plus you can sell your old drives when you’re done with them). (I use B2 for cloud storage backup since it’s a lot cheaper than S3 at US$20 for 4 TB, but that still is ~$105/year more than local)
  - dragontamer 5y ago
    > 4 TB CMR HDD is ~$140 And that's with bad pricing! Last year, 4TB was $120 and 6TB was ~$150. Building out a NAS for $1000 ($600 in 4x hard drives, $400 for other components) is very reasonable. Last year that was 4x6TB == 12TB storage + 12TB redundancy, but this year prices are worse so you "only" get 8TB + 8TB redundancy. $400 can afford a Synology or various NAS devices. It can also afford a new desktop that you can install FreeNAS or whatever onto. ----------- Eventually, when the 8TB is not enough, just buy a new HBA card and shove 4x more hard drives in there for a 2nd storage on the same NAS. Maybe 8TB x 4 == 16TB usable + 16TB redundancy. Except no need to copy everything over, just keep the old 8TB cluster working, and just start writing to the new 16TB storage.
    - rhacker 5y ago
      This is the true answer to this entire thread.

the_third_wave 5y ago

Assuming you have family living elsewhere but reachable through a fast internet connection you can do what I do by making a deal with them: they hang your backup box off their net and you will do the same for them. The backup box is some piece of computing equipment with storage media attached, e.g. a single board computer hooked up to a JBOD tower. Depending on the level of trust between you and your family you can use the thing as an rsnapshot target - giving you fine-grained direct access to time-based snapshots (I use 4-hour intervals for my rsnapshot targets which are located on-premises in different buildings spread over the farm) or as a repository of encrypted tarballs, or something in between. Allow the drives to spin down to save power, they'll be active only a fraction of the day. The average power consumption of the whole contraption does not need to exceed 10-15W making electricity costs negligible. You can have as much storage capacity as you want/can afford at the moment, keep for for as long as you want or until it breaks without having to pay any fees (other than hosting their contraption on your network - possibly including building it for them if they're not that computer-savvy).

deleted 5y ago
[deleted]
yjftsjthsd-h 5y ago
> Depending on the level of trust between you and your family you can use the thing as an rsnapshot target - giving you fine-grained direct access to time-based snapshots (I use 4-hour intervals for my rsnapshot targets which are located on-premises in different buildings spread over the farm) or as a repository of encrypted tarballs, or something in between. These days, openzfs native encryption is the best of all worlds, I think.
- the_third_wave 5y ago
  In that case, use that. ZFS seems to be a somewhat touchy subject with some people using it as widely as possible while others - myself included - prefer more modular storage systems where the tasks of volume manager, striping/slicing/raid management, encryption layers and file systems are performed by discrete software layers. Both systems work, both have their pros and cons, in the end it comes down what you value the most.

lordnacho 5y ago

What are the requirements? Large capacity, redundancy, reasonable access speed, always on but maintenance downtime of even a few hours a year tolerable? Easy but your data isn't yours: sync your data to GDrive or Apple or whatever, and sync a NAS to that. A little harder but still doable: get a Hetzner and set that up as your storage, set up your own access, sync to local NAS. A Hetz is also really useful for running a load of other services, so for 50 bucks or so it seems pretty reasonable. You could just buy a huge disk and run a server in your house, but it gets annoying in various ways. Kids unplug the power, it creates heat, maybe noise, multiple disks end up needing management, that kind of thing.

darklighter3 5y ago

Amazon Snowball? https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/

wmf 5y ago

The solution is either NAS or cloud archive (not backup) such as SmugMug or ExpanDrive.

JohnJamesRambo 5y ago

Almost everything I create doesn’t really need to be saved and I could never find it if I needed to. I just let Google Photos keep them. It’s good enough.

ur-whale 5y ago

The real question being, of course, given that your life is finite ... how often will you actually look at any of these pics. The more pix there are, the less likely it is you'll ever even open any of these.

tablespoon 5y ago
> The real question being, of course, given that your life is finite ... how often will you actually look at any of these pics. > The more pix there are, the less likely it is you'll ever even open any of these. Well, there are also future generations to think about, but their interest will fall off too (until you reach the genealogical profile level, which maxes out at a few portrait-type pictures of any regular individual). It's pretty essential to aggressively curate and organize data like this.
- sliken 5y ago
  Dunno, tagging seems worth it, deleting not so much. Storage is getting cheaper ever year. I've looked through a many albums from parents, grand parents, and great grand parents. I hope that my pictures make it that far, but seems like the world needs a "free" (except for storage/bandwidth) photo backup service to have any chance of surviving multiple generations.
  - tablespoon 5y ago
    > Dunno, tagging seems worth it, deleting not so much. Storage is getting cheaper ever year. The way I think about it is that you need relatively small "SAVE THIS" drive/disc. Sure, storage is getting cheaper, but you don't want to burden someone with some massive storage array, and you can't count on whatever software you used to tag to still work in someone else's hands.
    - sliken 5y ago
      Dunno, can you guess what will be important to the next generation? 2 generations? 3? You might play a small part of important events, people, etc that will not be realized later. Even silly things like the first robin of the season, or a picture at the pool on the hottest day of the year.

D13Fd 5y ago

I back up about 15tb of photos with Arq to Wasabi cloud storage. It has been running with zero maintenance (other than occasional partial restores) since late 2018. I transitioned off of AWS cloud storage when they raised the prices. I’m not sure if Wasabi is still the cheapest and fastest, but they have been great to deal with. And Arq is an excellent set-it-and-forget-it encrypted cloud backup app. I also run a server with a 16tb RAID 1 array and a set of local backup drives. Sadly it is almost full, and the volume of data makes it a hassle to upgrade (not to mention the cost). I’ve found standard 1Gig Ethernet to be just barely fast enough for editing photos over the local network. However, for my own sanity, I usually do the initial editing on a local drive before sending the files to the server (and from there to the cloud backup).

wmf 5y ago
Note that he's looking for primary storage not backup.
chillage 5y ago
How do you deal with merging Lightroom catalog files if they are all indexed remote? Sounds like a tricky workflow

wyager 5y ago

Spend a few hundred to a few thousand buck and build a chunky NAS. Get a bunch of 12TB SSDs and put them in a RAIDZn. Also, what shitty phone takes 108MP photos? That's guaranteed to be some stupid Android phone gimmick. There's no way having that many pixels with a teeny optical path and a teeny sensor is useful. I'd only want above 100MP on a medium format sensor.

kondro 5y ago
And then what happens WHEN the NAS fails? And if the answer is to buy 2 NASes, what happens if something physically destroys the location. Fire & theft isn't all that uncommon over a lifetime. Or if you (probably a more likely) accidentally format or delete its contents.
- wyager 5y ago
  It’s quite unlikely that a RAIDZ2-3 system fails in a way that you lose your data (short of catching on fire), but I also have offsite snapshot backups at rsync.net. 10mbps up is plenty to keep that synced, even if you write hundreds of GB per day.

jcoq 5y ago

A few people have raised this question in the thread already, but why are we compelled to store such unreasonable numbers of images? I'm definitely not immune and I find that the satisfaction I get from my photo collection is inversely proportional to the size of my collection. At this point, I'm just lugging around this huge mass of data "just in case". There's no way I will ever have time to sort my images. There are probably 2000 wedding images alone, let alone the tens of thousands thousands of random snapshots that may or may not be something I ever care to see again. At this point, I would almost prefer a smart solution opt-in solution similar to what Google photos provides for smart albums: "We found these images that would be good long-term. Save them indefinitely?"

sliken 5y ago
Sounds like you are not organized. Are all your photos tagged with location, long/lat, date, and subject matter? Can you in 5 seconds or less pull up every waterfall? Photo of a loved one? Every rainbow? Every trailhead? I reference mine quite often, usually for a purpose I didn't originally think of. The name of that cool restaurant in a city you visit rarely? How about how many double rainbows did you get last year in June? Pinning down dates from a previous pet. Or the funny pet photo with a squirrel in it? Where were you on March 2nd when your credit card got charged for $1202.20? When did you meet your new significant other 3 years ago or so? What trail head was it that you saw a bear? What is the serial number, VIN, or similar for just about anything valuable you own... or owned. How old where you when you won that race? The cost of a store a photo for life isn't much, seems silly to spend hours and hours trying to delete them, doubly so because mistakes will be made. Just tag them so you can find what you want. I was surprised how much tagging helped. Grand mothers poured over ever damn photo I tagged with the grand kids. Starting collections of wild animals we've seen by state. etc. When bored I find it rather fun to relive a vacation or hike. Even my kid seems to quite enjoy "visiting" places I've been. Sure 95-99% of my photos don't get viewed, ever. So? Why waste man hours on useless photos, just make sure you are organized enough to find the 1-5% that you do care about.
- blooalien 5y ago
  I personally use https://www.digikam.org/ https://www.digikam.org/ to keep my own photo/image library organized and tagged. It's also easy to embed a bunch of that data directly into the image file metadata, too. Like you, I enjoy the ability to easily call up entire categories of images at will/need with a simple keyword/location search. Really handy when I'm doin' graphics stuff (Blender 3D, Inkscape, GIMP, whatever) and want a particular type of image for some purpose.

ilamont 5y ago

Isn’t this the business model that Dropbox, iCloud and Google Drive, etc. are based upon?

splittingTimes 5y ago

Maybe the data retention policy of OP is the real problem. Sound like a data hoarder/virtual messy to me. When ever I take pictures on a trip or vacation, i go through them at the end of each day, delete most of them and keep maybe 2 or 3 max, beautify them and the rest goes into the bin. No matter how long the trip, i try to only keep at max the 15 best Pictures. All filler no killer. That way I am comfortable to show others pictures of a trip without boring them and I also like to look at them from time to time as I know those are the best moments. At the end of each year we create a calendar with photo collages of 4 to 6 pictures per month. The calendar goes to relatives and we create a photo book from the print outs. That is what gets archived.

arch-ninja 5y ago
There's a ton of easy low-hanging fruit along these lines that could be improved. Good example: I have a python script that routinely scans my ~/downloads directory and quarantines anything more than 24 hours old. It then scans the quarantine directory and deletes anything more than a week old. If the data is important, I'll have 2 chances to save it someplace meaningful.
- jnieminen 5y ago
  I use the downloads folder to store the interesting pieces. It is fascinating to jump back in time and see what you have came across.
  - Arrath 5y ago
    Same. For example my downloads folder has near every major release version of Starsector, Dwarf Fortress, and Factorio.
- pfranz 5y ago
  This may not be directly related to your workflow, but in macOS there's a preference to delete items that have been in Trash after 30 days. If you're more proactive about putting stuff in the Trash (and less proactive about emptying it), that (plus backups) gives a similar behavior. My wife doesn't ever empty her Trash and previously I had written a script that did it.

fermentation 5y ago

What do folks typically use for storing media you don't access very often, but when you do access it you want it relatively quickly and at high transfer speeds? I like to backup my UHD blurays losslessly and stream them from my PC via plex (~100mbps). I'm thinking local storage via a NAS is probably the cheapest option here, right?

geerlingguy 5y ago
NAS, definitely. Even hard drives are a ton faster on a local network than most people's internet connection over a cloud service.

phaedrus 5y ago

I tried to back up a 60% full Ubuntu Linux RAID to a 4TB USB3 hard drive. I set the backup going and left it a few hours. I was shocked to come back and find the RAID 100% full and only a few GB copied. It took me a few minutes to figure out what happened. Maddeningly, the USB drive has a feature that it goes to sleep after 15 minutes or so, even if Ubuntu is actively using it to write files! Maybe the drivers on Windows do something that keep it awake, but on Linux it just goes to sleep in the middle of being used. Now here's the thing, the drive DOES realize it has more to do and wake back up. But this sleep-wake causes a USB disconnect and reconnect. Which causes Ubuntu to unmount and remount the drive. Now here's the problem, perhaps because the backup program is still making the the original mount point "busy", Ubuntu doesn't re-mount the media to the path. Instead it, gets mounted at "/media/path-1". Since Linux uses regular folders as mount points, the old mount point at "/media/path" becomes a valid folder on the local disk. So the backup program keeps going, but now it's filling up the local disk. I haven't found a solution for this problem that will allow me to complete a backup (or even complete a long-duration manual copy).

nightfly 5y ago
Most linux tools have an option that keeps you from crossing filesystem boundaries
- Filligree 5y ago
  In this case he needs the opposite -- an option keeping it from not crossing filesystem boundaries.
- ta988 5y ago
  I knew that for reads how do you do that for writes?
  - p1mrx 5y ago
    chattr +i /mnt/foo This makes the mount point immutable when nothing is mounted there.

throwaway290232 5y ago

I recently did a comparison of cloud storage options: - If you want "personal storage" (no programmatic access) the options are generally around $5 per TB/month - but that's a lot of drag-and-dropping and praying the connection doesn't die during transfer. - If you want object storage (think S3) the cheapest is $5 per TB/month, the average is $10 per TB/month, and the "high end" is $20 per TB/month, with extra costs for bandwidth ranging from $10 to $120 per TB/month. Honestly, just store less crap. Marie Condo your digital life. Does it not spark joy? Have you not looked at it in the last 6 months? Does it not serve a useful purpose, such as tax records? Ditch it. Even if you want to keep some pictures/video, either print a copy in original-quality, or compress/downscale it. I took a 3 minute video on my phone and it's 425MB. And it was still grainy! I used to download two-hour movies that were 725MB and looked like a DVD! errrrrrr... I mean, I heard about a guy that did that.

SilverRed 5y ago
>And it was still grainy! Because the grain is not from video compression but because the camera sensor wasn't receiving enough light so it boosted the ISO to compensate which causes grain. Also the fact that phones have to live encode video which is very non optimal while movies are shot uncompressed and then a powerful CPU can spend as long as it needs for a perfect compression.
- deathanatos 5y ago
  The CPU in a real machine makes a huge difference: the last video I shot on my phone went from 3.9M → 1.6M, ~59% compressed. (With no noticable quality loss, but the original isn't winning any Oscars…)
  - SilverRed 5y ago
    And your phone is already pretty well off. Cheap dashcams spit out awful quality video with large file sizes since they don't have the benefit of expensive encoder chips.

edrxty 5y ago

There's a huge market here for something that professional photographers could use. I know several and they all tell each other to keep buying more external 1tb hard drives then back everything up to random cloud image hosts. Most of them end up with a bunch of zip lock bags laying around full of disks and thumb drives. I made the mistake of building a NAS for one at one point but it turns out that you need to be really diligent about documenting the setup as rebuilding a raid5 array when it inevitably fails can be very difficult if you don't remember what the actual configuration was. The crux of the issue is that they need massive storage capacity but whatever the solution is needs to be able to manage merging disjoint Adobe Lightroom catalogs while de-duplicating and without the possibility of data loss, all while needing basically no maintenance because these people tend not to be incredibly computer literate.

notyourday 5y ago
You are confusing the market of people who say that want something with market of people who will pay for it.
ksec 5y ago
This has been a problem for what? Nearly a decade now? I thought NAS vendor would one day figure it out. Nope. No one seems to give a damn. Not to mention some tools to help with all the duplication we had on different HDD or USB Sticks.
hardwaresofton 5y ago
Would you mind sharing why these photographers can't use Dropbox/Photoshelter/etc? Photoshelter looks to have a $50/month unlimited cloud storage plan[0], which seems pretty cheap if they're going to really take advantage of the "unlimited" aspect. For a professional photographer $50/month to not have to worry about carrying around this stuff seems like a deal (and less if they don't have that much to store). [0]: https://www.photoshelter.com/signup/subscriber https://www.photoshelter.com/signup/subscriber
- pfranz 5y ago
  I haven't done serious photography in years, but my guess is that the market isn't quite as big as the parent suspects (the people who want it aren't willing the pay). There's a giant graveyard of push-button photo backup sites targeting photographers. Sadly, I would be suspicious of using any service (since it would mean uploading 100s of gigs of images for the kinds of people who would use it) since they shut down so often. I've never heard of Photoshelter, but it sounds like you upload high-resolution final images that are ready to view and print--it doesn't seem all that different from Smugmug or Flickr. Most photographers want to archive the source files and edits (project files). I see you can technically upload RAW files, but unless that integrates very well with your photo management software, I'm not sure if it's of much use. Lightroom databases can get relatively large since it includes caches and thumbnails. So you'd need a separate system for backing up those that ties it back to the RAW files. Dropbox might work, but I'm not sure how approachable "smart sync" is and their pricing tiers don't seem like a great fit. My guess is a well-loved open source project that wraps around b2 or AWS might be the best fit. But, honestly, a pile of hard drives will probably win out. You make your investment upfront and it's a simple enough mental-model. Unfortunately, they tend to die and with large data sizes photographers don't always go for redundancy.

snowwrestler 5y ago

10mbps up will upload about 30GB in 8 hours overnight while you sleep. Most people are not taking 30GB of new photos every day. Or using their Internet connection all day when they are awake either. So cloud storage is totally viable for most people with 10mbps upload speeds. Now, a person might have a big backlog of data. This guy has 7TB of photos. If you start uploading, you’ll get caught up eventually! If you want to get caught up faster, you just need to find a faster pipe and park your computer/HDs on it for a little while. For me, it was a friend with FiOS. For many professionals, it might be their office. The point is, once you’re through the backlog, 10mbps is probably enough to maintain. Again: for most people.

boring_twenties 5y ago
You might be forgetting about data caps. Comcast for instance has a 1.2TB/mo cap, graciously upped from 1.0TB last year.
- blooalien 5y ago
  Indeed. And upload speed caps, as well. My 100 megabit downstream only allows 5 megabits upload speed for example.
  - boring_twenties 5y ago
    Indeed! Mine is advertised as 16Mbps which isn't great to begin with, but I can only burst to that for a few seconds after which it quickly drops to something like 3-4.

crossroadsguy 5y ago

> The best new phones come with the ability to shoot 108 megapixel photos, record 4K video with stereo sound, and pack the results into a terabyte of onboard storage. But what do you do when that storage fills up? A good starting point for me is to not keep everything I shoot. I have often noticed that every 50-100 photos I shoot I only want to keep 5-10. Rest are bad/weird shots, pics I’m not interested in etc. I never faced this when I used film.

newscracker 5y ago

If you want more reliable backups, Backblaze personal backup should never be on your list. The client takes anywhere between 1-8 hours to upload the index of files to the server after uploading the actual data for the backup, but it will say that the backup is complete before this happens and give a false assurance. So if your machine dies in that span (data uploaded but index not uploaded), then it’s as good as the backup not happening because the server has data blobs but no metadata, and cannot provide you a way to restore that. This is an intentionally designed behavior (for some other reasons) that adds risks that I belief nobody should take. See this blog post and comment from nearly four years ago for a better description. [1] And here’s Backblaze’s support page on the same, last updated a month ago (June 2021) indicating that this behavior remains. [2] Any backup solution that doesn’t, at a minimum, follow the 3-2-1 rule will cause more instances of regret in the future. If the data is of value, it needs better and constant care. [1]: https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-back-up/#what-backblaze-doesnt-back-up-update-2017-08-24 https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-bac... [2]: https://help.backblaze.com/hc/en-us/articles/217665498-Why-hasn-t-Backblaze-backed-up-my-new-files-yet- https://help.backblaze.com/hc/en-us/articles/217665498-Why-h...

nixpulvis 5y ago

The short answer to "can I replace physical storage with cloud storage?" is no.

deleted 5y ago

[deleted]

SlowBall 5y ago

I use borgbase (https://www.borgbase.com https://www.borgbase.com) for my online backups. It took a few days to make the initial upload, but now it's it only takes a couple of minute every hour. And you can select the pruning strategy you want. I haven't had to use it yet, but I check the files from time to time, and so far I'm satisfied with the service.

adreamingsoul 5y ago

I used Backblaze B2 as an offsite backup for RAW digital film. 1TB took about a month to completely backup, which is not ideal. B2 worked great and the pricing is indeed excellent, but the volume of raw data to backup over broadband is just not realistic. I think it would be better to physically mail a drive for archiving raw footage, or use tap drives.

gnicholas 5y ago

Lots of people talking about photo retention policies here, but for me the issue is videos. We have little kids, and that means lots of cute videos. It's harder to scan through them and see what's useful versus what isn't. I try to remember to delete videos that didn't turn out when I take them, but that doesn't help with the videos that I shoot that are 2 mins long but only have 20 seconds of something worth keeping. So far I've been able to get by with retail HDDs; I have a 5TB drive at home and another that we keep off-site and refresh a couple times a year. This seems to be a sustainable strategy for me, as affordable (~$100) portable HDDs are growing at a rate that is faster than my storage needs. I don't know if it would be cheaper to pay for iCloud, but for whatever reason I feel safer having two HDDs with my media than trusting Apple's (or anyone else's) cloud.

ValentineC 5y ago
> I don't know if it would be cheaper to pay for iCloud, but for whatever reason I feel safer having two HDDs with my media than trusting Apple's (or anyone else's) cloud. I'd suggest paying for iCloud just for the ability to automagically backup your iPhone(s). You can exclude Photos from being backed up if you prefer. I use iCloud Photo Library as well on a larger plan, but the backup feature alone is definitely well worth $0.99.
fouc 5y ago
Would be nice if AI could suggest what to cut, make it really quick to clean up a video
- gnicholas 5y ago
  I can't imagine AI would be able to tell what constitutes the cute part of a video of a little kid, but it would sure be nice if I could just tell Siri to "delete all but the last 20 seconds of this video". It's not that manually cropping is that many steps, it's that on an iPhone, it's hard to get the crop tool to work reliably (or my fingers are just really fat).

foysaluix 5y ago

This storage talk abut hybrid drives sounds a lot like 2010. This isnt new and SSDs are really not much more expensive than your regular HDD and they offer much more benefits. Its just that Apple charges you for 1 drive the price of at least 4 same drives if you were buying it on your own.

unionpivo 5y ago
This is for long term storage. SSD has 0 benefits over spinning rust. and is still more expensive. I can buy decent spinning rust 10TB disk anywhere, if i want that in ssd's, id need 5 drives, or start hinting for enterprise grade equipment. So for me for raw storage (if you need more than a couple of TB's) spinningg rust is still the winner.

WalterBright 5y ago

I remember when my life's professional work fit on a CD. It was a strange feeling.

salmonlogs 5y ago

I previously had a 4-bay Synology and loved the UI but couldn't deal with the price jump to an 8-bay. I discovered xpenology.org which allows you to bootload Synology software on x86 hardware. I turned an old desktop PC into an 8-bay storage machine with a PCIe SATA card and now have all the flexibility I want, without the price penalty for Synology hardware. The best place to find hard drives is https://diskprices.com/ https://diskprices.com/ which scrapes Amazon and a few other sites to show TB per £/$. It's almost always cheaper to buy a retail packaged external disk and extract the hard drive. Note: Some hard drives once shucked from external cases won't turn on when attached to regular SATA power adapters due to mismatches in power connector specification. Easily solved by using a SATA power to Molex adapter, and connecting that to a Molex to SATA power adapter (SATA->Molex->SATA) as Molex does not support the 3.3V pin which is the issue.

nazgulnarsil 5y ago

Hoarding is a mental illness

dragonwriter 5y ago
> Hoarding is a mental illness No, hoarding is a behavior which may or may not, in any particular case, be a symptom of one or another mental illness.

devops000 5y ago

Never heard about cloud storage?

ricardobeat 5y ago

I can't fathom how you get to 7TB of photos without being a photographer, or recording video all the time. Granted I haven't been too prolific in the last few years, but my entire collection from 3 DSLRs + iPhones going back to ~2005 totals 300GB. All backed up in B2 + google cloud.

ValentineC 5y ago
> Granted I haven't been too prolific in the last few years, but my entire collection from 3 DSLRs + iPhones going back to ~2005 totals 300GB. You don't keep the RAW files from your DSLRs? That'll bring the total size up pretty quick.

nazrulmum10 5y ago

I want to replace those external drives with cloud storage. Is that possible?

hadlock 5y ago

I ended up buying a 4 disk synology NAS unit/appliance for about $400 a couple of years ago. It's been remarkably reliable, and has a built-in feature to backup to Glacier, although there's third party tools for most any online data backup system you can think of. Prior to that I had this monsterous, 4U server that I had to maintain, update, debug, not to mention build and move from house to house, for 8 years. At the time it seemed like a good move, but over time, as my time got shorter to work on personal products, I started to look for other ways to solve the storage problem. I much prefer the Synology NAS to the 4U custom built system. It's about as close to a toaster-style appliance as I can think of. If it died tomorrow, I'd buy another one (then restore from my Glacier backups).

svilen_dobrev 5y ago

i am using this jpeg recompressor: https://github.com/danielgtaylor/jpeg-archive https://github.com/danielgtaylor/jpeg-archive Tried a lot of combinations of quality, and settled for -q medium --max 75% --accurate Anything higher makes no sense/difference, even if you print it at A3. And my images are 8-18-24Mpix. i'm not a pro-foto but somewhat of an old pre-print school - Your level of pickiness may vary :/ Effectively it is like 4-8 times down - Total went from 150Gb into 30Gb - that's about 40,000 images. Which still doesn't solve the OP's or anyone's problem of too-easy-producible digital "assets".. but is more manageable.

ValentineC 5y ago
I guess it depends on how much one values "archiving", but WebP has a lossless option [1] that might be a good alternative to the above. [1] https://imagemagick.org/script/webp.php https://imagemagick.org/script/webp.php