5 ms·

How do people use such large HDDs when internet download speeds are still low as compared to downloading directly on cloud services (like aws)?

detaro 8y ago
What do you consider low? On a 100Mbit/s connection you can fill a TB in a day or two, and those are somewhat common. And of course some people actually create the data themselves and don't download it.
aidenn0 8y ago
My dad and I created backups locally, mailed each other hard-drives, and we just do a weekly rsync. Storage is large, but network traffic is relatively low.
- astrodust 8y ago
  Having a "backup buddy" you can swap drives with once in a while is never a bad idea. Encrypted backup drives can save your bacon if you're ever caught in a bad situation.
bitL 8y ago
If you are doing serious Deep/Machine Learning you often get massive datasets, you might need to pre-process substantial portions of them for different models you want to try, you might want to do custom models that are trained e.g. on 4k/8k video footage (think DLSS, self-driving car or drone footage); space gets exhausted pretty quickly. You might also want multiple independent drives, as you might do this all in parallel and it would be too slow to do it on a single drive at the same time, and even RAID with two drives would have a large performance penalty for seeks.
peterburkimsher 8y ago
Internet speeds are slow in some places, such as France (specifically the Pays de Gex near Geneva, where my parents live). My dad uses iCloud, but he drives to CERN to upload (he just retired). I have 18 TB: 5 TB Seagate (x2), 4 TB WesternDigital, 2 TB WesternDigital, and 2 TB internal. Backups take the most space - I fix laptops for friends from church, and they don't back up but still want their files to be safe. I had to shuffle some files around to free up 650 GB for a recent repair, mostly photos & videos. Virtual machines use a lot of space too. I made VMWare Fusion images of every Mac OS version 10.5-10.13, Windows 95, 98, 2000, XP, 7, and 10, in several languages ( https://peterburk.github.com/i2018n https://peterburk.github.com/i2018n ), and some Linux distros. Another 1 TB is a dataset of Chinese characters from a machine learning project of mine ( https://blog.usejournal.com/making-of-a-chinese-characters-dataset-92d4065cc7cc https://blog.usejournal.com/making-of-a-chinese-characters-d... ). Music, mostly from repaired iPods back in high school, accounts for a lot as well. There's some movies too, though I missed a chance to get 2 TB from a friend because I didn't have enough space at the time. If I upload those, even those that I legally ripped from CDs & DVDs, I'm worried that it'll trigger content filters. For these, local disks are more useful than cloud services in my opinion.
- fencepost 8y ago
  I'm going to highly recommend looking at drives with WizTree, which does a very fast display of what's taking up space based on parsing the MFT rather than scanning the entire drive. You may find that there's a massive amount of data where you wouldn't expect it, such as in the Windows Temp directory - if so and it's a bunch of files named "cab_something", you can kill all of those and prevent recurrence with a little housekeeping. Details: https://www.computerworld.com/article/3112358/microsoft-windows/windows-7-log-file-compression-bug-can-fill-up-your-hard-drive.html https://www.computerworld.com/article/3112358/microsoft-wind... (update log files in windows\logs\cbs get auto-compressed, but compression breaks and leaves big temp files if the file to compress >2GB)
yellowapple 8y ago
Over a scale of a few years, things add up.

detaro 8y ago

What do you consider low? On a 100Mbit/s connection you can fill a TB in a day or two, and those are somewhat common. And of course some people actually create the data themselves and don't download it.

aidenn0 8y ago

My dad and I created backups locally, mailed each other hard-drives, and we just do a weekly rsync. Storage is large, but network traffic is relatively low.

astrodust 8y ago
Having a "backup buddy" you can swap drives with once in a while is never a bad idea. Encrypted backup drives can save your bacon if you're ever caught in a bad situation.

bitL 8y ago

If you are doing serious Deep/Machine Learning you often get massive datasets, you might need to pre-process substantial portions of them for different models you want to try, you might want to do custom models that are trained e.g. on 4k/8k video footage (think DLSS, self-driving car or drone footage); space gets exhausted pretty quickly. You might also want multiple independent drives, as you might do this all in parallel and it would be too slow to do it on a single drive at the same time, and even RAID with two drives would have a large performance penalty for seeks.

peterburkimsher 8y ago

Internet speeds are slow in some places, such as France (specifically the Pays de Gex near Geneva, where my parents live). My dad uses iCloud, but he drives to CERN to upload (he just retired). I have 18 TB: 5 TB Seagate (x2), 4 TB WesternDigital, 2 TB WesternDigital, and 2 TB internal. Backups take the most space - I fix laptops for friends from church, and they don't back up but still want their files to be safe. I had to shuffle some files around to free up 650 GB for a recent repair, mostly photos & videos. Virtual machines use a lot of space too. I made VMWare Fusion images of every Mac OS version 10.5-10.13, Windows 95, 98, 2000, XP, 7, and 10, in several languages ( https://peterburk.github.com/i2018n https://peterburk.github.com/i2018n ), and some Linux distros. Another 1 TB is a dataset of Chinese characters from a machine learning project of mine ( https://blog.usejournal.com/making-of-a-chinese-characters-dataset-92d4065cc7cc https://blog.usejournal.com/making-of-a-chinese-characters-d... ). Music, mostly from repaired iPods back in high school, accounts for a lot as well. There's some movies too, though I missed a chance to get 2 TB from a friend because I didn't have enough space at the time. If I upload those, even those that I legally ripped from CDs & DVDs, I'm worried that it'll trigger content filters. For these, local disks are more useful than cloud services in my opinion.

fencepost 8y ago
I'm going to highly recommend looking at drives with WizTree, which does a very fast display of what's taking up space based on parsing the MFT rather than scanning the entire drive. You may find that there's a massive amount of data where you wouldn't expect it, such as in the Windows Temp directory - if so and it's a bunch of files named "cab_something", you can kill all of those and prevent recurrence with a little housekeeping. Details: https://www.computerworld.com/article/3112358/microsoft-windows/windows-7-log-file-compression-bug-can-fill-up-your-hard-drive.html https://www.computerworld.com/article/3112358/microsoft-wind... (update log files in windows\logs\cbs get auto-compressed, but compression breaks and leaves big temp files if the file to compress >2GB)

yellowapple 8y ago

Over a scale of a few years, things add up.