5 ms·

What size drives is reasonable to get started something like this?

Frotag 3y ago
Personally anime uses around 30TB for me. For the best quality (mostly 1080p + bluray) of everything released in the last year (~250 shows / movies), but also including older releases for the same franchise (eg past seasons / related spin offs). For western media it'll really depend on how much quality you're willing to forgo (4k? hdr? blu-ray rips?) but 20GB for a 1080p season pack (~10 eps) and 5GB for a movie is pretty normal. And a cursory glance at certain sites suggests 250 seasons and 850 movies were released in 2023, so ~15TB seems like a good estimate for a years worth. Releases for asian stuff should be pretty similar in size, except less hdr / bluray options. Storage wise I just have a raid 1 (zfs) setup with a bunch of 14tb drives (at 8~10usd per TB [1]). I did need to buy a hba doodad to convert a pcie slot to more sata slots though. [1] https://diskprices.com/ https://diskprices.com/
- notso411 3y ago
  [dead]
- TacticalCoder 3y ago
  > 5GB for a movie is pretty normal. So 200 movies per TB. So 6 000 movies on a 30 TB. I know someone who has tens of thousands of movies and series. He's got 200 TB (with redundancy).
- dmw_ng 3y ago
  Wouldn't mind knowing total number of hours / audio / video codec / resolutions for that 30TB collection. Immediate thought is ridiculously overspecced codec configuration or signs of an addiction. Are they stored as lossless 10-bit or something?
  - numpad0 3y ago
    1920x1080x24bpp@30fps = 187MB/s or 1500Mbps, 11.2GB/min, 670GB/hr. 30TB / 670GB/hr = 44.7hrs raw. Shows and episodes wise, 30TB / 250 shows is 120GB/show, if by show GP meant a season, (120 GB) / (24 * 60 min) = 11.1 Mbps for a standard TV series, or (120 GB) / (12 * 30 min) = 44.4 Mbps for a standard 3-month anime seasons. Most 2K TV standards as well as streaming such as Netflix 4K are at 20-30Mbps bitrates, coming in already encoded in MPEG(-2/-4), so that's likely what GP is hoar-, collecting.
    - Frotag 3y ago
      Sorry I should clarify, the 250 shows + movies estimate comes from number of shows / movies listed on MAL for this season times 4. But since I also auto-download "related" releases for each thing airing (like seasons / movies from previous years), this comes out to ~24000 video files on disk (mp4 / mkv, excluding the million bluray raw segments). This tally is probably inflated by the bluray extras / op / ed files and a tiny bit by dupes (tracker is poorly organized) I'll dump the actual stats in a few minutes but point is the 30TB has a lot more than the year's worth I mentioned ("250 seasons / movies"), because myself and the people I share the library with like to catch up on stuff before watching the newest release. But yeah looks like ~15TB are from bluray remuxes which is definitely overkill (40~100 GiB per season).
  - Frotag 3y ago
    Bleh, took like 2 hours to scan every file. Anyways, overall stats are here https://gist.github.com/anon962/26bedc907e1f85123ae92df194de30f8 https://gist.github.com/anon962/26bedc907e1f85123ae92df194de...
  - deleted 3y ago
    [deleted]
anonym29 3y ago
Even a simple 4TB HDD is better than nothing. Though beware, this can easily turn into a hobby/passion/obsession that becomes more expensive than years or decades of underlying subscription costs. Before you know it, you start building your first dedicated NAS with a fancy file system that few people use (ZFS), configured with data striping and a distributed parity bit in RAIDZ (RAIDZ1), with UPS systems for high power availability. You'll become the go-to expert on hard drives among your friend group, and find yourself correctly answering questions like "What's the difference between CMR and SMR?". Backblaze's quarterly drive stats reports will start to become something you look forward to the release of, like a new episode coming out on your favorite TV show. You'll get acquainted with Kodi, Plex, Emby, Jellyfin, and more. You'll figure out the allure of mpv over VLC. You'll become the guy who reminds people that the cheapest hard drives are often the most expensive. You'll learn what private bittorrent trackers are, and pine after membership in the most exclusive ones. You will become intimately familiar with commercial VPNs, SSH, VPS offerings, and more. Be cautious, friend - once you decide to start taking this hobby seriously, you will dicover this is no mere hobby, but a lifestyle.
- leosanchez 3y ago
  > You'll figure out the allure of mpv over VLC Can you explain what mpv offers over vlc ? > cheapest hard drives are often the most expensive What hard drives do you buy / recommend?
  - dirtyhippiefree 3y ago
    Looking it up, MPV supports the LUA scripting language. The advantage there is cropping video, from what I’m reading. If cropping is possible with VLC, I’m not aware of it, and I’m a VLC fan.
  - iwontberude 3y ago
    18TB Exos Renewed have over 24 of them and none have failed in over a year now
    - iwontberude 3y ago
dharmab 3y ago
Optimal price per GB is somewhere in the 14-20TB range these days.
- tremon 3y ago
  Yes, this. Price/GB is probably the most important metric to consider, since the other metrics are either impossible to know (e.g. new drive reliability), hardly differ between products (e.g. power usage), or aren't particularly relevant for long-term archiving (e.g. read/write throughput, random seek latency). Though for archiving purposes, I would steer clear of SSD's. While the overall (claimed) reliability of SSD's is similar or better than HDD's, the failure mode is completely different: where HDD's tend to develop read errors over time which gives you enough warning to plan a replacement of the disk, the SSD's I've had just brick themselves after X time, which means the data loss is sudden and total, rather than gradual.
  - dharmab 3y ago
    I've also had SSDs become unreadable through normal means in as little as 3 months without power. Spinning disk is still the way to go for offline consumer archival.
  - gray_-_wolf 3y ago
    > Price/GB is probably the most important metric to consider, Sooner or later you will start hitting the physical space limit of the computer case. So if one wants to stick to "single PC" setup, it might be worth just picking the largest disks available, event if slightly more expensive per GB.
    - wincy 3y ago
      Yes, but if you check out what's available to a consumer right now, you have 18TB and 20TB drives, which are about $10 per terabyte, then 22TB jumps to $13+ per terabyte. If you insist on 24TB, you're looking at $23+, so more than double the price per terabyte. My beat up full sized ATX computer case I got on Craigslist can hold 10 HDD, so I'd go from 200TB to 240TB? 20% increase in capacity for an extra $3500 doesn't seem worth it for storing Gilmore Girls episodes because my wife swears she's gonna rewatch it!
morphle 3y ago
You usually want to start comparing the largest drives sizes for mass production consumer drives. In Q1 2024 that is around 20 TB. But size does not matter, price per GB (including price for the connection hardware to your workplace) matters most. I first check prices at [1] and decide if used or state of the art speeds are best. I then check prices at this Dutch comparison site [2] for internal drives or [3] for external drives. The advantage of [2] and [3] is that these prices are doublechecked by consumers and include all costs including shipping, while Amazon [1] are not always true. When I find the lowest price, I go check [4] to see if the brand and type drives have low error rates. I will then design and build an aggregate of drives, usually a few NAS (Network Attached Storage) servers based on USB 3.0 (slow and cheap), Thunderbolt (40 Gbps with a maximum of 240 Gbps), Ethernet (1-800 Gbps) or a network of FPGA's (10 Gbps to 25 Tbps). The overhead to aggregate and connect a bunch of drives ranges from $3 to $92 per drive, depending on your minimum and maximum speed requirements. For a small fee I will buy and build your storage array for you [5]. This is always cheaper for you than DIY or buying online because I pass on the lower bulk buy disk prices to you. [1] https://diskprices.com https://diskprices.com [2] https://tweakers.net/externe-harde-schijven/vergelijken/#filter:q1bKzCtLLSoJzsgvCE7NSU0uyczPU7IqKSpN1VHKL0pJLXLLTM1JUbJSKi5ITc5My0xOBKuASgbnF5UA5RKLk-EiQGWeQPXmRma1AA https://tweakers.net/externe-harde-schijven/vergelijken/#fil... [3] https://tweakers.net/externe-harde-schijven/vergelijken/#filter:q1bKzCtLLSoJzsgvCE7NSU0uyczPU7JKS8wpTtVRyi9KSS1yy0zNSVGyUiooyswqVoIKBucXlQDFEouT4SIFqcmeQHW6hrUA https://tweakers.net/externe-harde-schijven/vergelijken/#fil... [4] https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-2023/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-... [5] Please contact me at morphle at ziggo dot nl for price quotes of custom build drive arrays. Q1 2024 I build midrange 100Gbps storage NAS for around $0,009 per GB excl. VAT. Maximum speed storage is a few terabytes per second. 10GB/s around $0,009 per GB, 100GB/s around $0,011 per GB.
RamenJunkie_ 3y ago
It's going to depend entirely on the quality you are looking for. You don't have to go nuts like these people are talking about, you can store tons of movies in 4-10 TBs.
edmundsauto 3y ago
Used 12tb drives are < $100. Unraid is a nice UI to have a contiguous storage array with data protection via parity. Once I started down that road I couldn’t go back. Self hosting is fun and the ecosystem is pretty nice what with docker and all.
6510 3y ago
Just get a new drive when the previous one is full.

Personally anime uses around 30TB for me. For the best quality (mostly 1080p + bluray) of everything released in the last year (~250 shows / movies), but also including older releases for the same franchise (eg past seasons / related spin offs). For western media it'll really depend on how much quality you're willing to forgo (4k? hdr? blu-ray rips?) but 20GB for a 1080p season pack (~10 eps) and 5GB for a movie is pretty normal. And a cursory glance at certain sites suggests 250 seasons and 850 movies were released in 2023, so ~15TB seems like a good estimate for a years worth. Releases for asian stuff should be pretty similar in size, except less hdr / bluray options. Storage wise I just have a raid 1 (zfs) setup with a bunch of 14tb drives (at 8~10usd per TB [1]). I did need to buy a hba doodad to convert a pcie slot to more sata slots though. [1] https://diskprices.com/ https://diskprices.com/

notso411 3y ago
[dead]
TacticalCoder 3y ago
> 5GB for a movie is pretty normal. So 200 movies per TB. So 6 000 movies on a 30 TB. I know someone who has tens of thousands of movies and series. He's got 200 TB (with redundancy).
dmw_ng 3y ago
Wouldn't mind knowing total number of hours / audio / video codec / resolutions for that 30TB collection. Immediate thought is ridiculously overspecced codec configuration or signs of an addiction. Are they stored as lossless 10-bit or something?
- numpad0 3y ago
  1920x1080x24bpp@30fps = 187MB/s or 1500Mbps, 11.2GB/min, 670GB/hr. 30TB / 670GB/hr = 44.7hrs raw. Shows and episodes wise, 30TB / 250 shows is 120GB/show, if by show GP meant a season, (120 GB) / (24 * 60 min) = 11.1 Mbps for a standard TV series, or (120 GB) / (12 * 30 min) = 44.4 Mbps for a standard 3-month anime seasons. Most 2K TV standards as well as streaming such as Netflix 4K are at 20-30Mbps bitrates, coming in already encoded in MPEG(-2/-4), so that's likely what GP is hoar-, collecting.
  - Frotag 3y ago
    Sorry I should clarify, the 250 shows + movies estimate comes from number of shows / movies listed on MAL for this season times 4. But since I also auto-download "related" releases for each thing airing (like seasons / movies from previous years), this comes out to ~24000 video files on disk (mp4 / mkv, excluding the million bluray raw segments). This tally is probably inflated by the bluray extras / op / ed files and a tiny bit by dupes (tracker is poorly organized) I'll dump the actual stats in a few minutes but point is the 30TB has a lot more than the year's worth I mentioned ("250 seasons / movies"), because myself and the people I share the library with like to catch up on stuff before watching the newest release. But yeah looks like ~15TB are from bluray remuxes which is definitely overkill (40~100 GiB per season).
- Frotag 3y ago
  Bleh, took like 2 hours to scan every file. Anyways, overall stats are here https://gist.github.com/anon962/26bedc907e1f85123ae92df194de30f8 https://gist.github.com/anon962/26bedc907e1f85123ae92df194de...
- deleted 3y ago
  [deleted]

anonym29 3y ago

Even a simple 4TB HDD is better than nothing. Though beware, this can easily turn into a hobby/passion/obsession that becomes more expensive than years or decades of underlying subscription costs. Before you know it, you start building your first dedicated NAS with a fancy file system that few people use (ZFS), configured with data striping and a distributed parity bit in RAIDZ (RAIDZ1), with UPS systems for high power availability. You'll become the go-to expert on hard drives among your friend group, and find yourself correctly answering questions like "What's the difference between CMR and SMR?". Backblaze's quarterly drive stats reports will start to become something you look forward to the release of, like a new episode coming out on your favorite TV show. You'll get acquainted with Kodi, Plex, Emby, Jellyfin, and more. You'll figure out the allure of mpv over VLC. You'll become the guy who reminds people that the cheapest hard drives are often the most expensive. You'll learn what private bittorrent trackers are, and pine after membership in the most exclusive ones. You will become intimately familiar with commercial VPNs, SSH, VPS offerings, and more. Be cautious, friend - once you decide to start taking this hobby seriously, you will dicover this is no mere hobby, but a lifestyle.

leosanchez 3y ago
> You'll figure out the allure of mpv over VLC Can you explain what mpv offers over vlc ? > cheapest hard drives are often the most expensive What hard drives do you buy / recommend?
- dirtyhippiefree 3y ago
  Looking it up, MPV supports the LUA scripting language. The advantage there is cropping video, from what I’m reading. If cropping is possible with VLC, I’m not aware of it, and I’m a VLC fan.
- iwontberude 3y ago
  18TB Exos Renewed have over 24 of them and none have failed in over a year now
  - iwontberude 3y ago

dharmab 3y ago

Optimal price per GB is somewhere in the 14-20TB range these days.

tremon 3y ago
Yes, this. Price/GB is probably the most important metric to consider, since the other metrics are either impossible to know (e.g. new drive reliability), hardly differ between products (e.g. power usage), or aren't particularly relevant for long-term archiving (e.g. read/write throughput, random seek latency). Though for archiving purposes, I would steer clear of SSD's. While the overall (claimed) reliability of SSD's is similar or better than HDD's, the failure mode is completely different: where HDD's tend to develop read errors over time which gives you enough warning to plan a replacement of the disk, the SSD's I've had just brick themselves after X time, which means the data loss is sudden and total, rather than gradual.
- dharmab 3y ago
  I've also had SSDs become unreadable through normal means in as little as 3 months without power. Spinning disk is still the way to go for offline consumer archival.
- gray_-_wolf 3y ago
  > Price/GB is probably the most important metric to consider, Sooner or later you will start hitting the physical space limit of the computer case. So if one wants to stick to "single PC" setup, it might be worth just picking the largest disks available, event if slightly more expensive per GB.
  - wincy 3y ago
    Yes, but if you check out what's available to a consumer right now, you have 18TB and 20TB drives, which are about $10 per terabyte, then 22TB jumps to $13+ per terabyte. If you insist on 24TB, you're looking at $23+, so more than double the price per terabyte. My beat up full sized ATX computer case I got on Craigslist can hold 10 HDD, so I'd go from 200TB to 240TB? 20% increase in capacity for an extra $3500 doesn't seem worth it for storing Gilmore Girls episodes because my wife swears she's gonna rewatch it!

morphle 3y ago

You usually want to start comparing the largest drives sizes for mass production consumer drives. In Q1 2024 that is around 20 TB. But size does not matter, price per GB (including price for the connection hardware to your workplace) matters most. I first check prices at [1] and decide if used or state of the art speeds are best. I then check prices at this Dutch comparison site [2] for internal drives or [3] for external drives. The advantage of [2] and [3] is that these prices are doublechecked by consumers and include all costs including shipping, while Amazon [1] are not always true. When I find the lowest price, I go check [4] to see if the brand and type drives have low error rates. I will then design and build an aggregate of drives, usually a few NAS (Network Attached Storage) servers based on USB 3.0 (slow and cheap), Thunderbolt (40 Gbps with a maximum of 240 Gbps), Ethernet (1-800 Gbps) or a network of FPGA's (10 Gbps to 25 Tbps). The overhead to aggregate and connect a bunch of drives ranges from $3 to $92 per drive, depending on your minimum and maximum speed requirements. For a small fee I will buy and build your storage array for you [5]. This is always cheaper for you than DIY or buying online because I pass on the lower bulk buy disk prices to you. [1] https://diskprices.com https://diskprices.com [2] https://tweakers.net/externe-harde-schijven/vergelijken/#filter:q1bKzCtLLSoJzsgvCE7NSU0uyczPU7IqKSpN1VHKL0pJLXLLTM1JUbJSKi5ITc5My0xOBKuASgbnF5UA5RKLk-EiQGWeQPXmRma1AA https://tweakers.net/externe-harde-schijven/vergelijken/#fil... [3] https://tweakers.net/externe-harde-schijven/vergelijken/#filter:q1bKzCtLLSoJzsgvCE7NSU0uyczPU7JKS8wpTtVRyi9KSS1yy0zNSVGyUiooyswqVoIKBucXlQDFEouT4SIFqcmeQHW6hrUA https://tweakers.net/externe-harde-schijven/vergelijken/#fil... [4] https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-2023/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-... [5] Please contact me at morphle at ziggo dot nl for price quotes of custom build drive arrays. Q1 2024 I build midrange 100Gbps storage NAS for around $0,009 per GB excl. VAT. Maximum speed storage is a few terabytes per second. 10GB/s around $0,009 per GB, 100GB/s around $0,011 per GB.

RamenJunkie_ 3y ago

It's going to depend entirely on the quality you are looking for. You don't have to go nuts like these people are talking about, you can store tons of movies in 4-10 TBs.

edmundsauto 3y ago

Used 12tb drives are < $100. Unraid is a nice UI to have a contiguous storage array with data protection via parity. Once I started down that road I couldn’t go back. Self hosting is fun and the ecosystem is pretty nice what with docker and all.

6510 3y ago

Just get a new drive when the previous one is full.