5 ms·

How big of a NAS do you have?

by dfee 3y ago

compsciphd 3y ago
a single 1080p bluray movie (not including extras on the disc) is generally going to be in the low to mid 20GBs to 40GB range. Lets take 40GB for a fairly conservative measure. a single 8TB drive can therefore fit 200 (or possibly more) movies (or double for 16GB drives). i.e. one doesn't need "crazy" (i.e. more than a handful of consumer hard drives) amount of storage to store a lot of bluray movies and tv shows and keep them online available to you.
- vondur 3y ago
  I compress mine using Handbrake. It gets them down to the 2.5->8GB per movie range.
  - tombert 3y ago
    I used to compress them and generally that's fine, but honestly storage has gotten cheap enough to where I don't bother. Every now and then the compression would introduce artifacts, even at a relatively low CRF level in ffmpeg, and just for peace of mind I decided to keep the raw movies. That said, since I have an elaborate tape backup system now, I've debated keeping the "masters" as archives on the tapes, and keeping compressed versions for streaming, but laziness has kind of won out on that.
    - extragood 3y ago
      Agreed on your points about cheap storage and compression artifacts. I've entertained setting up a tape drive local backup for my NAS (4x 14 TB) for a few years. Is it worthwhile from your experience?
      - tombert 3y ago
        Yes, the tape drive is something I am very happy that I ended up doing. I have two LTO-6 tape drives. They only connect via SAS, which is a little annoying, but I bought a 2-port used SAS card on ebay for $20. I ended up buying a pack of like 100 LTO-6 tapes on eBay two years ago for a steal, like $150 in total. Each tape is 2.5TB, so I can pretty easily back up my movies. Each movies is written to two separate tapes. I keep a spreadsheet of which tape every movie lives, in case I need to restore a specific backup. I would really like to get an LTO-7 or LTO-8 drive with Thunderbolt, but they cost an arm and a leg even used on eBay right now. If you get lucky, an external SAS LTO-6 drive can be had for about $200-$300. It's a slightly expensive up-front cost I'll admit, however, it came after I was using Google archive storage, and when I did a restore of 21TB of movies, the next day I had a bill for like $700! It's saved my ass one time thus far; I accidentally broke my ZFS RAID and lost everything, but fortunately I was able to restore all my blu-ray rips from tape, and it was pretty easy, and more importantly, didn't cost me anything extra. If you decide to go down this road, I do recommend getting one that has support for LTFS; otherwise you're going to be stuck using tar commands I think.
        extragood 3y ago
        Thanks for the detailed response! That's really encouraging to hear, and I think thats enough to push me over the edge. Had a close call recently and lost a disk as I was preparing to migrate to my current hardware. I may adopt your 2 tape system as well. It'd be heartbreaking to lose my collection
  - Retric 3y ago
    Why bother? You're spending time saving ~50 cents per blue ray and you get some artifacts. You could just buy DVD's if you want to save space and are willing to take a hit to image quality.
    - compsciphd 3y ago
      I think many people dont get this. Let's do simple math. Let's imagine an 8TB drive costs $100 and can store 200 movies as their raw bluray encoding. that means each movie costs 50 cents to store without any real work. If one can divide the space needed by half, one only saved 25c. But how much did it cost to save that 25c. Lets imagine one can encode a movie with x264/5, but it will consume 100watts of cpu power over 10 hours to do that (or a kilowatt total). If one is paying 20c a killowat hour (US national average is 19c a kilowatt hour, california is 33c per google) one just paid 20c more to encode it, so if one cut the size in half, one in practice only saved 5c. If one spent 5-10 minutes setting up the encode, did one spend 5c of one's time doing that? Personally, the setup for me was also 25c worth of time. Now, one can "save more", if one is storing duplicate copies. (i.e. I'm storing it on 2 or more hard drives, not 1), but its still not really that significant. Now, if one isn't spending one's own time and energy to encode it (i.e. is pirating it as others did the work), then perhaps one can argue that a "good" encode vs the raw is better and makes more efficient usage of one's space. But the argument here isn't about pirating, but about storing one's own legally obtained media. As I've argued elsewhere, the only reasonable reason I've found for encoding is that one runs into "physical" space limitations (i.e. how many hard drives one can keep online). I'm just unconvinced that should override the time and cost of doing the encoding oneself.
  - vundercind 3y ago
    4.5-10GB in h.265, depending on the movie (length, how many low-contrast scenes, how much quick movement) and how well-tuned the encoder was, is indistinguishable from the 1080p blu ray on my 10’ diagonal screen at 6-10’ viewing distance. 4k, something like 25GB-45GB, depending, but I have several in the teens of GB that I wouldn’t guess weren’t full quality. I reserve full-quality rips for movies I both care about a lot and that benefit from it. I don’t need more than 4.5GB dedicated to Happy Gilmore or what have you.
tombert 3y ago
Keep in mind before you read this: I use my server for a lot of stuff other than movies so it's really over-provisioned. I play with LLM models sometimes, and I also have a Kafka server with tons of stock-trading info being written to nearly all the time taking a lot of space. That said, I have 24x16TB hard drives in ZFS RAID. It's three separate RAIDZ2's, so the amount of space on there is 18x16TB, so about 288TB. It's a fairly expensive amount of waste, but it's nice to be able to lose up to six disks at a time without having to worry. I could fairly easily get another 8 drives in there if I really needed it, but thus far my total consumption is only about 50TB in total, and I delete actually stuff when I'm done with it.
- BLKNSLVR 3y ago
  Ok, now I need a description of the setup. Specifically: - What hardware caters to 32x disks (24+8)? (I'm picturing enterprise gear) - What software are you using to coordinate it? TrueNAS?
  - davidzweig 3y ago
    You can fit a SAS card (like LSI 9207-8e) and hook up external disk shelves (like ds4246). This gear is on ebay and is mostly plug and play on linux.
  - tombert 3y ago
    So bear with me and hear me out on this. It's a fairly typical 1U HP Proliant server, which counts as "enterprise gear". I don't remember the model but it's somewhat unremarkable. It only has like six bays in the front, and I use one for a NixOS install, and two of them with SSDs for read and write cache. It has an old 10 gigabit ethernet card for connectivity. The server I bought used for around $150, the ethernet card I got for about $35 used on ebay. The actual disks live in 8 bay external chassis, and connected via USB3.0. This may sound horrifying, but USB3 is rated for like 5 gigabits, way more than you can easily get from a home RAID with spinning disks. I have a PCIe 3.0 x16 USB3 card in there, with four plugs, each with a dedicated controller, meaning that in theory each port could get 5 gigabits. The reason I could fairly easily get eight more in there is that I'm only using three of the four ports, so without much effort I could buy 8 more drives (of any size I suppose), put them into another 8 bay chassis, and just add vdev to my RAID. This particular PCIe USB card was $41 on eBay. I use ZFS for a software RAID. This sort of happened accidentally; I didn't used to have a rack mount server, I was using a bunch of Nvidia Jetson Nanos, and as such there wasn't a real way directly plug hard drives in, so I ended up having to use the external chassis. Eventually I bought three of those cases, had all the drives in there, and when I decided to buy a "real" server it was substantially easier and cheaper to just add proper USB support to a server than it was to find a decent 24-bay server at a decent price, particularly since I already had the cases. In hindsight I probably should have got chassis that have eSATA support, but they've been getting decent speed and it's an extremely convenient system. It's also come in handy once when I accidentally broke my server, and I needed to get files off the RAID; all I had to do was plug them into my laptop and mount the ZFS RAID there. ETA: Forgot to answer what I use to manage it. I installed NixOS on my server with a pretty vanilla ZFS on Linux install and an NFS mount. I do the tmpfs-on-root trick, so the root partition gets nuked on every reboot. https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ I don't use any fancy GUI tools or anything; NixOS does a good enough automating away the un-fun parts of maintaining a server, and if I break my server I can very easily roll back to a previous state, and if I really have to, reinstalling it from scratch takes like 20 minutes since I can just copy over my configuration file (which I back up on Gitlab) and have everything automatically set up again. NixOS is pretty cool. ETAA: According to `dmidecode` and `lscpu`, server model is `ProLiant DL380 G7`, 24 cores, 128GB of RAM. I think it's actually 12 cores, and it says 24 because of hyperthreading. It's pretty unremarkable. I like it a lot, but it's a fairly typical and bog-standard used server.
crispyambulance 3y ago
It's not ~too~ geeky to maintain a NAS. I got a Synology. It has 5 drives, and the ones I have are fairly quiet so the thing goes under the TV counter. However, if you want to rip 4K blurays, you've got to flash the firmware on your bluray drive and then run MakeMkV + handbrake (or much harder CLI stuff) to process a disc into a useable media file which can THEN go into Plex. All of that takes time and effort, and usually some trial and error. I also keep a bluray player next to the NAS. Simply because it's too much of a pain to deal with ripping sometimes. I still have to get through my 30-disc Ingmar Bergman Criterion Boxset that I bought 2 years ago. Much easier to pop a disc in the player!
- sroussey 3y ago
  Details of the flash firmware?
  - aspenmayer 3y ago
    Probably LibreDrive for use with MakeMKV. https://forum.makemkv.com/forum/viewtopic.php?t=18856 https://forum.makemkv.com/forum/viewtopic.php?t=18856
    - crispyambulance 3y ago
      Yep, that's it. It seems to be the only way to rip a 4K. No drives (that I know of) allow ripping 4K "out of the box"!
      - aspenmayer 3y ago
        I think you're correct; as far as I know ripping 4K/UHD with no modification to the drive and/or drive firmware is not possible without other workarounds, which apparently are possible, as I elaborate on below. Apparently as of recently at [CCC 37C3: Unlocked] (December 2023) Adam Batori demonstrated a side channel attack against Intel SGX and AACSv2 in his talk called "Full AACSess: Exposing and exploiting AACSv2 UHD DRM for your viewing pleasure" although that has nothing to do with the specific drive used to my knowledge, though the specific Intel CPU used may have quirks which may make their particular implementation CPU and/or CPU microcode and/or motherboard dependent. Previously on HN: Full AACSess: Exposing and exploiting AACSv2 UHD DRM for your viewing pleasure [video] 6 points by hcadam | 2023-12-29 | 1 comment https://news.ycombinator.com/item?id=38810011 https://news.ycombinator.com/item?id=38810011 https://media.ccc.de/v/37c3-12296-full_aacsess_exposing_and_exploiting_aacsv2_uhd_drm_for_your_viewing_pleasure https://media.ccc.de/v/37c3-12296-full_aacsess_exposing_and_... Video of the live demo above. Additionally, I have included a version of video on from the official media.ccc.de YouTube channel: https://www.youtube.com/watch?v=SEBuiecLZGg https://www.youtube.com/watch?v=SEBuiecLZGg Referenced by the above talk is prior work on defeating/bypassing AACS[v1] by Peter Eckersley in his talk titled "A Spotter's Guide to AACS Keys" at [CCC 24C3: Volldampf voraus!] in December 2007, which I also link below: https://media.ccc.de/v/24c3-2331-en-spotter_guide_to_aacs_keys https://media.ccc.de/v/24c3-2331-en-spotter_guide_to_aacs_ke... Also of interest from [CCC 36C3: Resource Exhaustion] is a talk by oct0xor titled "Hacking Sony PlayStation Blu-ray Drives" in December 2019: https://media.ccc.de/v/36c3-10567-hacking_sony_playstation_blu-ray_drives https://media.ccc.de/v/36c3-10567-hacking_sony_playstation_b... https://www.youtube.com/watch?v=WW39dsbffMw https://www.youtube.com/watch?v=WW39dsbffMw Apologies for lateness of my reply! I hope that helps.

How big of a NAS do you have?

by dfee 3y ago

How big of a NAS do you have?

a single 1080p bluray movie (not including extras on the disc) is generally going to be in the low to mid 20GBs to 40GB range. Lets take 40GB for a fairly conservative measure. a single 8TB drive can therefore fit 200 (or possibly more) movies (or double for 16GB drives). i.e. one doesn't need "crazy" (i.e. more than a handful of consumer hard drives) amount of storage to store a lot of bluray movies and tv shows and keep them online available to you.

vondur 3y ago
I compress mine using Handbrake. It gets them down to the 2.5->8GB per movie range.
- tombert 3y ago
  I used to compress them and generally that's fine, but honestly storage has gotten cheap enough to where I don't bother. Every now and then the compression would introduce artifacts, even at a relatively low CRF level in ffmpeg, and just for peace of mind I decided to keep the raw movies. That said, since I have an elaborate tape backup system now, I've debated keeping the "masters" as archives on the tapes, and keeping compressed versions for streaming, but laziness has kind of won out on that.
  - extragood 3y ago
    Agreed on your points about cheap storage and compression artifacts. I've entertained setting up a tape drive local backup for my NAS (4x 14 TB) for a few years. Is it worthwhile from your experience?
    - tombert 3y ago
      Yes, the tape drive is something I am very happy that I ended up doing. I have two LTO-6 tape drives. They only connect via SAS, which is a little annoying, but I bought a 2-port used SAS card on ebay for $20. I ended up buying a pack of like 100 LTO-6 tapes on eBay two years ago for a steal, like $150 in total. Each tape is 2.5TB, so I can pretty easily back up my movies. Each movies is written to two separate tapes. I keep a spreadsheet of which tape every movie lives, in case I need to restore a specific backup. I would really like to get an LTO-7 or LTO-8 drive with Thunderbolt, but they cost an arm and a leg even used on eBay right now. If you get lucky, an external SAS LTO-6 drive can be had for about $200-$300. It's a slightly expensive up-front cost I'll admit, however, it came after I was using Google archive storage, and when I did a restore of 21TB of movies, the next day I had a bill for like $700! It's saved my ass one time thus far; I accidentally broke my ZFS RAID and lost everything, but fortunately I was able to restore all my blu-ray rips from tape, and it was pretty easy, and more importantly, didn't cost me anything extra. If you decide to go down this road, I do recommend getting one that has support for LTFS; otherwise you're going to be stuck using tar commands I think.
      - extragood 3y ago
        Thanks for the detailed response! That's really encouraging to hear, and I think thats enough to push me over the edge. Had a close call recently and lost a disk as I was preparing to migrate to my current hardware. I may adopt your 2 tape system as well. It'd be heartbreaking to lose my collection
- Retric 3y ago
  Why bother? You're spending time saving ~50 cents per blue ray and you get some artifacts. You could just buy DVD's if you want to save space and are willing to take a hit to image quality.
  - compsciphd 3y ago
    I think many people dont get this. Let's do simple math. Let's imagine an 8TB drive costs $100 and can store 200 movies as their raw bluray encoding. that means each movie costs 50 cents to store without any real work. If one can divide the space needed by half, one only saved 25c. But how much did it cost to save that 25c. Lets imagine one can encode a movie with x264/5, but it will consume 100watts of cpu power over 10 hours to do that (or a kilowatt total). If one is paying 20c a killowat hour (US national average is 19c a kilowatt hour, california is 33c per google) one just paid 20c more to encode it, so if one cut the size in half, one in practice only saved 5c. If one spent 5-10 minutes setting up the encode, did one spend 5c of one's time doing that? Personally, the setup for me was also 25c worth of time. Now, one can "save more", if one is storing duplicate copies. (i.e. I'm storing it on 2 or more hard drives, not 1), but its still not really that significant. Now, if one isn't spending one's own time and energy to encode it (i.e. is pirating it as others did the work), then perhaps one can argue that a "good" encode vs the raw is better and makes more efficient usage of one's space. But the argument here isn't about pirating, but about storing one's own legally obtained media. As I've argued elsewhere, the only reasonable reason I've found for encoding is that one runs into "physical" space limitations (i.e. how many hard drives one can keep online). I'm just unconvinced that should override the time and cost of doing the encoding oneself.
- vundercind 3y ago
  4.5-10GB in h.265, depending on the movie (length, how many low-contrast scenes, how much quick movement) and how well-tuned the encoder was, is indistinguishable from the 1080p blu ray on my 10’ diagonal screen at 6-10’ viewing distance. 4k, something like 25GB-45GB, depending, but I have several in the teens of GB that I wouldn’t guess weren’t full quality. I reserve full-quality rips for movies I both care about a lot and that benefit from it. I don’t need more than 4.5GB dedicated to Happy Gilmore or what have you.

tombert 3y ago

Keep in mind before you read this: I use my server for a lot of stuff other than movies so it's really over-provisioned. I play with LLM models sometimes, and I also have a Kafka server with tons of stock-trading info being written to nearly all the time taking a lot of space. That said, I have 24x16TB hard drives in ZFS RAID. It's three separate RAIDZ2's, so the amount of space on there is 18x16TB, so about 288TB. It's a fairly expensive amount of waste, but it's nice to be able to lose up to six disks at a time without having to worry. I could fairly easily get another 8 drives in there if I really needed it, but thus far my total consumption is only about 50TB in total, and I delete actually stuff when I'm done with it.

BLKNSLVR 3y ago
Ok, now I need a description of the setup. Specifically: - What hardware caters to 32x disks (24+8)? (I'm picturing enterprise gear) - What software are you using to coordinate it? TrueNAS?
- davidzweig 3y ago
  You can fit a SAS card (like LSI 9207-8e) and hook up external disk shelves (like ds4246). This gear is on ebay and is mostly plug and play on linux.
- tombert 3y ago
  So bear with me and hear me out on this. It's a fairly typical 1U HP Proliant server, which counts as "enterprise gear". I don't remember the model but it's somewhat unremarkable. It only has like six bays in the front, and I use one for a NixOS install, and two of them with SSDs for read and write cache. It has an old 10 gigabit ethernet card for connectivity. The server I bought used for around $150, the ethernet card I got for about $35 used on ebay. The actual disks live in 8 bay external chassis, and connected via USB3.0. This may sound horrifying, but USB3 is rated for like 5 gigabits, way more than you can easily get from a home RAID with spinning disks. I have a PCIe 3.0 x16 USB3 card in there, with four plugs, each with a dedicated controller, meaning that in theory each port could get 5 gigabits. The reason I could fairly easily get eight more in there is that I'm only using three of the four ports, so without much effort I could buy 8 more drives (of any size I suppose), put them into another 8 bay chassis, and just add vdev to my RAID. This particular PCIe USB card was $41 on eBay. I use ZFS for a software RAID. This sort of happened accidentally; I didn't used to have a rack mount server, I was using a bunch of Nvidia Jetson Nanos, and as such there wasn't a real way directly plug hard drives in, so I ended up having to use the external chassis. Eventually I bought three of those cases, had all the drives in there, and when I decided to buy a "real" server it was substantially easier and cheaper to just add proper USB support to a server than it was to find a decent 24-bay server at a decent price, particularly since I already had the cases. In hindsight I probably should have got chassis that have eSATA support, but they've been getting decent speed and it's an extremely convenient system. It's also come in handy once when I accidentally broke my server, and I needed to get files off the RAID; all I had to do was plug them into my laptop and mount the ZFS RAID there. ETA: Forgot to answer what I use to manage it. I installed NixOS on my server with a pretty vanilla ZFS on Linux install and an NFS mount. I do the tmpfs-on-root trick, so the root partition gets nuked on every reboot. https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ I don't use any fancy GUI tools or anything; NixOS does a good enough automating away the un-fun parts of maintaining a server, and if I break my server I can very easily roll back to a previous state, and if I really have to, reinstalling it from scratch takes like 20 minutes since I can just copy over my configuration file (which I back up on Gitlab) and have everything automatically set up again. NixOS is pretty cool. ETAA: According to `dmidecode` and `lscpu`, server model is `ProLiant DL380 G7`, 24 cores, 128GB of RAM. I think it's actually 12 cores, and it says 24 because of hyperthreading. It's pretty unremarkable. I like it a lot, but it's a fairly typical and bog-standard used server.

crispyambulance 3y ago

It's not ~too~ geeky to maintain a NAS. I got a Synology. It has 5 drives, and the ones I have are fairly quiet so the thing goes under the TV counter. However, if you want to rip 4K blurays, you've got to flash the firmware on your bluray drive and then run MakeMkV + handbrake (or much harder CLI stuff) to process a disc into a useable media file which can THEN go into Plex. All of that takes time and effort, and usually some trial and error. I also keep a bluray player next to the NAS. Simply because it's too much of a pain to deal with ripping sometimes. I still have to get through my 30-disc Ingmar Bergman Criterion Boxset that I bought 2 years ago. Much easier to pop a disc in the player!

sroussey 3y ago
Details of the flash firmware?
- aspenmayer 3y ago
  Probably LibreDrive for use with MakeMKV. https://forum.makemkv.com/forum/viewtopic.php?t=18856 https://forum.makemkv.com/forum/viewtopic.php?t=18856
  - crispyambulance 3y ago
    Yep, that's it. It seems to be the only way to rip a 4K. No drives (that I know of) allow ripping 4K "out of the box"!
    - aspenmayer 3y ago
      I think you're correct; as far as I know ripping 4K/UHD with no modification to the drive and/or drive firmware is not possible without other workarounds, which apparently are possible, as I elaborate on below. Apparently as of recently at [CCC 37C3: Unlocked] (December 2023) Adam Batori demonstrated a side channel attack against Intel SGX and AACSv2 in his talk called "Full AACSess: Exposing and exploiting AACSv2 UHD DRM for your viewing pleasure" although that has nothing to do with the specific drive used to my knowledge, though the specific Intel CPU used may have quirks which may make their particular implementation CPU and/or CPU microcode and/or motherboard dependent. Previously on HN: Full AACSess: Exposing and exploiting AACSv2 UHD DRM for your viewing pleasure [video] 6 points by hcadam | 2023-12-29 | 1 comment https://news.ycombinator.com/item?id=38810011 https://news.ycombinator.com/item?id=38810011 https://media.ccc.de/v/37c3-12296-full_aacsess_exposing_and_exploiting_aacsv2_uhd_drm_for_your_viewing_pleasure https://media.ccc.de/v/37c3-12296-full_aacsess_exposing_and_... Video of the live demo above. Additionally, I have included a version of video on from the official media.ccc.de YouTube channel: https://www.youtube.com/watch?v=SEBuiecLZGg https://www.youtube.com/watch?v=SEBuiecLZGg Referenced by the above talk is prior work on defeating/bypassing AACS[v1] by Peter Eckersley in his talk titled "A Spotter's Guide to AACS Keys" at [CCC 24C3: Volldampf voraus!] in December 2007, which I also link below: https://media.ccc.de/v/24c3-2331-en-spotter_guide_to_aacs_keys https://media.ccc.de/v/24c3-2331-en-spotter_guide_to_aacs_ke... Also of interest from [CCC 36C3: Resource Exhaustion] is a talk by oct0xor titled "Hacking Sony PlayStation Blu-ray Drives" in December 2019: https://media.ccc.de/v/36c3-10567-hacking_sony_playstation_blu-ray_drives https://media.ccc.de/v/36c3-10567-hacking_sony_playstation_b... https://www.youtube.com/watch?v=WW39dsbffMw https://www.youtube.com/watch?v=WW39dsbffMw Apologies for lateness of my reply! I hope that helps.