11 ms·

Redditor acquires decommissioned Netflix cache server with 262TB of storage

303 points by rcarmo 4y ago

thejosh 4y ago
Yeah, with a 9+ year old server, i wouldn't trust those disks, plus the amount of power...
- eric-hu 4y ago
  Would the hard disks be unreliable too?
- Nextgrid 4y ago
  With enough redundancy anything can be made reliable. ZFS can do some crazy RAID levels that will pretty much make anything reliable given enough disks and CPU resources. Whether it's a good idea (especially in terms of power usage) is a good question though.
  - ilikejam 4y ago
    You'd think so, but failing disks in raidz arrays can cause extreme performance problems, and failing disk electronics can cause lockups on SAS/SATA controllers. Old disks are always risky. Avoid.
- olavgg 4y ago
  I would love to have this. Old disks are no problem. They will most likely continue to run fine until they no longer are worth it due to either insane electricity prices or 50TB hard drives becomes available for a bargain. My backup server is still running almost 17-18 years old 500GB hard drives. They will be retired soon though, as they consume too much power and lack spindown when not in use.
  - throw0101c 4y ago
    > […] and lack spindown when not in use. I was under the impression that constant on-off was what put the most wear on a drive, and that if it is constantly on it would last longer.
    - fluoridation 4y ago
      Yep. In a previous iteration of my file server I killed a WD green drive in about two years because I unwittingly left the head park feature on. It was parking after 8 seconds of inactivity, and in two years it had accumulated like 2.5 million parks.
    - olavgg 4y ago
      Not really. With minimum 50000 cycles, you can spin up and down 28 times every day for 5 years. It may not sound a lot, but in reality it is. I have my NAS set to spindown after 15 minutes idle time and have on average around 1000 cycles every year.
  - gjulianm 4y ago
    I think there's a difference between the wear and tear in a personal backup server and in a Netflix cache server. We manage several storage servers which see intensive usage, and usually at around 5 years we start to see disk failures.
    - dachryn 4y ago
      its fair to assume netflix is mostly doing reads and not writes, and also fairly sequential. Thats very good for longevity of the disks
      - michaelt 4y ago
        True - but presumably they've been spinning and doing seeks pretty much 24/7. It's not like backblaze/coldline where the drives are practically powered down most of the time. And of course, 36x the drives means 36x the drive failures - and even if you avoid losing data, you've still got the chore of swapping each failed drive.
        dijit 4y ago
        Constantly spinning is better for the motor than starting and stopping.
        Maursault 4y ago
        > And of course, 36x the drives means 36x the drive failures I think you must mean 36x the chance of drive failure. Drive failure probability per year is published at around 1% though practically as high as 10%. 36 drives means there's somewhere between a 30.35% (1 - 0.99^36) and 97.7% (1 - 0.9^36) chance per year of a single drive failing.
      - justsomehnguy 4y ago
        > and also fairly sequential ... No. When you hit hundreds of concurrent operations there is no such thing as sequential. Also there is no need for sequential reads because a stripe of 7-15 disks would give you much more than streaming bitrate needs.
  - aasasd 4y ago
    For some reason the make and model of those disks is absent from your comment, must've been lost in the wires.
    - olavgg 4y ago
      They are a mix of Samsung, WD and Seagate. 2 of them are PATA, while the rest is SATA
  - dylan604 4y ago
    >or 50TB hard drives becomes available for a bargain. Maybe my old age is showing, but this just frightens me to no end. 1 50TB failure, and you've lost 50TB. Build an array of smaller drives to get to 50TB, and much less catastrophe if 1 drive dies.
    - NicoJuicy 4y ago
      Maintenance costs money too. What if you have to replace 3 times more often because of the smaller drives?
      - dylan604 4y ago
        if it costs some $ to be able to recover is still better than not being able to recover at all because the single point of failure fails
- AnIdiotOnTheNet 4y ago
  You should never trust any disk, that's why we have backups and redundancy in the first place.
- stfp 4y ago
  Also: noise. Those things were loud.
- winrid 4y ago
  Only about 180w average draw from the disks depending on workload
- runlevel1 4y ago
  According to this article, it consumes about 500W of power: https://www.phoronix.com/news/MTExNDM#:~:text=500w https://www.phoronix.com/news/MTExNDM#:~:text=500w Though I assume that's at typical Netflix workload levels.
- js2 4y ago
  I buy old SAS/SATA drives that have lots of hours on them. They tend to look like this: https://pastebin.com/raw/E39HuK59 https://pastebin.com/raw/E39HuK59 Run SMART, check the stats, make sure the grown defect list is 0, make sure it's never been over-temp, run a SMART long test, run bad blocks, run another SMART long test. If it passes all those I'd be fine continuing to use it. The HUH728080ALE600 drives in that server[1] idle at 5.1 watts[2], so it's 184 watts just on the drives. I'm guessing at idle the server runs around 300 watts. Which is not great, but it's not terrible. I pay ~ $1/watt/year in NC, so running that server 24x7 would cost me $300 annually. If it were me I'd disconnect at least half the drives to keep as spares. 1. https://imgur.com/a/zb6Mqty https://imgur.com/a/zb6Mqty 2. https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/data-center-drives/ultrastar-sas-series/data-sheet-ultrastar-he8.pdf https://documents.westerndigital.com/content/dam/doc-library...
  - dylan604 4y ago
    Cold spares can sometimes disappoint. Lots of techs preferred hot spares not just for immediate availability for the rebuild, but just because the spin-up was just as likely to kill the drive as other failure options. With hot-spares, you could rotate the drives by migrating data around like the old farmers partitioning their fields to let sections rest.
  - icedchai 4y ago
    I use HGST / old Hitachi drives in most of my servers, typically either 2 TB or 3 TB, though I do have some older 1 TBs still kicking (over 10 years, running 24x7.) In all the years, I've only had 1 fail. I just checked and one of them has over 91000 hours!
jdsnape 4y ago
I once worked for a large telco, and we wanted to run an Elasticsearch cluster to do some data analysis. Unfortunately, we worked in the wrong part of the organisation so we weren't allowed to buy servers :( However, we knew someone in the building opposite who was responsible for the company's streaming TV service. They were retiring a whole bunch of CDN machines, similar to these, which had a crazy amount of storage per node. While we weren't allowed to buy servers, we were allowed to buy components, so I got a bunch of the retired servers and bought enough RAM to max them out. Made a lovely ES cluster :)
- petercooper 4y ago
  While we weren't allowed to buy servers, we were allowed to buy components I'm curious. Could you have bought all the components to build your own servers under that policy? :-D
  - bluedino 4y ago
    Reminds me of the Sun repairman who would order extra parts here and there for customers, and ended up building his own machine.
    - registeredcorn 4y ago
      If you happen to have a link, I'd love to give that a read. :)
      - djbusby 4y ago
        A Listen: https://m.youtube.com/watch?v=uErKI0zWgjg https://m.youtube.com/watch?v=uErKI0zWgjg One piece at a time, Johnny Cash.
    - dylan604 4y ago
      Except one story is about people doing things legitimately to get things done within the constraints of their corporate overlords while the other is about a thief ripping people off. Not so similar after all
pearjuice 4y ago
Imagine if they weren't wiped and receiving (a snapshot of) the entire Netflix catalogue.
- crazysim 4y ago
  I think for most DRM they are stored and distributed pre-encrypted. Encryption isn't performed on these servers, they just shove bytes out.
- hhmc 4y ago
  I assume that content has all been comprehensively pirated already. So apart from the optics, it’s not clear it would do much real damage.
  - dachryn 4y ago
    it would just be a convenient nice to have, but indeed add no value over saving time and internet bandwidth
- nly 4y ago
  Would the entire netflix catalogue fit on this node?
  - Namidairo 4y ago
    Even if the entire catalogue didn't fit on there, they mostly exist as edge caches , so stuffing the most popular/upcoming things on there and watching the hit rate is probably good enough to reduce most of the needless transit. That and they had different tiers of hardware I believe.
  - guywhocodes 4y ago
    Maybe not if there are many encodings.
  - helsinkiandrew 4y ago
    You could fit 13,000 4K movies in 262TB. But Netflix will have versions for different resolutions/output devices pre baked so I reckon (I wouldn't want to gamble to much on this) that a single device wouldn't have everything.
eric4smith 4y ago
Use it as one of those BitTorrent storage servers.
- aasasd 4y ago
  I would finally be able to download all the Assetto Corsa and rFactor mods I want, and seed those things where there's just me and a couple other dudes so bailing feels bad—with still some storage left for peace of the mind.
andsoitis 4y ago
And if you’re interested in how Netflix’s CDN works: - https://www.theverge.com/22787426/netflix-cdn-open-connect https://www.theverge.com/22787426/netflix-cdn-open-connect - https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
- easton 4y ago
  The "briefing" PDF on that page has a GIF in it. Never seen that before.
  - retube 4y ago
    It seems wildly under-utilised, but .pdf supports fully dynamic and interactive graphics
    - brnt 4y ago
      This being underutilized is a good thing.
    - albrewer 4y ago
      At a job I had long ago, embedding an interactive 3d model of a laser scan diffed with a CAD model was one of the products of my work. It was probably the best and easiest way to get that information to other people because otherwise we would have had to find some other way for people to view point clouds with color embedded.
      - jmrm 4y ago
        IIRC Altium Designer can embed a complete PCB with their components rendered in 3D. Pretty useful to being able to see the model without having to open the software or even having it installed at all
    - dylan604 4y ago
      Good way to mess with people that like to print PDFs
vincengomes 4y ago
Link to the Reddit Post https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_... Followup Post https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ittgzsq/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_...
- snowwrestler 4y ago
  I wonder why these are deleted now.
  - anonu 4y ago
    Cease and desist from Netflix?
haunter 4y ago
There is a small Q&A from a Netflix dev about how they worked https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ittov57/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_... I noticed the display too, you can still buy them https://www.crystalfontz.com/product/cfa635tmlku-display-module-usb-20x4-text https://www.crystalfontz.com/product/cfa635tmlku-display-mod...
- newsclues 4y ago
  Those displays seem to be the same one sold to gamers years ago. I yearned for one as a youth!
  - r2_pilot 4y ago
    The Pertelian? I still have mine and came across the printed instruction set just the other day. Fun times. I've thought about trying to get it to work in the modern era, but there are small and info-dense screens these days.
    - 411111111111111 4y ago
      Displays just like that were on the Logitech G15 iirc. Nobody would take these keyboard seriously anymore but rubberdome was kinda the norm back then
  - Crosseye_Jack 4y ago
    Well they are "just" HD44780 driven LCD displays (which are sold on eBay pretty cheap these days). Back in the day when every machine had a parallel port we used to just drive them directly bit banging out what we wanted. These days you can just slap an Arduino inbetween one and a USB port and jobs pretty much done.
    - Slartie 4y ago
      Haha, yeah, I have fond memories of these displays. Got my hands onto one by accident as a kid, but didn't have any software to drive it, so I wrote me some software to drive those on the parallel port for display of CPU and RAM usage, number of mails in inbox, fan speeds and stuff. That program even gained a certain popularity in the case-modding scene, it was a hideous pile of crappy Visual Basic 4.0 code called "just another LCD software" or "jaLCDs".
dsfyu404ed 4y ago
We had one of these colocated at the rural state university I used to work at. Saved us tons of bandwidth not having tens of thousands of undergrads need to get their 1s and 0s from further afield. Probably improved typical download speeds for everyone upstream of us too.
komfushee 4y ago
Oh yeah, we used that as an ISP. Loved it for the deployment on local cable networks.
wscott 4y ago
Looks at the blog posts from "Drew Gallatin" on serving 100 Gbps (or more) from a single server. This is the server and that is no small task.
- 867-5309 4y ago
  >one quad-port 10-gigabit Ethernet NIC card
- drewg123 4y ago
  No, these pre-date this performance work. The original 100g work was done on a Xeon e5-2697a with NVME storage and no mechanical hard drives. I still use this class of machine as a benchmark to measure performance improvements
  - runlevel1 4y ago
    I'm curious, do you re-image the machines to test each of your changes or do you have a method with a faster feedback loop?
dagurp 4y ago
Can You Imagine a Beowulf Cluster of These?
- snapcaster 4y ago
  This comment always makes me laugh and then remember how old I am and get a little sad
  - xtracto 4y ago
    it's one of the earliest /. Era memes that I remember, along with "year of linux desktop" , "netcraft confirms" and "in Russia X does YOU". But I think all those are contemporary, along with "email is for old people " and others I don't remember now. Darn those were nice times.
    - rerx 4y ago
      Is a Beowulf cluster something conceptually different from any compute cluster / super computer running Linux today?
      - trelane 4y ago
        I think the main distinction is that modern clusters are shiny, sleek, fancy, and expensive. Whereas a Beowulf cluster is grungy, full of bailing twine, chewing gum, and possessing high levels of Bodge. And dirt cheap, built with what you can get, often used.
      - icco 4y ago
        Beowulfs were huge hacks back in the day, but are similar to most super computers. The main difference from how many think of compute clusters these days is it is managed as a single machine, so historically needed identical hardware. Parallel processing and concurrency used to be more rare :D https://en.wikipedia.org/wiki/Beowulf_cluster https://en.wikipedia.org/wiki/Beowulf_cluster
      - dekhn 4y ago
perihelions 4y ago
Other thread here: https://news.ycombinator.com/item?id=33350152 https://news.ycombinator.com/item?id=33350152 ("I got a Netflix cache server. 262TB", 27 comments)
anshumankmr 4y ago
Has anyone been able to get their hands on a decomissioned Google server?
- privong 4y ago
  > Has anyone been able to get their hands on a decomissioned Google server? Depending on what specifically you're after, there's a number of them on ebay: https://www.ebay.com/sch/i.html?_nkw=google+search+appliance https://www.ebay.com/sch/i.html?_nkw=google+search+appliance
- INTPenis 4y ago
  The re-branded Dell servers Google sent out as search indexers (I believe) were pretty common 10-15 years ago. I remember seeing two of them here in Sweden alone, during my career. But that is not the caseless DIY shelf-servers they've displayed in various PR videos.
  - runlevel1 4y ago
    In fact, there are a dozen on eBay right now: https://www.ebay.com/sch/i.html?_nkw=google+search+appliance https://www.ebay.com/sch/i.html?_nkw=google+search+appliance According to Wikipedia, the 2U models were indeed based on Dell PowerEdge servers, and the 1U models were apparently based on Super Micro servers.[^1] [1]: https://en.wikipedia.org/wiki/Google_Search_Appliance https://en.wikipedia.org/wiki/Google_Search_Appliance
  - snowwrestler 4y ago
    For those custom servers, I remember reading that Google did not bother unracking and disposing of them, but let the dead ones sit in the rack indefinitely. They figured out that it was more efficient to have hardware techs spend 100% of their time provisioning new hardware, and just let their software detect the broken servers, power them down, and route the tasks to working servers.
    - monocasa 4y ago
      At some point the hyperscalers end up decommissioning their servers because of perf/watt concerns. I don't think they bother with single failures for the reasons you've stated, but they do decommission full sections of DCs once the wattage calculus no longer makes sense.
drited 4y ago
Do Disney plus and HBO Max also have storage embedded with local ISPs or is this an advantage unique to Netflix?
- nosequel 4y ago
  Disney+, HBO, most streaming services not called Netflix use CDN services. Disney+ is heavily multi-CDN, at least they were at launch: https://www.csimagazine.com/csi/Disney-eating-up-Euro-peering-traffic-with-multi-CDN-approach.php https://www.csimagazine.com/csi/Disney-eating-up-Euro-peerin.... HBO also uses multiple CDNs, here is a reference from 2020 about GoT: https://www.prnewswire.com/news-releases/hbo-streams-game-of-thrones-season-7-using-level-3s-cdn-300488213.html https://www.prnewswire.com/news-releases/hbo-streams-game-of.... They were also mad that one of their distribution partners last week leaked the finale of House of the Dragon early: https://www.theguardian.com/tv-and-radio/2022/oct/21/house-of-the-dragon-hbo-season-finale-leak https://www.theguardian.com/tv-and-radio/2022/oct/21/house-o.... I don't have a link handy, but even Amazon has been known to use CDNs outside of CloudFront for their live events.
  - dylan604 4y ago
    >They were also mad that one of their distribution partners last week leaked the finale of House of the Dragon early I used to work for one of the companies supporting studios getting their content to streamers. This specific story is related to iTunes, but it's easy to do for any of them. The early days of "ramping up" meant putting butts in seats to do the work before automation was in place. The push to automate shot to highest priority when on of the butts in seats incorrectly copy/paste from one spreadsheet to the next which allowed an episode to be downloaded before it aired to anyone that subscribed to that season on iTunes. The fallout from that was incredible, but credit to the company they did not fire the employee for making a human mistake.
  - drited 4y ago
    I see, thank you!
bluedino 4y ago
> 36 7.2TB Western Digital hard disks Kind of a letdown > Interestingly, the now-defunct dial-up online service Prodigy used a local caching system to distribute data more efficiently using the same basic principle as Open Connect in the 1980s and '90s. Odd 'fact' to bring up.
loceng 4y ago
Reminds me of when I my dad had acquired two decommissioned hard drives that were used to store seafloor scans for the Halifax harbour and surrounding areas. From my own poor recollection, their dimensions were about 2' x 3' x 1.5' - and obviously quite heavy. They stored 1GB each.
steve_john 4y ago

303 points by rcarmo 4y ago

thejosh 4y ago

Yeah, with a 9+ year old server, i wouldn't trust those disks, plus the amount of power...

eric-hu 4y ago
Would the hard disks be unreliable too?
Nextgrid 4y ago
With enough redundancy anything can be made reliable. ZFS can do some crazy RAID levels that will pretty much make anything reliable given enough disks and CPU resources. Whether it's a good idea (especially in terms of power usage) is a good question though.
- ilikejam 4y ago
  You'd think so, but failing disks in raidz arrays can cause extreme performance problems, and failing disk electronics can cause lockups on SAS/SATA controllers. Old disks are always risky. Avoid.
olavgg 4y ago
I would love to have this. Old disks are no problem. They will most likely continue to run fine until they no longer are worth it due to either insane electricity prices or 50TB hard drives becomes available for a bargain. My backup server is still running almost 17-18 years old 500GB hard drives. They will be retired soon though, as they consume too much power and lack spindown when not in use.
- throw0101c 4y ago
  > […] and lack spindown when not in use. I was under the impression that constant on-off was what put the most wear on a drive, and that if it is constantly on it would last longer.
  - fluoridation 4y ago
    Yep. In a previous iteration of my file server I killed a WD green drive in about two years because I unwittingly left the head park feature on. It was parking after 8 seconds of inactivity, and in two years it had accumulated like 2.5 million parks.
  - olavgg 4y ago
    Not really. With minimum 50000 cycles, you can spin up and down 28 times every day for 5 years. It may not sound a lot, but in reality it is. I have my NAS set to spindown after 15 minutes idle time and have on average around 1000 cycles every year.
- gjulianm 4y ago
  I think there's a difference between the wear and tear in a personal backup server and in a Netflix cache server. We manage several storage servers which see intensive usage, and usually at around 5 years we start to see disk failures.
  - dachryn 4y ago
    its fair to assume netflix is mostly doing reads and not writes, and also fairly sequential. Thats very good for longevity of the disks
    - michaelt 4y ago
      True - but presumably they've been spinning and doing seeks pretty much 24/7. It's not like backblaze/coldline where the drives are practically powered down most of the time. And of course, 36x the drives means 36x the drive failures - and even if you avoid losing data, you've still got the chore of swapping each failed drive.
      - dijit 4y ago
        Constantly spinning is better for the motor than starting and stopping.
      - Maursault 4y ago
        > And of course, 36x the drives means 36x the drive failures I think you must mean 36x the chance of drive failure. Drive failure probability per year is published at around 1% though practically as high as 10%. 36 drives means there's somewhere between a 30.35% (1 - 0.99^36) and 97.7% (1 - 0.9^36) chance per year of a single drive failing.
    - justsomehnguy 4y ago
      > and also fairly sequential ... No. When you hit hundreds of concurrent operations there is no such thing as sequential. Also there is no need for sequential reads because a stripe of 7-15 disks would give you much more than streaming bitrate needs.
- aasasd 4y ago
  For some reason the make and model of those disks is absent from your comment, must've been lost in the wires.
  - olavgg 4y ago
    They are a mix of Samsung, WD and Seagate. 2 of them are PATA, while the rest is SATA
- dylan604 4y ago
  >or 50TB hard drives becomes available for a bargain. Maybe my old age is showing, but this just frightens me to no end. 1 50TB failure, and you've lost 50TB. Build an array of smaller drives to get to 50TB, and much less catastrophe if 1 drive dies.
  - NicoJuicy 4y ago
    Maintenance costs money too. What if you have to replace 3 times more often because of the smaller drives?
    - dylan604 4y ago
      if it costs some $ to be able to recover is still better than not being able to recover at all because the single point of failure fails
AnIdiotOnTheNet 4y ago
You should never trust any disk, that's why we have backups and redundancy in the first place.
stfp 4y ago
Also: noise. Those things were loud.
winrid 4y ago
Only about 180w average draw from the disks depending on workload
runlevel1 4y ago
According to this article, it consumes about 500W of power: https://www.phoronix.com/news/MTExNDM#:~:text=500w https://www.phoronix.com/news/MTExNDM#:~:text=500w Though I assume that's at typical Netflix workload levels.
js2 4y ago
I buy old SAS/SATA drives that have lots of hours on them. They tend to look like this: https://pastebin.com/raw/E39HuK59 https://pastebin.com/raw/E39HuK59 Run SMART, check the stats, make sure the grown defect list is 0, make sure it's never been over-temp, run a SMART long test, run bad blocks, run another SMART long test. If it passes all those I'd be fine continuing to use it. The HUH728080ALE600 drives in that server[1] idle at 5.1 watts[2], so it's 184 watts just on the drives. I'm guessing at idle the server runs around 300 watts. Which is not great, but it's not terrible. I pay ~ $1/watt/year in NC, so running that server 24x7 would cost me $300 annually. If it were me I'd disconnect at least half the drives to keep as spares. 1. https://imgur.com/a/zb6Mqty https://imgur.com/a/zb6Mqty 2. https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/data-center-drives/ultrastar-sas-series/data-sheet-ultrastar-he8.pdf https://documents.westerndigital.com/content/dam/doc-library...
- dylan604 4y ago
  Cold spares can sometimes disappoint. Lots of techs preferred hot spares not just for immediate availability for the rebuild, but just because the spin-up was just as likely to kill the drive as other failure options. With hot-spares, you could rotate the drives by migrating data around like the old farmers partitioning their fields to let sections rest.
- icedchai 4y ago
  I use HGST / old Hitachi drives in most of my servers, typically either 2 TB or 3 TB, though I do have some older 1 TBs still kicking (over 10 years, running 24x7.) In all the years, I've only had 1 fail. I just checked and one of them has over 91000 hours!

jdsnape 4y ago

I once worked for a large telco, and we wanted to run an Elasticsearch cluster to do some data analysis. Unfortunately, we worked in the wrong part of the organisation so we weren't allowed to buy servers :( However, we knew someone in the building opposite who was responsible for the company's streaming TV service. They were retiring a whole bunch of CDN machines, similar to these, which had a crazy amount of storage per node. While we weren't allowed to buy servers, we were allowed to buy components, so I got a bunch of the retired servers and bought enough RAM to max them out. Made a lovely ES cluster :)

petercooper 4y ago
While we weren't allowed to buy servers, we were allowed to buy components I'm curious. Could you have bought all the components to build your own servers under that policy? :-D
- bluedino 4y ago
  Reminds me of the Sun repairman who would order extra parts here and there for customers, and ended up building his own machine.
  - registeredcorn 4y ago
    If you happen to have a link, I'd love to give that a read. :)
    - djbusby 4y ago
      A Listen: https://m.youtube.com/watch?v=uErKI0zWgjg https://m.youtube.com/watch?v=uErKI0zWgjg One piece at a time, Johnny Cash.
  - dylan604 4y ago
    Except one story is about people doing things legitimately to get things done within the constraints of their corporate overlords while the other is about a thief ripping people off. Not so similar after all

pearjuice 4y ago

Imagine if they weren't wiped and receiving (a snapshot of) the entire Netflix catalogue.

crazysim 4y ago
I think for most DRM they are stored and distributed pre-encrypted. Encryption isn't performed on these servers, they just shove bytes out.
hhmc 4y ago
I assume that content has all been comprehensively pirated already. So apart from the optics, it’s not clear it would do much real damage.
- dachryn 4y ago
  it would just be a convenient nice to have, but indeed add no value over saving time and internet bandwidth
nly 4y ago
Would the entire netflix catalogue fit on this node?
- Namidairo 4y ago
  Even if the entire catalogue didn't fit on there, they mostly exist as edge caches , so stuffing the most popular/upcoming things on there and watching the hit rate is probably good enough to reduce most of the needless transit. That and they had different tiers of hardware I believe.
- guywhocodes 4y ago
  Maybe not if there are many encodings.
- helsinkiandrew 4y ago
  You could fit 13,000 4K movies in 262TB. But Netflix will have versions for different resolutions/output devices pre baked so I reckon (I wouldn't want to gamble to much on this) that a single device wouldn't have everything.

eric4smith 4y ago

Use it as one of those BitTorrent storage servers.

aasasd 4y ago
I would finally be able to download all the Assetto Corsa and rFactor mods I want, and seed those things where there's just me and a couple other dudes so bailing feels bad—with still some storage left for peace of the mind.

andsoitis 4y ago

And if you’re interested in how Netflix’s CDN works: - https://www.theverge.com/22787426/netflix-cdn-open-connect https://www.theverge.com/22787426/netflix-cdn-open-connect - https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/

easton 4y ago
The "briefing" PDF on that page has a GIF in it. Never seen that before.
- retube 4y ago
  It seems wildly under-utilised, but .pdf supports fully dynamic and interactive graphics
  - brnt 4y ago
    This being underutilized is a good thing.
  - albrewer 4y ago
    At a job I had long ago, embedding an interactive 3d model of a laser scan diffed with a CAD model was one of the products of my work. It was probably the best and easiest way to get that information to other people because otherwise we would have had to find some other way for people to view point clouds with color embedded.
    - jmrm 4y ago
      IIRC Altium Designer can embed a complete PCB with their components rendered in 3D. Pretty useful to being able to see the model without having to open the software or even having it installed at all
  - dylan604 4y ago
    Good way to mess with people that like to print PDFs

vincengomes 4y ago

Link to the Reddit Post https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_... Followup Post https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ittgzsq/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_...

snowwrestler 4y ago
I wonder why these are deleted now.
- anonu 4y ago
  Cease and desist from Netflix?

haunter 4y ago

There is a small Q&A from a Netflix dev about how they worked https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ittov57/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_... I noticed the display too, you can still buy them https://www.crystalfontz.com/product/cfa635tmlku-display-module-usb-20x4-text https://www.crystalfontz.com/product/cfa635tmlku-display-mod...

newsclues 4y ago
Those displays seem to be the same one sold to gamers years ago. I yearned for one as a youth!
- r2_pilot 4y ago
  The Pertelian? I still have mine and came across the printed instruction set just the other day. Fun times. I've thought about trying to get it to work in the modern era, but there are small and info-dense screens these days.
  - 411111111111111 4y ago
    Displays just like that were on the Logitech G15 iirc. Nobody would take these keyboard seriously anymore but rubberdome was kinda the norm back then
- Crosseye_Jack 4y ago
  Well they are "just" HD44780 driven LCD displays (which are sold on eBay pretty cheap these days). Back in the day when every machine had a parallel port we used to just drive them directly bit banging out what we wanted. These days you can just slap an Arduino inbetween one and a USB port and jobs pretty much done.
  - Slartie 4y ago
    Haha, yeah, I have fond memories of these displays. Got my hands onto one by accident as a kid, but didn't have any software to drive it, so I wrote me some software to drive those on the parallel port for display of CPU and RAM usage, number of mails in inbox, fan speeds and stuff. That program even gained a certain popularity in the case-modding scene, it was a hideous pile of crappy Visual Basic 4.0 code called "just another LCD software" or "jaLCDs".

dsfyu404ed 4y ago

We had one of these colocated at the rural state university I used to work at. Saved us tons of bandwidth not having tens of thousands of undergrads need to get their 1s and 0s from further afield. Probably improved typical download speeds for everyone upstream of us too.

komfushee 4y ago

Oh yeah, we used that as an ISP. Loved it for the deployment on local cable networks.

wscott 4y ago

Looks at the blog posts from "Drew Gallatin" on serving 100 Gbps (or more) from a single server. This is the server and that is no small task.

867-5309 4y ago
>one quad-port 10-gigabit Ethernet NIC card
drewg123 4y ago
No, these pre-date this performance work. The original 100g work was done on a Xeon e5-2697a with NVME storage and no mechanical hard drives. I still use this class of machine as a benchmark to measure performance improvements
- runlevel1 4y ago
  I'm curious, do you re-image the machines to test each of your changes or do you have a method with a faster feedback loop?

dagurp 4y ago

Can You Imagine a Beowulf Cluster of These?

snapcaster 4y ago
This comment always makes me laugh and then remember how old I am and get a little sad
- xtracto 4y ago
  it's one of the earliest /. Era memes that I remember, along with "year of linux desktop" , "netcraft confirms" and "in Russia X does YOU". But I think all those are contemporary, along with "email is for old people " and others I don't remember now. Darn those were nice times.
  - rerx 4y ago
    Is a Beowulf cluster something conceptually different from any compute cluster / super computer running Linux today?
    - trelane 4y ago
      I think the main distinction is that modern clusters are shiny, sleek, fancy, and expensive. Whereas a Beowulf cluster is grungy, full of bailing twine, chewing gum, and possessing high levels of Bodge. And dirt cheap, built with what you can get, often used.
    - icco 4y ago
      Beowulfs were huge hacks back in the day, but are similar to most super computers. The main difference from how many think of compute clusters these days is it is managed as a single machine, so historically needed identical hardware. Parallel processing and concurrency used to be more rare :D https://en.wikipedia.org/wiki/Beowulf_cluster https://en.wikipedia.org/wiki/Beowulf_cluster
    - dekhn 4y ago

perihelions 4y ago

Other thread here: https://news.ycombinator.com/item?id=33350152 https://news.ycombinator.com/item?id=33350152 ("I got a Netflix cache server. 262TB", 27 comments)

anshumankmr 4y ago

Has anyone been able to get their hands on a decomissioned Google server?

privong 4y ago
> Has anyone been able to get their hands on a decomissioned Google server? Depending on what specifically you're after, there's a number of them on ebay: https://www.ebay.com/sch/i.html?_nkw=google+search+appliance https://www.ebay.com/sch/i.html?_nkw=google+search+appliance
INTPenis 4y ago
The re-branded Dell servers Google sent out as search indexers (I believe) were pretty common 10-15 years ago. I remember seeing two of them here in Sweden alone, during my career. But that is not the caseless DIY shelf-servers they've displayed in various PR videos.
- runlevel1 4y ago
  In fact, there are a dozen on eBay right now: https://www.ebay.com/sch/i.html?_nkw=google+search+appliance https://www.ebay.com/sch/i.html?_nkw=google+search+appliance According to Wikipedia, the 2U models were indeed based on Dell PowerEdge servers, and the 1U models were apparently based on Super Micro servers.[^1] [1]: https://en.wikipedia.org/wiki/Google_Search_Appliance https://en.wikipedia.org/wiki/Google_Search_Appliance
- snowwrestler 4y ago
  For those custom servers, I remember reading that Google did not bother unracking and disposing of them, but let the dead ones sit in the rack indefinitely. They figured out that it was more efficient to have hardware techs spend 100% of their time provisioning new hardware, and just let their software detect the broken servers, power them down, and route the tasks to working servers.
  - monocasa 4y ago
    At some point the hyperscalers end up decommissioning their servers because of perf/watt concerns. I don't think they bother with single failures for the reasons you've stated, but they do decommission full sections of DCs once the wattage calculus no longer makes sense.

drited 4y ago

Do Disney plus and HBO Max also have storage embedded with local ISPs or is this an advantage unique to Netflix?

nosequel 4y ago
Disney+, HBO, most streaming services not called Netflix use CDN services. Disney+ is heavily multi-CDN, at least they were at launch: https://www.csimagazine.com/csi/Disney-eating-up-Euro-peering-traffic-with-multi-CDN-approach.php https://www.csimagazine.com/csi/Disney-eating-up-Euro-peerin.... HBO also uses multiple CDNs, here is a reference from 2020 about GoT: https://www.prnewswire.com/news-releases/hbo-streams-game-of-thrones-season-7-using-level-3s-cdn-300488213.html https://www.prnewswire.com/news-releases/hbo-streams-game-of.... They were also mad that one of their distribution partners last week leaked the finale of House of the Dragon early: https://www.theguardian.com/tv-and-radio/2022/oct/21/house-of-the-dragon-hbo-season-finale-leak https://www.theguardian.com/tv-and-radio/2022/oct/21/house-o.... I don't have a link handy, but even Amazon has been known to use CDNs outside of CloudFront for their live events.
- dylan604 4y ago
  >They were also mad that one of their distribution partners last week leaked the finale of House of the Dragon early I used to work for one of the companies supporting studios getting their content to streamers. This specific story is related to iTunes, but it's easy to do for any of them. The early days of "ramping up" meant putting butts in seats to do the work before automation was in place. The push to automate shot to highest priority when on of the butts in seats incorrectly copy/paste from one spreadsheet to the next which allowed an episode to be downloaded before it aired to anyone that subscribed to that season on iTunes. The fallout from that was incredible, but credit to the company they did not fire the employee for making a human mistake.
- drited 4y ago
  I see, thank you!

bluedino 4y ago

> 36 7.2TB Western Digital hard disks Kind of a letdown > Interestingly, the now-defunct dial-up online service Prodigy used a local caching system to distribute data more efficiently using the same basic principle as Open Connect in the 1980s and '90s. Odd 'fact' to bring up.

loceng 4y ago

Reminds me of when I my dad had acquired two decommissioned hard drives that were used to store seafloor scans for the Halifax harbour and surrounding areas. From my own poor recollection, their dimensions were about 2' x 3' x 1.5' - and obviously quite heavy. They stored 1GB each.

steve_john 4y ago