7 ms·

I Built a 1 Petabyte Server from Scratch [video]

132 points by zdw 10mo ago

Brajeshwar 9mo ago
What would a typical household/homelab use such a storage for? (“Games & Stuff”)
- NoPicklez 9mo ago
  Memes
  - xipho 9mo ago
    That's no meme- it's a fully operational crazy custom build. A remarkable video that doesn't end in "like and mash subscribe".
    - bombcar 9mo ago
      I think they meant "store memes on it".
      - xipho 9mo ago
        That too! There seem to be a lot of YT videos that are memes on this theme as well though. He alludes to the fact that people make build videos that are essentially adding a couple of pre-existing more or less built components together. This is another level.
      - NoPicklez 9mo ago
        Yeah "memes" was the answer to the question of the post. Archival of the memes
    - OrvalWintermute 9mo ago
      Amazed by how many things he built in the process of making his NAS. High quality vid!
      - johanvts 9mo ago
        Cant respond to your EU tariff comment but just wanted to tell you that VAT and luxury tax applies to EU made products as well, so it doesn’t seem relevant to add when talking about tariffs.
- deleted 9mo ago
  [deleted]
- willtemperley 9mo ago
  Store the Sentinel 2 imagery for a year - about 500TB. Now I just have to find a way to avoid the $50k egress cost from AWS.
  - deleted 9mo ago
    [deleted]
- lanthade 9mo ago
  Data hoarders. I'm in a plex group on fb and there's people there with libraries that they could never personally watch all of. It sometimes seems like it's more a game of collecting all the things than it is about actually enjoying the collection.
  - stingraycharles 9mo ago
    This is correct, I personally aim to have all the highest quality versions of all movies, ie original Blu-ray. I have plenty of people that make use of it, it’s a hobby.
  - hrimfaxi 9mo ago
    Either that or they share their library with others (or maybe a bit of both)
  - tuananh 9mo ago
    how much does it cost in term of electricity per month?
    - bayindirh 9mo ago
      From "negligible" to "I have another house inside my house" levels, depending on your hardware. A close friend of mine runs a single beefy server at home, which is currently ~35% of his monthly bill if I'm not making mental-math mistakes.
      - overfeed 9mo ago
        As far as serious hobbies go, 35% of monthly electric bill is not breaking the bank. How much do people with other hobbies spend monthly on parts for their project cars, race entry fees, gym fees, "high performance" gear, Track fees, etc?
        bayindirh 9mo ago
        I didn't imply that it's a high amount. It's just an example I can refer reliably. I'm not a stranger to expensive hobbies. I have at least a couple of them. Photography and high end stationery being two of them.
- qingcharles 9mo ago
  "Linux ISOs"
  - bayindirh 9mo ago
    The funny thing is, I keep a set of historical Linux ISOs to be able to work with older servers in my fleet. Needing Debian 8 because that Lights Out connection requires JVM-something for the Java Web Start based console of the system. Moreover, funnily, some newer servers work wonkier with more modern ipmitools and browser versions while connecting remotely. Intricacies of older embedded systems.
  - jdboyd 9mo ago
    I do have an archive of Linux ISOs, but it is not anywhere near petabyte sized. Well I'm not trying to be comprehensive, everyone I download for many years now gets archived, and I am not sure that it is reached a terabyte yet.
- kybernetikos 9mo ago
  Z-library mirror maybe.
- PeterStuer 9mo ago
  I know a person that just "collects" games. They don't play them, they don't distribute them, it's just dowloaded and (poorly) cataloged like a Pokemon collection, unironically trying to catch them all.
- ahnick 9mo ago
  Chia farming.
- cookiengineer 9mo ago
  Knowledge archive. Everything I need to know to practice my job and life, I have an offline copy of. Wikipedia, SO, mediawikis, devdocs, git repos of dependencies etc. If google decides to shove AI generated results up our throats, that's the reasonable alternative. Currently I am building zimdex, as an alternative to the zim tools. Also if that's your thing, check out the kiwix.org project. It's really nice.
  - bayindirh 9mo ago
    Most of them should be pretty compressible though. How do you store them? Currently I'm running TrueNAS on a small NUC w/ 4SSDs and working on adding a mirrored pool via an external enclosure, but I'll be doing some bug fixing, it seems.
    - deleted 9mo ago
      [deleted]
lanthade 9mo ago
[flagged]
- irishcoffee 9mo ago
  I’ll just leave this here: https://news.ycombinator.com/item?id=46512881 https://news.ycombinator.com/item?id=46512881
- geerlingguy 9mo ago
  Note that here DIY means designing the chassis, the drive loader mechanism, PCBs for backplate, power distribution (including crimping dozens of power cables), modding PCIe cards to fit. It's not a boring "I bought a Chia mining server and inserted lots of hard drives" build.
- stingraycharles 9mo ago
  You should watch the video before commenting; in the first 10 seconds it’s explained that everything is built from scratch, including chassis.
  - orthoxerox 9mo ago
    Everything?
    - stingraycharles 9mo ago
      Well not everything of course, but at least the chassis and PCB boards. I don’t think the most upvoted comment on this thread should be dismissing this as “not impressive”, not even having watched the video.
      - sampullman 9mo ago
        nit: the last letter of PCB stands for "board"
  - lanthade 9mo ago
geoah 9mo ago
Any suggestions for UK alternatives for cutting and bending the steel sheets needed for the chassis to the suggested US company in the video? I’ve reached out to a couple for quotes a couple of days back but haven’t heard back yet.
- zipy124 9mo ago
  Here you go dude: - https://fractory.com/ https://fractory.com/ - https://lasercutsend.uk/ https://lasercutsend.uk/ - https://www.andoverlaser.co.uk/ https://www.andoverlaser.co.uk/ The second is the most similar to the company in the video, but the first is a much more established company. If none of these can do what you need, I suggest looking around the large manufacturing cities, so sheffield/derby/birmingham where there are still lots of small bespoke workshops that service large companies like forgemaster, rolls royce and JLR etc....
OrvalWintermute 9mo ago
full captures of VMs, snapshotting & ISOs eats up lots of storage Previously ran an mp3 scraper recording 30 stations simultaneously Good exposure to more music
KaiserPro 9mo ago
I dunno what the budget is, I've not had time to watch it all. however getting a second hand dell md3060 (they are rebadged from an OEM) for about £1k is also a good option. Its 60drives and mostly bulletproof. Downside is that you'll either need SAS controller on your server, or find the vanishingly rare Sata controllers. JBOD that badboy into ZFS and you'll have something fast enough for most things (streaming) How we used them was hardware raid 7 in 4 groups of 13 with the rest as hot spares. LVM raid 0 and good to go (this was a time before production ZFS on linux) I'm not sure what the compatibility is with larger sata drives given how old it is. I suspect you might be limited to JBOD.
- bayindirh 9mo ago
  I believe they're NetApp boxes, in fact. I have a ZFS JBOD supporting a 40ish machine cluster, and it works really well, 99.99% of the time, which is good enough. Mine is not very dense. 150TB/box.
  - KaiserPro 9mo ago
    Yeah I've deffo seen them on Netapps, but I'm not entirely convinced they are made by them. I saw some OEM versions of them "naked" as it were, but I can't remember what the company was called. When you look at them, they really don't have the same style as netapp did at the time. (or i'm wrong and senile.)
Tepix 9mo ago
Also relevant: Backblaze Storage Pod 6.0 https://www.backblaze.com/cloud-storage/resources/storage-pod https://www.backblaze.com/cloud-storage/resources/storage-po... https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv...
- Palomides 9mo ago
  as an aside, backblaze has since switched to off the shelf supermicro systems
- deleted 9mo ago
  [deleted]
_-_-__-_-_- 9mo ago
I watched this last week, it is a wonderful project. It took so much work and engineering.

132 points by zdw 10mo ago

Brajeshwar 9mo ago

What would a typical household/homelab use such a storage for? (“Games & Stuff”)

NoPicklez 9mo ago
Memes
- xipho 9mo ago
  That's no meme- it's a fully operational crazy custom build. A remarkable video that doesn't end in "like and mash subscribe".
  - bombcar 9mo ago
    I think they meant "store memes on it".
    - xipho 9mo ago
      That too! There seem to be a lot of YT videos that are memes on this theme as well though. He alludes to the fact that people make build videos that are essentially adding a couple of pre-existing more or less built components together. This is another level.
    - NoPicklez 9mo ago
      Yeah "memes" was the answer to the question of the post. Archival of the memes
  - OrvalWintermute 9mo ago
    Amazed by how many things he built in the process of making his NAS. High quality vid!
    - johanvts 9mo ago
      Cant respond to your EU tariff comment but just wanted to tell you that VAT and luxury tax applies to EU made products as well, so it doesn’t seem relevant to add when talking about tariffs.
deleted 9mo ago
[deleted]
willtemperley 9mo ago
Store the Sentinel 2 imagery for a year - about 500TB. Now I just have to find a way to avoid the $50k egress cost from AWS.
- deleted 9mo ago
  [deleted]
lanthade 9mo ago
Data hoarders. I'm in a plex group on fb and there's people there with libraries that they could never personally watch all of. It sometimes seems like it's more a game of collecting all the things than it is about actually enjoying the collection.
- stingraycharles 9mo ago
  This is correct, I personally aim to have all the highest quality versions of all movies, ie original Blu-ray. I have plenty of people that make use of it, it’s a hobby.
- hrimfaxi 9mo ago
  Either that or they share their library with others (or maybe a bit of both)
- tuananh 9mo ago
  how much does it cost in term of electricity per month?
  - bayindirh 9mo ago
    From "negligible" to "I have another house inside my house" levels, depending on your hardware. A close friend of mine runs a single beefy server at home, which is currently ~35% of his monthly bill if I'm not making mental-math mistakes.
    - overfeed 9mo ago
      As far as serious hobbies go, 35% of monthly electric bill is not breaking the bank. How much do people with other hobbies spend monthly on parts for their project cars, race entry fees, gym fees, "high performance" gear, Track fees, etc?
      - bayindirh 9mo ago
        I didn't imply that it's a high amount. It's just an example I can refer reliably. I'm not a stranger to expensive hobbies. I have at least a couple of them. Photography and high end stationery being two of them.
qingcharles 9mo ago
"Linux ISOs"
- bayindirh 9mo ago
  The funny thing is, I keep a set of historical Linux ISOs to be able to work with older servers in my fleet. Needing Debian 8 because that Lights Out connection requires JVM-something for the Java Web Start based console of the system. Moreover, funnily, some newer servers work wonkier with more modern ipmitools and browser versions while connecting remotely. Intricacies of older embedded systems.
- jdboyd 9mo ago
  I do have an archive of Linux ISOs, but it is not anywhere near petabyte sized. Well I'm not trying to be comprehensive, everyone I download for many years now gets archived, and I am not sure that it is reached a terabyte yet.
kybernetikos 9mo ago
Z-library mirror maybe.
PeterStuer 9mo ago
I know a person that just "collects" games. They don't play them, they don't distribute them, it's just dowloaded and (poorly) cataloged like a Pokemon collection, unironically trying to catch them all.
ahnick 9mo ago
Chia farming.
cookiengineer 9mo ago
Knowledge archive. Everything I need to know to practice my job and life, I have an offline copy of. Wikipedia, SO, mediawikis, devdocs, git repos of dependencies etc. If google decides to shove AI generated results up our throats, that's the reasonable alternative. Currently I am building zimdex, as an alternative to the zim tools. Also if that's your thing, check out the kiwix.org project. It's really nice.
- bayindirh 9mo ago
  Most of them should be pretty compressible though. How do you store them? Currently I'm running TrueNAS on a small NUC w/ 4SSDs and working on adding a mirrored pool via an external enclosure, but I'll be doing some bug fixing, it seems.
  - deleted 9mo ago
    [deleted]

lanthade 9mo ago

[flagged]

irishcoffee 9mo ago
I’ll just leave this here: https://news.ycombinator.com/item?id=46512881 https://news.ycombinator.com/item?id=46512881
geerlingguy 9mo ago
Note that here DIY means designing the chassis, the drive loader mechanism, PCBs for backplate, power distribution (including crimping dozens of power cables), modding PCIe cards to fit. It's not a boring "I bought a Chia mining server and inserted lots of hard drives" build.
stingraycharles 9mo ago
You should watch the video before commenting; in the first 10 seconds it’s explained that everything is built from scratch, including chassis.
- orthoxerox 9mo ago
  Everything?
  - stingraycharles 9mo ago
    Well not everything of course, but at least the chassis and PCB boards. I don’t think the most upvoted comment on this thread should be dismissing this as “not impressive”, not even having watched the video.
    - sampullman 9mo ago
      nit: the last letter of PCB stands for "board"
- lanthade 9mo ago

geoah 9mo ago

Any suggestions for UK alternatives for cutting and bending the steel sheets needed for the chassis to the suggested US company in the video? I’ve reached out to a couple for quotes a couple of days back but haven’t heard back yet.

zipy124 9mo ago
Here you go dude: - https://fractory.com/ https://fractory.com/ - https://lasercutsend.uk/ https://lasercutsend.uk/ - https://www.andoverlaser.co.uk/ https://www.andoverlaser.co.uk/ The second is the most similar to the company in the video, but the first is a much more established company. If none of these can do what you need, I suggest looking around the large manufacturing cities, so sheffield/derby/birmingham where there are still lots of small bespoke workshops that service large companies like forgemaster, rolls royce and JLR etc....

OrvalWintermute 9mo ago

full captures of VMs, snapshotting & ISOs eats up lots of storage Previously ran an mp3 scraper recording 30 stations simultaneously Good exposure to more music

KaiserPro 9mo ago

I dunno what the budget is, I've not had time to watch it all. however getting a second hand dell md3060 (they are rebadged from an OEM) for about £1k is also a good option. Its 60drives and mostly bulletproof. Downside is that you'll either need SAS controller on your server, or find the vanishingly rare Sata controllers. JBOD that badboy into ZFS and you'll have something fast enough for most things (streaming) How we used them was hardware raid 7 in 4 groups of 13 with the rest as hot spares. LVM raid 0 and good to go (this was a time before production ZFS on linux) I'm not sure what the compatibility is with larger sata drives given how old it is. I suspect you might be limited to JBOD.

bayindirh 9mo ago
I believe they're NetApp boxes, in fact. I have a ZFS JBOD supporting a 40ish machine cluster, and it works really well, 99.99% of the time, which is good enough. Mine is not very dense. 150TB/box.
- KaiserPro 9mo ago
  Yeah I've deffo seen them on Netapps, but I'm not entirely convinced they are made by them. I saw some OEM versions of them "naked" as it were, but I can't remember what the company was called. When you look at them, they really don't have the same style as netapp did at the time. (or i'm wrong and senile.)

Tepix 9mo ago

Also relevant: Backblaze Storage Pod 6.0 https://www.backblaze.com/cloud-storage/resources/storage-pod https://www.backblaze.com/cloud-storage/resources/storage-po... https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv...

Palomides 9mo ago
as an aside, backblaze has since switched to off the shelf supermicro systems
deleted 9mo ago
[deleted]

_-_-__-_-_- 9mo ago

I watched this last week, it is a wonderful project. It took so much work and engineering.