3 ms·

The actual videos that I store locally are in RAIDZ6, replicated using ZFS snapshots to two local online copies, with a third copy every night pushing data up t

The actual videos that I store locally are in RAIDZ6, replicated using ZFS snapshots to two local online copies, with a third copy every night pushing data up to an Amazon S3 Glacier Deep Archive vault :)

mikestorrent 2y ago
While it's nice to have a backup video site, it really makes me miss the 90s style internet, where we had protocols and servers separated from clients. What you really need is to be able to trivially publish videos to something like IPFS, and have a large contingent of folks out there using convenient local video apps so there's a market for the content. I could imagine something like "community indexes" (perhaps a spiritual descendant of Usenet) to help find content, playlists, etc. and handle recommendation ML stuff locally. If we had been a little more wise and avoided the walled gardens of social networks in favour of identi.ca or GNU social etc. aeons ago, we would be in a better state now in many ways.
- dehrmann 2y ago
  > 90s style internet In the early days of the internet, people would host servers out of a university lab or their home for fun or to give back to the community. This didn't scale as more people came online. There were too many people and too few hosts. Protocols moved slower than product development. People needed to download apps (and this was before app stores). So the corporate web won.
- geerlingguy 2y ago
  PeerTube is supposedly that, but it has many of the same drawbacks of self-hosting anything that gets mildly popular. Once it gets more traffic, you get more bots, more AI tools pulling all the content, more botnets and DDoS attempts, eventually have to deal with governments and DMCA... It's tough :(
  - deleted 2y ago
    [deleted]
mvanbaak 2y ago
RAIDZ2 you mean? (which is comparable with RAID-6). Or do you really run 6 parity disks in your vdev?
- geerlingguy 2y ago
  Yes that haha! I was typing RAID6 then was like no, these servers are using ZFS... My configuration, in case anyone's interested: https://github.com/geerlingguy/arm-nas https://github.com/geerlingguy/arm-nas
skyyler 2y ago
I understand if you don't want to disclose this, but I'm very curious. Can you give a rough estimate of how much your Glacier Deep Archive usage costs?
- seized 2y ago
  It's about $1USD/TB/month (varies slightly by AWS region). Unless you need to retrieve and download, where it works out to about $90USD/TB.
- geerlingguy 2y ago
  For 13 TB of data, it's something like $20/month? That includes the S3 calls to check all data and upload new stuff.
  - skyyler 2y ago
    That's pretty reasonable, honestly. Thanks!

mikestorrent 2y ago

While it's nice to have a backup video site, it really makes me miss the 90s style internet, where we had protocols and servers separated from clients. What you really need is to be able to trivially publish videos to something like IPFS, and have a large contingent of folks out there using convenient local video apps so there's a market for the content. I could imagine something like "community indexes" (perhaps a spiritual descendant of Usenet) to help find content, playlists, etc. and handle recommendation ML stuff locally. If we had been a little more wise and avoided the walled gardens of social networks in favour of identi.ca or GNU social etc. aeons ago, we would be in a better state now in many ways.

dehrmann 2y ago
> 90s style internet In the early days of the internet, people would host servers out of a university lab or their home for fun or to give back to the community. This didn't scale as more people came online. There were too many people and too few hosts. Protocols moved slower than product development. People needed to download apps (and this was before app stores). So the corporate web won.
geerlingguy 2y ago
PeerTube is supposedly that, but it has many of the same drawbacks of self-hosting anything that gets mildly popular. Once it gets more traffic, you get more bots, more AI tools pulling all the content, more botnets and DDoS attempts, eventually have to deal with governments and DMCA... It's tough :(
- deleted 2y ago
  [deleted]

mvanbaak 2y ago

RAIDZ2 you mean? (which is comparable with RAID-6). Or do you really run 6 parity disks in your vdev?

geerlingguy 2y ago
Yes that haha! I was typing RAID6 then was like no, these servers are using ZFS... My configuration, in case anyone's interested: https://github.com/geerlingguy/arm-nas https://github.com/geerlingguy/arm-nas

skyyler 2y ago

I understand if you don't want to disclose this, but I'm very curious. Can you give a rough estimate of how much your Glacier Deep Archive usage costs?

seized 2y ago
It's about $1USD/TB/month (varies slightly by AWS region). Unless you need to retrieve and download, where it works out to about $90USD/TB.
geerlingguy 2y ago
For 13 TB of data, it's something like $20/month? That includes the S3 calls to check all data and upload new stuff.
- skyyler 2y ago
  That's pretty reasonable, honestly. Thanks!