5 ms·

We did the same. We've replaced nginx/lua with a cache server (for video) written in Golang - now serving up to 100 Gbit/s per node. It's more CPU and memory ef

We did the same. We've replaced nginx/lua with a cache server (for video) written in Golang - now serving up to 100 Gbit/s per node. It's more CPU and memory efficient and completely tailored to our needs. We are happy that we moved away from nginx.

BonoboIO 4y ago
Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.
- trillic 4y ago
  100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.
  - mplewis 4y ago
    “Only” :)
    - 988747 4y ago
      Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.
      - nly 4y ago
        It's almost as if on-demand video streaming isnt suitable for broadcasting live events.
  - totallyunknown 4y ago
    100 Gbit/s / 5000 KiB/s is 20000.
    - ImprobableTruth 4y ago
      KiB = Kibibyte, not Kibibit
  - zamadatix 4y ago
    40 Mbps seems roughly 5x-10x the bitrate I'd expect for this kind of thing.
- totallyunknown 4y ago
  A german company building an app for watching linear TV. Netflix is actually serving 400Gbit/s per node and already have 800Gbit/s ready. I think we can scale our setup up to 200 Gbit/s but we are too small. Total traffic is ~2 Tbit/s. Most challenging is the missing support of QUIC/http3 and KTLS in Golang. Also 100G NIC supply chain is difficult. We use NVIDIA Connect-X 6, but it's impossible to get a version with TLS offloading.
  - BonoboIO 4y ago
    Es liegt mir auf der Zunge welche Firma das ist. I think it starts with a Wa ... you don’t have to say. I kind of remember to have been stumbled on a Twitter engineering ipv6 tweet. Maybe I m wrong. For me it’s impressive to get so much data through a computer. But I have one question, what does count as a node, is a node like 1 machine with dual sockets, a lot of ram and a lot of nics or is it like multiple machines combined that act as 1 node like a whole 19 inch rack.
    - totallyunknown 4y ago
      A single node is single socket AMD EPCY node with 32 cores, 64 or 128GB RAM, 1x NIC with 2x100 GB/s. Ja, es ist waipu.tv.
      - ddorian43 4y ago
        What about the disks? Edit: or linear TV means the recent stream can be all in ram?
        totallyunknown 4y ago
        All in memory.
  - lossolo 4y ago
    Interesting, do you do a lot of processing in Golang or basically you just use it as a wrapper around sendfile[1] ? 1. https://man7.org/linux/man-pages/man2/sendfile.2.html https://man7.org/linux/man-pages/man2/sendfile.2.html

BonoboIO 4y ago

Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.

trillic 4y ago
100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.
- mplewis 4y ago
  “Only” :)
  - 988747 4y ago
    Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.
    - nly 4y ago
      It's almost as if on-demand video streaming isnt suitable for broadcasting live events.
- totallyunknown 4y ago
  100 Gbit/s / 5000 KiB/s is 20000.
  - ImprobableTruth 4y ago
    KiB = Kibibyte, not Kibibit
- zamadatix 4y ago
  40 Mbps seems roughly 5x-10x the bitrate I'd expect for this kind of thing.
totallyunknown 4y ago
A german company building an app for watching linear TV. Netflix is actually serving 400Gbit/s per node and already have 800Gbit/s ready. I think we can scale our setup up to 200 Gbit/s but we are too small. Total traffic is ~2 Tbit/s. Most challenging is the missing support of QUIC/http3 and KTLS in Golang. Also 100G NIC supply chain is difficult. We use NVIDIA Connect-X 6, but it's impossible to get a version with TLS offloading.
- BonoboIO 4y ago
  Es liegt mir auf der Zunge welche Firma das ist. I think it starts with a Wa ... you don’t have to say. I kind of remember to have been stumbled on a Twitter engineering ipv6 tweet. Maybe I m wrong. For me it’s impressive to get so much data through a computer. But I have one question, what does count as a node, is a node like 1 machine with dual sockets, a lot of ram and a lot of nics or is it like multiple machines combined that act as 1 node like a whole 19 inch rack.
  - totallyunknown 4y ago
    A single node is single socket AMD EPCY node with 32 cores, 64 or 128GB RAM, 1x NIC with 2x100 GB/s. Ja, es ist waipu.tv.
    - ddorian43 4y ago
      What about the disks? Edit: or linear TV means the recent stream can be all in ram?
      - totallyunknown 4y ago
        All in memory.
- lossolo 4y ago
  Interesting, do you do a lot of processing in Golang or basically you just use it as a wrapper around sendfile[1] ? 1. https://man7.org/linux/man-pages/man2/sendfile.2.html https://man7.org/linux/man-pages/man2/sendfile.2.html