drewg123

75 ms·

931.
by drewg123 5y ago
We actually try really, really hard not to blast 400Gb/s at a single client. The 400Gb/s is in aggregate. Our transport team is working on packet pacing, or really packet spreading, so that any bursts we send are small enough to
932.
by drewg123 5y ago
Caching is left off for simplicity. We keep track of popular titles, and try to cache them in RAM, using the normal page cache LRU mechanism. Other titles are marked with SF_NOCACHE and are discarded from RAM ASAP.
933.
by drewg123 5y ago
We test on production traffic. We don't have a testbench. This is problematic, because sometimes results are not reproducible. Eg, if I test on the day of a new release of a popular title, we might be serving a lot of it cached from R
934.
by drewg123 5y ago
In terms of LACP in general, not for hw TLS. For HW TLS, the keys and crypto state are NIC specific.
935.
by drewg123 5y ago
We're open to migrating to the best technology for the use case. Arm beat Intel to Gen4 PCIe, which is one of the reasons we tried them. Our stack runs just fine on arm64, thanks to the core FreeBSD support for arm64 The 800Gb is jus
936.
by drewg123 5y ago
Yes. I ran it for a bake off ~2 years ago. At the time, the code in linux was pretty raw, and I had to fix a bug in their SW kTLS that caused data corruption that was visible to clients. So I worry that it was not in frequent use at the
937.
by drewg123 5y ago
Thanks for the feeback. I was hoping that the other words on the slide (LACP and bonding) would give enough context. I'm afraid that my presentation didn't really have room to dive much into LACP. I briefly said something like t
938.
by drewg123 5y ago
To be honest, it was mostly the suggestion from AMD. At the time, AMD was the only Gen4 PCIe available, and it was hard to determine if the Mellanox NIC or the AMD PCIe root was the limiting factor. When AMD suggested Relaxed Ordering, tha
939.
by drewg123 5y ago
No, 400Gb/s == 50 GB/s With software kTLS, the data is moved an and out of memory 4x, so that's 200GB/s of bandwidth needed to serve 50GB/s (400Gb/s) of data.
940.
by drewg123 5y ago
Yes. See a tech blog from last year: https://netflixtechblog.com/how-netflix-brings-safer-and-fas...
941.
by drewg123 5y ago
If you could email me your ranges, I can try to look into it internally. Use my last name at gmail.com (or at freebsd.org)
942.
by drewg123 5y ago
We've looked at http3 / quic, and have not yet seen any advantages for our workload. Just a large loss of CPU efficiency.
943.
by drewg123 5y ago
Its important to consider that we've poured man years into this workload on FreeBSD. Just off the top of my head, we've worked on in house, and/or contributed to or funded, or encouraged vendors to pursue: - async sendfile (
944.
by drewg123 5y ago
One thing to remember with ZFS is that it does not use the page cache; it uses ARC. This means that sendfile is not zero-copy with ZFS, and I think that async sendfile may not work at all. We use ZFS only for boot/root/logs and
945.
by drewg123 5y ago
My understanding is that licensing did factor into the decision. However, I didn't join Netflix until after the decision had been made.
946.
by drewg123 5y ago
Wow, I never zoomed in on that. I've been using that as clip art in my slides for years now. You should work for CSI
947.
by drewg123 5y ago
Because its network bandwidth, and network units are in bits for historical reasons. I actually griped about this in my talk .. it makes it a pain to talk about memory bw vs network bw without mental gymnastics to move around that factor
948.
by drewg123 5y ago
There is a EuroBSDCon youtube channel. The last time I presented, it was a few weeks before the recordings were processed and posted to the channel. I'm not sure how soon they will be available this year. I have to say that a lot of
949.
by drewg123 5y ago
IPv4 and IPv6 can both use kTLS. We offer service via V6, but most clients connect via IPv4. It differs by region, and even time of day, but IPv4 is still the vast majority of traffic.
950.
by drewg123 5y ago
I don't have the answer, and even if I did, I'm not sure I'd be allowed to tell you :) But note that these are flash servers; they serve the most popular content we have. We also have "storage" servers with huge nu
951.
by drewg123 5y ago
If we could use more than 1 IP, then we could treat 1 400Gb box as 4 100Gb boxes. That could lead to the "perfect case" every time, since connections would always stay local to the numa node where content is present.
952.
by drewg123 5y ago
I'm not on the hardware team, so I don't have the cost breakdown. But my understanding is that flash storage is the most expensive line item, and it matters little if its consolidated into one box, or spread over a rack, you stil
953.
by drewg123 5y ago
Yes, Linux has kTLS. When I tried to use it, it was horribly broken, so my fear is that its not well used/tested, but it exists. Mellanox, for example, developed their inline hardware kTLS offload on linux.
954.
by drewg123 5y ago
FreeBSD was selected at the outset of the Open Connect CDN (~2012 or so). We did a bake off a few years ago, and FOR THIS WORKLOAD FreeBSD outperformed Linux. I don't want to get into an OS war, that's not productive.
955.
by drewg123 5y ago
These are the slides from my EuroBSDCon presentation. AMA
956.
Serving Netflix Video at 400Gb/s on FreeBSD [pdf](people.freebsd.org)
707 points by drewg123 5y ago|292 comments
957.
by drewg123 5y ago
Rack is not tuned for single stream bandwidth. There are things that it does which may lead to lower single stream bandwidth. For example, if you have 1MB ready to send and enough receiver window, the default stack will loop around in tcp
958.
by drewg123 5y ago
This is in really poor taste.
959.
by drewg123 5y ago
Kirk gives great talks, I highly recommend him as a speaker. His talks about the history of BSD are amazing. I love the story about how a box of tapes containing the highly optimized Vax VMS context switch code fell off a truck that was d
960.
by drewg123 5y ago
Wow, this is interesting. It seems to be scan of the front page of a variety of physical newspapers. I've bookemarked it.

More ›