5 ms·
Author here. AMA
by drewg123 4y ago
Author here. AMA
- ManWith2Plans 4y agoDo you have a link to video or audio for this presentation? I'm probably don't speak for just myself when I say I would love to see it.
- quux 4y agoSomeone else linked the video here: https://news.ycombinator.com/item?id=32520750 https://news.ycombinator.com/item?id=32520750
- Bluecobra 4y agoWhat kind of tuning is done in the BIOS? Is that profile available to view to everyone? Are you using a custom BIOS from Dell?
- drewg123 4y agoNot much tuning needed to be done. The little that was is mentioned in the talk, and was basically to set NPS=1 and to disable DLWM, in order to be able to access the full xgmi interconnect bandwidth at all times, even when the CPU is not heavily loaded.
- throw0101c 4y agoYou're using ConnectX-6 Dx here. Any technical reason for that particular NIC, or just haven't gotten around to ConnectX-7s yet? Have you examined other NIC vendors? (Chelsio?)
- drewg123 4y agoThis talk was given roughly 4 months ago. CX7 was not available yet. I'm looking forward to testing on them when we get some. We looked at Chelsio (as T6 was available well before CX6-DX). However, the CX6-DX offers a killer feature not available on T6. The CX6-DX can remember the crypto state of any in-order stream, while the T6 cannot. That means that the TCP stack can send, say, 4K of a TLS record, wait for acks, and come back 40ms later and send the next 4K and DMA just the requested 4K from the host. The T6 cannot remember the state, and would need to DMA the first 4K (which was already sent) in order to re-establish the crypto state, and then DMA the requested 4K. This could run the PCIe bus out of bandwidth. The alternative is to make TCP always chunk sends at the TLS record size, but this was horrible for streaming quality.
- deleted 4y ago[deleted]
- sophacles 4y agoEvery iteration of this prezzo I've seen over the years has made for a fascinating morning read, thanks! As much as I enjoy the results of the work, I'm always a bit curious how the sausage is made. Is pushing the hardware limits your primary job or something you do periodically? How do you go about selecting the gear you use? How much do you work with the vendors? (etc etc) I'd really enjoy a behind the scenes blog post or something wrt this serving absurd amounts of traffic from a single box.
- drewg123 4y agoMy role is to make our CDN servers more efficient. One of the easiest and most fun ways to do that is to push servers as hard as I can and see what breaks and what doesn't scale. I also work with our hardware team and their vendors to evaluate new hardware and how it can fit into our system. But I do plenty of other things as well, including fixing random kernel bugs. You can read the git log of the FreeBSD main branch to see some of the things I've been working on..
- gopaz 4y agoWhat about the storage? Is it using Raid? Does blocksize matter? What filsystem is used?
- drewg123 4y agoEvery storage device is independent (no RAID), and runs UFS. We use UFS because, unlike ZFS, it integrates directly with the kernel page cache.
- ComputerGuru 4y agoWhich is part of the reason sendfile doesn’t work well for ZFS.
- phantomathkg 4y ago> Serve only static media files This part I don't get. How about DRM? Unless Netflix pre-DRM all contents for all user?
- Bluecobra 4y agoI would think that media files would be already encrypted and gets decrypted by the Netflix client. Otherwise the DRM could easily be defeated by using something like Wireshark.
- drewg123 4y agoYes, all our content is also DRMed. Else somebody could easily pirate content..
- onedr0p 4y agoTo be fair, it already seems easily pirated. DRM is useless, if content is able to be viewed on some personal device it can be ripped and shared. I'd be curious how much effort/money companies dump into adding DRM measures, it seems like a lost cause. Maybe it just makes the execs sleep better at night.
- Mindwipe 4y agoThis is a myth. Netflix's DRM is sufficiently good that the Reddit Piracy subreddit has spent the last three months moaning that they have no access to 4K Netflix rips, at least for weeks or months after the content comes out. Netflix's DRM and key management systems do what they care about pretty well at this point, which is protect the initial airing of popular shows.
- onedr0p 4y ago1080p rips are found within hours of the content first airing on NetFlix. You don't need to download Linux ISOs in full quality to enjoy them.
- 4y ago
- bri3d 4y agoAt what point does it make sense to replace the CPU and OS with custom hardware and software? At this point the CPU is basically doing TCP state maintenance and DMA supervision, but not much else, right? I totally get the cost, convenience, and supply chain risk-value in commodity stuff that you can just go out and buy, but once you're bound to a single network card, this advantage starts to go away, and it seems like you're fighting with the entire system topology when it comes to NUMA, no? Why not a "TCP file send accelerator" instead of a whole computer?
- wmf 4y agoI suppose you could attach NVMe drives directly to Bluefield and cut out x86.
- pyrolistical 4y agoWhen are you going to cut the CPU/main memory out completely? The bottleneck is at your NIC anyways, so seems like there would be a market for NIC that can directly read from disk into NIC's working memory
- drewg123 4y agoWe've looked at this. The problem is that NICs want to read in TCP MSS size chunks (1448 bytes, for example), while storage devices are highly optimized for block-aligned (4K) chunks. So you need to buffer the storage reads someplace, and for now the only practical answer is host memory. There are NVME technologies that could help, but they are either too small, or come at too large of a price premium. CXL memory looks promising, but its not ready yet.
- Matthias247 4y agoDoes it? I thought with segmentation offloads the NIC basically gets TCP stream data in more or less arbitrary sizes, and then segments in into MTU sizes on its own?
- drewg123 4y agoWe do fairly sophisticated TCP pacing, which requires sending down some small multiple of MSS to the NIC, so it doesn't always have the freedom to pull 4K at a time.
- deleted 4y ago[deleted]
- sam0x17 4y ago> Sendfile Ah, so this is why everything stutters / falls apart when you switch subtitles on or off -- it has to access a whole different file and resume at the same place in that file I assume? I would think you would want the (verbal) audio separated out in a different file so it can be swapped out on the fly without re-initializing the video stream, and same thing with subtitle files? I'm just making some assumptions based on the behavior I've seen but would be cool to know how this works.
- w10-1 4y agoThank you very much "drewg123"! Future technology advances increasingly looks like this complex work integrating hardware, OS fixes, team collaboration. People and teams and companies working together, and contributing to shared resources like FreeBSD. Tolerating mistakes at scale, giving credit where credit is due, and all the other things that make respect real, which creates a space to get things done. Most of us will never get close to these opportunities or contexts, but still it helps us advance our own technique/culture to observe and model your story. And perhaps you'll help new collaborators find you. All the best.
- nh2 4y agoYou mention AIO in nginx. In 2021 somebody submitted a patch for io_uring support in nginx: https://mailman.nginx.org/pipermail/nginx-devel/2021-February/013860.html https://mailman.nginx.org/pipermail/nginx-devel/2021-Februar... I'm not sure if there has been further progress on it so far. In one comment feedback is "it doesn't seem to make the typical nginx use case much faster" [at that time]. But I find this interesting, because io_uring can make almost all things async that can't be used async so far in Linux (open(), stat(), etc) and thus in nginx. Would io_uring integration in nginx be relevant for you?