12 ms·

Boosting upload speed and improving Windows' TCP stack

261 points by el_duderino 5y ago

chokeartist 5y ago
I got excited when I saw that fancy Microsoft Message Analyzer tool and wanted to try it out. Sadly it appears to be retired and removed by MSFT? Sad!
- jabroni_salad 5y ago
  It's really too bad. I'm happy enough to use Wireshark, but I liked that MMA could filter by PID.
- hyperrail 5y ago
  Yeah, I have no idea either why Microsoft would want to remove Message Analyzer completely, even if they could not maintain it. You can still download it through the Internet Archive: * 32-bit x86: https://web.archive.org/web/20191104120802/https://download.microsoft.com/download/2/8/3/283DE38A-5164-49DB-9883-9D1CC432174D/MessageAnalyzer.msi https://web.archive.org/web/20191104120802/https://download.... * 64-bit x86: https://web.archive.org/web/20190420141924/http://download.microsoft.com/download/2/8/3/283DE38A-5164-49DB-9883-9D1CC432174D/MessageAnalyzer64.msi https://web.archive.org/web/20190420141924/http://download.m... (those links via: https://www.reddit.com/r/sysadmin/comments/e4qocq/microsoft_message_analyzer_download_links_removed/f9f4fqv/ https://www.reddit.com/r/sysadmin/comments/e4qocq/microsoft_... ) Or use the even older Microsoft utility Network Monitor, which is still available on Microsoft's website: https://www.microsoft.com/en-us/download/details.aspx?id=4865 https://www.microsoft.com/en-us/download/details.aspx?id=486... Supposedly Microsoft is working on adding to the existing Windows Performance Analyzer (great GUI tool for ETW performance tracing) to display ETW packet captures, which will succeed Message Analyzer and Network Monitor: https://techcommunity.microsoft.com/t5/networking-blog/introducing-packet-monitor/bc-p/1853444/highlight/true#M318 https://techcommunity.microsoft.com/t5/networking-blog/intro...
  - slowstart 5y ago
    You are spot on. PktMon is the next generation tool in newer Windows 10 versions and brings many of the same benefits referred to in this blog - particularly being able to view packet captures and traces together in the same text file. And WPA is also very useful when analyzing performance problems.
tyingq 5y ago
Interesting. Is the Dropbox client still an obfuscated python app? I'm curious if they spawn new processes for simultaneous uploads since they probably aren't threading.
- Twisol 5y ago
  > On one hand, Dropbox Desktop Client has just a few settings. On the other, behind this simple UI lies some pretty sophisticated Rust code with multi-threaded compression, chunking, and hashing. On the lowest layers, it is backed up by HTTP/2 and TLS stacks. And I found another Dropbox blog post about rewriting their sync engine from Python to Rust: https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine https://dropbox.tech/infrastructure/rewriting-the-heart-of-o... But it isn't clear whether the outer shell of the app might still be Python.
- kevingadd 5y ago
  The Windows client I have installed appears to be a native app using QT 5 and QT5WebEngine (embedded chromium) with an absolutely bonkers number of threads (240). It's possible there's still python in there but I suspect not, their UI has been completely overhauled since the python days.
dijit 5y ago
I had a similar issue with Windows kernels "recently" (2016~?)... I don't have the memory or patience to write a long and inspiring blog post, but it comes down to: Even with IOCP/multiple threads: network traffic is single threaded in the kernel, even worse, there's a mutex there. Putting the effective limit on PPS for windows to something like 1.1M for 3.0GHz. The task of this machine was /basically/ a connection multiplexer with some TLS offloading; so listen on a socket, get an encrypted connection, check your connection pool and forward where appropriate. Our machine basically sat waiting (in kernel space) for this lock 99.7% of the time, 0.3% was spent on SSL handshaking.. We solved our "issue" by spreading such load over many more machines and gave them low-core-count high-clock-speed Xeons instead of the normal complement of 20vCPU Xeons. AFAIK that issue persists, I'd be interested to know if someone else managed to coerce windows to do the right thing here.
- kevingadd 5y ago
  Do you know whether it's a single thread for all network devices, or just per device? It would be interesting if this ended up being a driver level constraint or something that can be fixed by having multiple NICs in the machine.
  - dijit 5y ago
    it was LACP'd... buuuuuuut; it was only one PCIe card. Otherwise you can't easily do LACP because it can't be offloaded to the card. We tried without LACP, but again, only one PCIe card.
- jandrese 5y ago
  Did it have to be Windows? This is the sort of thing Linux or *BSD boxes are better suited for. I wouldn't even consider a Windows machine for the task unless there's some sort of licensed software you need to run on it to get the job done.
  - dijit 5y ago
    > This is the sort of thing Linux or *BSD boxes are better suited for. Definitely, though enabling conntrack on Linux has similar characteristics (forces single thread with some kind of internal mutex) though it can do 5x the b/w.. We tried having stateful firewalls in front of our windows boxen, that's how I know. Seems like Cloudflare has an older blog post detailing this too: https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/ https://blog.cloudflare.com/conntrack-tales-one-thousand-and... Anyway, to answer your question: AAA GameDev (and their backends, if highly tailored) are Windows.
thrdbndndn 5y ago
Cool article, but I'm not impressed by DropBox's upload speed on my Windows computer, at all. I just tested rn with DropBox, GoogleDrive, and OneDrive, all with their native desktop apps. I simply put a 300MB file in the folder and let it sync. DB: 500 KiB/s GD: 3 MiB/s OD: 11 MiB/s (my max bandwidth with 100Mbps) I don't know what causes the disparity here, but I have been annoyed by this for years, and it's the same across multiple computers I use at different locations. Another funny thing is if you just use the webpage, both GD and DB can reach 100Mbps easily. Edit: should mention Google's DriveFS can reach max speed too, but it's not available for my personal account (which uses the "Backup and sync for Google" app).
- drewg123 5y ago
  Is Google Drive using QUIC? If so, then its using the same BBR congestion algorithm as the bbr tcp stack, and BBR's algorithm which does not view loss as congestion will help a lot. It would be interesting to re-try the experiment on Linux or FreeBSD using BBR as the TCP stack and see if the results are any better for dropbox. FWIW, my corp openvpn is kinda terrible. My upload speeds via the vpn did not improve at all when I moved and upgraded from 10Mb/s to 1Gb/s upstream speeds. When I switched to BBR, my bandwidth went from ~8Mb/s -> 60Mbs, which I think is the limit of the corp vpn endpoint.
  - virtuallynathan 5y ago
    Looking at the flows on my network while uploading a file, it seems Google Drive's mac client just uses regular old TCP, same for the website.
  - leadbase0 5y ago
    QUIC is UDP, and TCP does not use CCA in userspace.
    - dochtman 5y ago
      QUIC does run in user space, and also uses congestion controllers running inside the QUIC stack, in user space. (I work on a QUIC implementation in Rust.)
brundolf 5y ago
Dropbox always publishes such good technical blog posts. And as a user, it's reassuring to see how much they still care about technical excellence.
- whatever_dude 5y ago
  Do they? I constantly see DropBox taking days to sync files that are 30kb on size. Or doing dumbfounding things like download all files, then re-upload all files when I set sync to "online only" on a folder if just one of the files is not set to online only. Maybe they have grand academic visions and papers, but I've been using them for well over a decade and I feel the client quality has gone downhill over the past few years. They keep adding unnecessary stuff like a redundant file browser while the core service suffers.
  - brundolf 5y ago
    Maybe my usage stays in the golden path, but I've been using them for ten years too and I have no complaints about the core functionality. My only real complaint is that they've been adding lots of features I don't care about, getting slightly pushy about convincing you to try them, etc. But I haven't seen the core stuff actually go downhill.
mrpippy 5y ago
API Monitor is really useful, but unfortunately is closed-source and hasn't been updated in a few years.
stephc_int13 5y ago
Is TCP the best choice? Why not UDP?
- willis936 5y ago
  It's an ideal application of TCP. Dropbox servers are continually flooded by traffic from clients, so the good congestion behavior from TCP is valuable. There is also less need to implement error detection/correction/retransmission in higher layers.
- jandrese 5y ago
  Bulk data transfer is TCP's bread and butter. This is the protocol living the dream.
- bob1029 5y ago
  This is a good question in my opinion. Theoretically, UDP would be the best choice if you had the time & money to spend on building a very application-specific layer on top that replicates many of the semantics of TCP. I am not aware of any apps that require 100% of the TCP feature set, so there is always an opportunity to optimize. You would essentially be saying "I know TCP is great, but we have this one thing we really prefer to do our way so we can justify the cost of developing an in-house mostly-TCP clone and can deal with the caveats of UDP". If you know your communications channel is very reliable, UDP can be better than TCP. Now, I am absolutely not advocating that anyone go out and do this. If you are trying to bring a product like Dropbox to market (and you don't have their budget), the last thing you want to do is play games with low-level network abstractions across thousands of potential client device types. TCP is an excellent fit for this use case.
- michaelmcmillan 5y ago
  And reimplement TCP on top? Would not recommend.
  - stephc_int13 5y ago
    Why not? I am not saying that it should be done from scratch. But most recent research done in the recent years about the protocols used on the web tend to be built on top of UDP and not TCP, for many historical reasons. In theory TCP would be the better choice, but in practice this is more complex than you assume. I think that many people have a knee-jerk reaction when talking about TCP vs UDP, but they probably don't know as much as they think... (parrots)
freerk 5y ago
How come Linux doesn't have this issue? Why did Microsoft had to fix TCP with the RACK-TLP RFC when both Linux and MacOS implementations did fine already?
- SaveTheRbtz 5y ago
  Microsoft Devs explain this in their "Algorithmic improvements boost TCP performance on the Internet"[1] article. TL;DR is that they had RACK (RFC draft) implemented as an MVP but w/o the reordering heuristic. [1] https://techcommunity.microsoft.com/t5/networking-blog/algorithmic-improvements-boost-tcp-performance-on-the-internet/ba-p/2347061 https://techcommunity.microsoft.com/t5/networking-blog/algor...
- the8472 5y ago
  The linux implementation already had rapid acknowledgements and tail loss probe for a long time. I think it was prototyped there by google.
  - slowstart 5y ago
    It's called R(ecent) Acknowledgement and yes the work came out of Google. This is the single biggest change to TCP loss recovery in a decade. It is now a Standards Track RFC: https://datatracker.ietf.org/doc/html/rfc8985 https://datatracker.ietf.org/doc/html/rfc8985. The Windows implementation was one of the earliest amongst a handful and Microsoft participated in the standardization.
lowleveldesign 5y ago
If you’re on recent Windows system, you should have pktmon [1] available. I believe it’s the „netsh trace” successor and has much nicer command line. And you no longer need an external tool to convert the trace to .npcap format. [1] https://docs.microsoft.com/en-us/windows-server/networking/technologies/pktmon/pktmon https://docs.microsoft.com/en-us/windows-server/networking/t...
- slowstart 5y ago
  PktMon is the next generation tool in newer Windows 10 versions and brings many of the same benefits referred to in this blog - particularly being able to view packet captures and traces together in the same text file.
  - SaveTheRbtz 5y ago
    Thanks! Added it back to the blogpost (seems like we accidentally lost the `pktmon` reference during the editing process =)
mwcampbell 5y ago
I wonder how the Dropbox developers managed to get in contact with the Windows core TCP team. Maybe I'm too cynical, but I'm surprised that Microsoft would go out of their way to work with a competitor like this.
- paxys 5y ago
  Microsoft is a massive and highly compartmentalized company. Windows kernel developers have no reason to see Dropbox as a competitor.
- toast0 5y ago
  Even if OneDrive vs Dropbox is important, this is a win for Windows in general. People will switch OSes because the TCP throughput is better on the other side; it's easy to measure and easy to compare and makes a nice item in a pros and cons list. Fixing something like this can help lots of use cases, but may have been difficult to spot, so I'm sure the Windows TCP team was thrilled to get the detailed, reproducible report.
rootsudo 5y ago
"Dropbox is used by many creative studios, including video and game productions. These studios’ workflows frequently use offices in different time zones to ensure continuous progress around the clock. " Honestly I don't understand these orgs that don't go OneDrive/O365 suite. What product value does dropbox have when competing within Microsoft's own ecosystem?
Agingcoder 5y ago
Excellent article ! I got hit by the exact same issue which is described in the fermilab paper, namely packet reordering caused by intel drivers. It took me several days to diagnose the problem. Interestingly enough, the problem virtually disappeared when running tcpdump, which, after a lot of reading on the innards of the linux TCP stack, and prodding with ebpf, eventually led me to conjecture that it was a scheduling/core placement issue. Pinning my process clearly made the problem disappear, and then finding the paper nailed it. Networks are not my specialty (I come from a math background, am self taught, and had always dismissed them as mere plumbing) , but I have to say that I came out of this difficult (for me) investigation with a great appreciation for networking in general, and now enjoy reading anything I can find about them. It's never too late to learn, and I have yet to find something in software engineering which is not interesting once you take a closer look at it!
- SaveTheRbtz 5y ago
  > tcpdump ... linux TCP stack ... ebpf > Networks are not my specialty I wish all network non-specialist were like you!
  - Natfan 5y ago
    I have nowhere near any of these skills, and I know I'm not a specialist.
- brohee 5y ago
  I think the problem disappearing while running tcpdump is one of the truest instances of Schrödingbug...
  - Agingcoder 5y ago
    It was perfectly reproducible. The original symptom was very low throughput, which is what prompted the investigation . Without tcpdump, low throughput and high reordering, with tcpdump, high throughput ( which is why I couldn't figure out what was going on). I'd be very interested if someone with kernel experience could tell me what's specific about tcpdump.
emmericp 5y ago
The real root cause for all that flow director mess and core balancing is that there's a huge disconnect between how the hardware works and what the socket API offers by default. The scaling model of the hardware is rather simple: hash over packet headers and assign a queue based on this. And each queue should be pinned to a core by pinning the interrupts, so you got easy flow-level scaling. That's called RSS. It's simple and effective. What it means is: the hardware decides which core handles which flow. I wonder why the article doesn't mention RSS at all? Now the socket API works in a different way: your application decides which core handles which socket and hence which flow. So you get cache misses if you don't tak into account how the hardware is hashing your flows. That's bad. So you can do some work-arounds by using flow director to explicitly redirect flows to cores that handle things but that's just not really an elegant solution (and the flow director lookup tables are small-ish). I didn't follow kernel development regarding this recently, but there should be some APIs to get a mapping from a connection tuple to the core it gets hashed to on RX (hash function should be standardized to Toeplitz IIRC, the exact details on which fields and how they are put into the function are somewhat hardware- and driver-specific but usually configurable). So you'd need to take this information into account when scheduling your connections to cores. If you do that you don't get any cache misses and don't need to rely on the limited capabilities of explicit per-flow steering. Note that this problem will mostly go away once TAPS finally replaces BSD sockets :)
- SaveTheRbtz 5y ago
  We didn't mention RSS/RPS in the post mostly because they are stable. (Albeit, relatively ineffective in terms of L2 cache misses.) FlowDirector, OTOH, breaks that stability and causes a lot of migrations, and hence a lot of re-ordering. Anyways, nice reference for TAPS! Fo those wanting to dig into it a bit more, consider reading an introductory paper (before a myriad of RFC drafts from the "TAPS Working Group"): https://arxiv.org/pdf/2102.11035.pdf https://arxiv.org/pdf/2102.11035.pdf PS. We went through most of our low-level web-server optimization for the Edge Network in an old blogpost: https://dropbox.tech/infrastructure/optimizing-web-servers-for-high-throughput-and-low-latency https://dropbox.tech/infrastructure/optimizing-web-servers-f...
tims33 5y ago
I appreciate seeing a support and engineering org going this deep to resolve this kind of issue. Normally this is the stuff you waste hours on with a support org only to get told to clear your cookies and cache one more time. In particular, the collaboration with Microsoft was great.I wonder what it took to make that happen.
Seattle3503 5y ago
Has Dropbox ever experimented with SCTP or other protocols that don't enforce strict ordering of packets? I know some middleboxes struggle with SCTP (they expect TCP or UDP), but in that case you do SCTP over UDP or have a fall back.
- SaveTheRbtz 5y ago
  Sadly, middleboxes are a real problem, esp. with our Enterprise customers. We had this problem even with HTTP/2 rollout so there is even a special HTTP/1.1-only mode in the Desktop Client for environments where h2 is disabled. In the future we are planing on having an HTTP/3 support which will give us pretty much the same benefits as SCTP with a better middlebox compatibility.
slowstart 5y ago
I lead the Windows TCP team. We blogged about recent TCP advancements which is very relevant: https://techcommunity.microsoft.com/t5/networking-blog/algorithmic-improvements-boost-tcp-performance-on-the-internet/ba-p/2347061 https://techcommunity.microsoft.com/t5/networking-blog/algor...
- the8472 5y ago
  Are equivalents to linux' BQL/AQL, fq_codel, TCP_NOTSENT_LOWAT in the pipeline?
  - slowstart 5y ago
    I cannot comment on queuing disciplines and limits in future products. Re. TCP_NOTSENT_LOWAT, you may want to look at the Ideal Send Backlog API that allows an application to have just more than BDP queued to keep the performance at the maximum throughput while minimizing the amount of data queued: https://docs.microsoft.com/en-us/windows/win32/winsock/sio-ideal-send-backlog-change https://docs.microsoft.com/en-us/windows/win32/winsock/sio-i...
    - the8472 5y ago
      Interesting. ISB does seem less ergonomic for blocking IO but it makes sense for overlapped requests.
- SaveTheRbtz 5y ago
  A couple of questions: * What are the reasons for disabling TCP timestamps by default? (If you can answer) will they be eventually enabled by default? (The reason I'm asking is that Linux uses TS field as storage for syncookies, and without it will drop WScale and SACK options greatly degrading Windows TCP perf in case of a synflood.[1]) * I've noticed "Pacing Profile : off" in the `netsh interface tcp show global` output. Is that the same as tcp pacing in fq qdisc[2]? (If you can answer) will it be eventually enabled by default? [1] https://elixir.bootlin.com/linux/v5.13-rc2/source/net/ipv4/syncookies.c#L24 https://elixir.bootlin.com/linux/v5.13-rc2/source/net/ipv4/s... [2] https://man7.org/linux/man-pages/man8/tc-fq.8.html https://man7.org/linux/man-pages/man8/tc-fq.8.html
ovebepari 5y ago
I would've said "it's not our problem to solve"

261 points by el_duderino 5y ago

I got excited when I saw that fancy Microsoft Message Analyzer tool and wanted to try it out. Sadly it appears to be retired and removed by MSFT? Sad!

jabroni_salad 5y ago
It's really too bad. I'm happy enough to use Wireshark, but I liked that MMA could filter by PID.
hyperrail 5y ago
Yeah, I have no idea either why Microsoft would want to remove Message Analyzer completely, even if they could not maintain it. You can still download it through the Internet Archive: * 32-bit x86: https://web.archive.org/web/20191104120802/https://download.microsoft.com/download/2/8/3/283DE38A-5164-49DB-9883-9D1CC432174D/MessageAnalyzer.msi https://web.archive.org/web/20191104120802/https://download.... * 64-bit x86: https://web.archive.org/web/20190420141924/http://download.microsoft.com/download/2/8/3/283DE38A-5164-49DB-9883-9D1CC432174D/MessageAnalyzer64.msi https://web.archive.org/web/20190420141924/http://download.m... (those links via: https://www.reddit.com/r/sysadmin/comments/e4qocq/microsoft_message_analyzer_download_links_removed/f9f4fqv/ https://www.reddit.com/r/sysadmin/comments/e4qocq/microsoft_... ) Or use the even older Microsoft utility Network Monitor, which is still available on Microsoft's website: https://www.microsoft.com/en-us/download/details.aspx?id=4865 https://www.microsoft.com/en-us/download/details.aspx?id=486... Supposedly Microsoft is working on adding to the existing Windows Performance Analyzer (great GUI tool for ETW performance tracing) to display ETW packet captures, which will succeed Message Analyzer and Network Monitor: https://techcommunity.microsoft.com/t5/networking-blog/introducing-packet-monitor/bc-p/1853444/highlight/true#M318 https://techcommunity.microsoft.com/t5/networking-blog/intro...
- slowstart 5y ago
  You are spot on. PktMon is the next generation tool in newer Windows 10 versions and brings many of the same benefits referred to in this blog - particularly being able to view packet captures and traces together in the same text file. And WPA is also very useful when analyzing performance problems.

tyingq 5y ago

Interesting. Is the Dropbox client still an obfuscated python app? I'm curious if they spawn new processes for simultaneous uploads since they probably aren't threading.

Twisol 5y ago
> On one hand, Dropbox Desktop Client has just a few settings. On the other, behind this simple UI lies some pretty sophisticated Rust code with multi-threaded compression, chunking, and hashing. On the lowest layers, it is backed up by HTTP/2 and TLS stacks. And I found another Dropbox blog post about rewriting their sync engine from Python to Rust: https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine https://dropbox.tech/infrastructure/rewriting-the-heart-of-o... But it isn't clear whether the outer shell of the app might still be Python.
kevingadd 5y ago
The Windows client I have installed appears to be a native app using QT 5 and QT5WebEngine (embedded chromium) with an absolutely bonkers number of threads (240). It's possible there's still python in there but I suspect not, their UI has been completely overhauled since the python days.

dijit 5y ago

I had a similar issue with Windows kernels "recently" (2016~?)... I don't have the memory or patience to write a long and inspiring blog post, but it comes down to: Even with IOCP/multiple threads: network traffic is single threaded in the kernel, even worse, there's a mutex there. Putting the effective limit on PPS for windows to something like 1.1M for 3.0GHz. The task of this machine was /basically/ a connection multiplexer with some TLS offloading; so listen on a socket, get an encrypted connection, check your connection pool and forward where appropriate. Our machine basically sat waiting (in kernel space) for this lock 99.7% of the time, 0.3% was spent on SSL handshaking.. We solved our "issue" by spreading such load over many more machines and gave them low-core-count high-clock-speed Xeons instead of the normal complement of 20vCPU Xeons. AFAIK that issue persists, I'd be interested to know if someone else managed to coerce windows to do the right thing here.

kevingadd 5y ago
Do you know whether it's a single thread for all network devices, or just per device? It would be interesting if this ended up being a driver level constraint or something that can be fixed by having multiple NICs in the machine.
- dijit 5y ago
  it was LACP'd... buuuuuuut; it was only one PCIe card. Otherwise you can't easily do LACP because it can't be offloaded to the card. We tried without LACP, but again, only one PCIe card.
jandrese 5y ago
Did it have to be Windows? This is the sort of thing Linux or *BSD boxes are better suited for. I wouldn't even consider a Windows machine for the task unless there's some sort of licensed software you need to run on it to get the job done.
- dijit 5y ago
  > This is the sort of thing Linux or *BSD boxes are better suited for. Definitely, though enabling conntrack on Linux has similar characteristics (forces single thread with some kind of internal mutex) though it can do 5x the b/w.. We tried having stateful firewalls in front of our windows boxen, that's how I know. Seems like Cloudflare has an older blog post detailing this too: https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/ https://blog.cloudflare.com/conntrack-tales-one-thousand-and... Anyway, to answer your question: AAA GameDev (and their backends, if highly tailored) are Windows.

thrdbndndn 5y ago

Cool article, but I'm not impressed by DropBox's upload speed on my Windows computer, at all. I just tested rn with DropBox, GoogleDrive, and OneDrive, all with their native desktop apps. I simply put a 300MB file in the folder and let it sync. DB: 500 KiB/s GD: 3 MiB/s OD: 11 MiB/s (my max bandwidth with 100Mbps) I don't know what causes the disparity here, but I have been annoyed by this for years, and it's the same across multiple computers I use at different locations. Another funny thing is if you just use the webpage, both GD and DB can reach 100Mbps easily. Edit: should mention Google's DriveFS can reach max speed too, but it's not available for my personal account (which uses the "Backup and sync for Google" app).

drewg123 5y ago
Is Google Drive using QUIC? If so, then its using the same BBR congestion algorithm as the bbr tcp stack, and BBR's algorithm which does not view loss as congestion will help a lot. It would be interesting to re-try the experiment on Linux or FreeBSD using BBR as the TCP stack and see if the results are any better for dropbox. FWIW, my corp openvpn is kinda terrible. My upload speeds via the vpn did not improve at all when I moved and upgraded from 10Mb/s to 1Gb/s upstream speeds. When I switched to BBR, my bandwidth went from ~8Mb/s -> 60Mbs, which I think is the limit of the corp vpn endpoint.
- virtuallynathan 5y ago
  Looking at the flows on my network while uploading a file, it seems Google Drive's mac client just uses regular old TCP, same for the website.
- leadbase0 5y ago
  QUIC is UDP, and TCP does not use CCA in userspace.
  - dochtman 5y ago
    QUIC does run in user space, and also uses congestion controllers running inside the QUIC stack, in user space. (I work on a QUIC implementation in Rust.)

brundolf 5y ago

Dropbox always publishes such good technical blog posts. And as a user, it's reassuring to see how much they still care about technical excellence.

whatever_dude 5y ago
Do they? I constantly see DropBox taking days to sync files that are 30kb on size. Or doing dumbfounding things like download all files, then re-upload all files when I set sync to "online only" on a folder if just one of the files is not set to online only. Maybe they have grand academic visions and papers, but I've been using them for well over a decade and I feel the client quality has gone downhill over the past few years. They keep adding unnecessary stuff like a redundant file browser while the core service suffers.
- brundolf 5y ago
  Maybe my usage stays in the golden path, but I've been using them for ten years too and I have no complaints about the core functionality. My only real complaint is that they've been adding lots of features I don't care about, getting slightly pushy about convincing you to try them, etc. But I haven't seen the core stuff actually go downhill.

mrpippy 5y ago

API Monitor is really useful, but unfortunately is closed-source and hasn't been updated in a few years.

stephc_int13 5y ago

Is TCP the best choice? Why not UDP?

willis936 5y ago
It's an ideal application of TCP. Dropbox servers are continually flooded by traffic from clients, so the good congestion behavior from TCP is valuable. There is also less need to implement error detection/correction/retransmission in higher layers.
jandrese 5y ago
Bulk data transfer is TCP's bread and butter. This is the protocol living the dream.
bob1029 5y ago
This is a good question in my opinion. Theoretically, UDP would be the best choice if you had the time & money to spend on building a very application-specific layer on top that replicates many of the semantics of TCP. I am not aware of any apps that require 100% of the TCP feature set, so there is always an opportunity to optimize. You would essentially be saying "I know TCP is great, but we have this one thing we really prefer to do our way so we can justify the cost of developing an in-house mostly-TCP clone and can deal with the caveats of UDP". If you know your communications channel is very reliable, UDP can be better than TCP. Now, I am absolutely not advocating that anyone go out and do this. If you are trying to bring a product like Dropbox to market (and you don't have their budget), the last thing you want to do is play games with low-level network abstractions across thousands of potential client device types. TCP is an excellent fit for this use case.
michaelmcmillan 5y ago
And reimplement TCP on top? Would not recommend.
- stephc_int13 5y ago
  Why not? I am not saying that it should be done from scratch. But most recent research done in the recent years about the protocols used on the web tend to be built on top of UDP and not TCP, for many historical reasons. In theory TCP would be the better choice, but in practice this is more complex than you assume. I think that many people have a knee-jerk reaction when talking about TCP vs UDP, but they probably don't know as much as they think... (parrots)

freerk 5y ago

How come Linux doesn't have this issue? Why did Microsoft had to fix TCP with the RACK-TLP RFC when both Linux and MacOS implementations did fine already?

SaveTheRbtz 5y ago
Microsoft Devs explain this in their "Algorithmic improvements boost TCP performance on the Internet"[1] article. TL;DR is that they had RACK (RFC draft) implemented as an MVP but w/o the reordering heuristic. [1] https://techcommunity.microsoft.com/t5/networking-blog/algorithmic-improvements-boost-tcp-performance-on-the-internet/ba-p/2347061 https://techcommunity.microsoft.com/t5/networking-blog/algor...
the8472 5y ago
The linux implementation already had rapid acknowledgements and tail loss probe for a long time. I think it was prototyped there by google.
- slowstart 5y ago
  It's called R(ecent) Acknowledgement and yes the work came out of Google. This is the single biggest change to TCP loss recovery in a decade. It is now a Standards Track RFC: https://datatracker.ietf.org/doc/html/rfc8985 https://datatracker.ietf.org/doc/html/rfc8985. The Windows implementation was one of the earliest amongst a handful and Microsoft participated in the standardization.

lowleveldesign 5y ago

If you’re on recent Windows system, you should have pktmon [1] available. I believe it’s the „netsh trace” successor and has much nicer command line. And you no longer need an external tool to convert the trace to .npcap format. [1] https://docs.microsoft.com/en-us/windows-server/networking/technologies/pktmon/pktmon https://docs.microsoft.com/en-us/windows-server/networking/t...

slowstart 5y ago
PktMon is the next generation tool in newer Windows 10 versions and brings many of the same benefits referred to in this blog - particularly being able to view packet captures and traces together in the same text file.
- SaveTheRbtz 5y ago
  Thanks! Added it back to the blogpost (seems like we accidentally lost the `pktmon` reference during the editing process =)

mwcampbell 5y ago

I wonder how the Dropbox developers managed to get in contact with the Windows core TCP team. Maybe I'm too cynical, but I'm surprised that Microsoft would go out of their way to work with a competitor like this.

paxys 5y ago
Microsoft is a massive and highly compartmentalized company. Windows kernel developers have no reason to see Dropbox as a competitor.
toast0 5y ago
Even if OneDrive vs Dropbox is important, this is a win for Windows in general. People will switch OSes because the TCP throughput is better on the other side; it's easy to measure and easy to compare and makes a nice item in a pros and cons list. Fixing something like this can help lots of use cases, but may have been difficult to spot, so I'm sure the Windows TCP team was thrilled to get the detailed, reproducible report.

rootsudo 5y ago

"Dropbox is used by many creative studios, including video and game productions. These studios’ workflows frequently use offices in different time zones to ensure continuous progress around the clock. " Honestly I don't understand these orgs that don't go OneDrive/O365 suite. What product value does dropbox have when competing within Microsoft's own ecosystem?

Agingcoder 5y ago

Excellent article ! I got hit by the exact same issue which is described in the fermilab paper, namely packet reordering caused by intel drivers. It took me several days to diagnose the problem. Interestingly enough, the problem virtually disappeared when running tcpdump, which, after a lot of reading on the innards of the linux TCP stack, and prodding with ebpf, eventually led me to conjecture that it was a scheduling/core placement issue. Pinning my process clearly made the problem disappear, and then finding the paper nailed it. Networks are not my specialty (I come from a math background, am self taught, and had always dismissed them as mere plumbing) , but I have to say that I came out of this difficult (for me) investigation with a great appreciation for networking in general, and now enjoy reading anything I can find about them. It's never too late to learn, and I have yet to find something in software engineering which is not interesting once you take a closer look at it!

SaveTheRbtz 5y ago
> tcpdump ... linux TCP stack ... ebpf > Networks are not my specialty I wish all network non-specialist were like you!
- Natfan 5y ago
  I have nowhere near any of these skills, and I know I'm not a specialist.
brohee 5y ago
I think the problem disappearing while running tcpdump is one of the truest instances of Schrödingbug...
- Agingcoder 5y ago
  It was perfectly reproducible. The original symptom was very low throughput, which is what prompted the investigation . Without tcpdump, low throughput and high reordering, with tcpdump, high throughput ( which is why I couldn't figure out what was going on). I'd be very interested if someone with kernel experience could tell me what's specific about tcpdump.

emmericp 5y ago

The real root cause for all that flow director mess and core balancing is that there's a huge disconnect between how the hardware works and what the socket API offers by default. The scaling model of the hardware is rather simple: hash over packet headers and assign a queue based on this. And each queue should be pinned to a core by pinning the interrupts, so you got easy flow-level scaling. That's called RSS. It's simple and effective. What it means is: the hardware decides which core handles which flow. I wonder why the article doesn't mention RSS at all? Now the socket API works in a different way: your application decides which core handles which socket and hence which flow. So you get cache misses if you don't tak into account how the hardware is hashing your flows. That's bad. So you can do some work-arounds by using flow director to explicitly redirect flows to cores that handle things but that's just not really an elegant solution (and the flow director lookup tables are small-ish). I didn't follow kernel development regarding this recently, but there should be some APIs to get a mapping from a connection tuple to the core it gets hashed to on RX (hash function should be standardized to Toeplitz IIRC, the exact details on which fields and how they are put into the function are somewhat hardware- and driver-specific but usually configurable). So you'd need to take this information into account when scheduling your connections to cores. If you do that you don't get any cache misses and don't need to rely on the limited capabilities of explicit per-flow steering. Note that this problem will mostly go away once TAPS finally replaces BSD sockets :)

SaveTheRbtz 5y ago
We didn't mention RSS/RPS in the post mostly because they are stable. (Albeit, relatively ineffective in terms of L2 cache misses.) FlowDirector, OTOH, breaks that stability and causes a lot of migrations, and hence a lot of re-ordering. Anyways, nice reference for TAPS! Fo those wanting to dig into it a bit more, consider reading an introductory paper (before a myriad of RFC drafts from the "TAPS Working Group"): https://arxiv.org/pdf/2102.11035.pdf https://arxiv.org/pdf/2102.11035.pdf PS. We went through most of our low-level web-server optimization for the Edge Network in an old blogpost: https://dropbox.tech/infrastructure/optimizing-web-servers-for-high-throughput-and-low-latency https://dropbox.tech/infrastructure/optimizing-web-servers-f...

tims33 5y ago

I appreciate seeing a support and engineering org going this deep to resolve this kind of issue. Normally this is the stuff you waste hours on with a support org only to get told to clear your cookies and cache one more time. In particular, the collaboration with Microsoft was great.I wonder what it took to make that happen.

Seattle3503 5y ago

Has Dropbox ever experimented with SCTP or other protocols that don't enforce strict ordering of packets? I know some middleboxes struggle with SCTP (they expect TCP or UDP), but in that case you do SCTP over UDP or have a fall back.

SaveTheRbtz 5y ago
Sadly, middleboxes are a real problem, esp. with our Enterprise customers. We had this problem even with HTTP/2 rollout so there is even a special HTTP/1.1-only mode in the Desktop Client for environments where h2 is disabled. In the future we are planing on having an HTTP/3 support which will give us pretty much the same benefits as SCTP with a better middlebox compatibility.

slowstart 5y ago

I lead the Windows TCP team. We blogged about recent TCP advancements which is very relevant: https://techcommunity.microsoft.com/t5/networking-blog/algorithmic-improvements-boost-tcp-performance-on-the-internet/ba-p/2347061 https://techcommunity.microsoft.com/t5/networking-blog/algor...

the8472 5y ago
Are equivalents to linux' BQL/AQL, fq_codel, TCP_NOTSENT_LOWAT in the pipeline?
- slowstart 5y ago
  I cannot comment on queuing disciplines and limits in future products. Re. TCP_NOTSENT_LOWAT, you may want to look at the Ideal Send Backlog API that allows an application to have just more than BDP queued to keep the performance at the maximum throughput while minimizing the amount of data queued: https://docs.microsoft.com/en-us/windows/win32/winsock/sio-ideal-send-backlog-change https://docs.microsoft.com/en-us/windows/win32/winsock/sio-i...
  - the8472 5y ago
    Interesting. ISB does seem less ergonomic for blocking IO but it makes sense for overlapped requests.
SaveTheRbtz 5y ago
A couple of questions: * What are the reasons for disabling TCP timestamps by default? (If you can answer) will they be eventually enabled by default? (The reason I'm asking is that Linux uses TS field as storage for syncookies, and without it will drop WScale and SACK options greatly degrading Windows TCP perf in case of a synflood.[1]) * I've noticed "Pacing Profile : off" in the `netsh interface tcp show global` output. Is that the same as tcp pacing in fq qdisc[2]? (If you can answer) will it be eventually enabled by default? [1] https://elixir.bootlin.com/linux/v5.13-rc2/source/net/ipv4/syncookies.c#L24 https://elixir.bootlin.com/linux/v5.13-rc2/source/net/ipv4/s... [2] https://man7.org/linux/man-pages/man8/tc-fq.8.html https://man7.org/linux/man-pages/man8/tc-fq.8.html

ovebepari 5y ago

I would've said "it's not our problem to solve"