4 ms·
This work is really cool and is attacking an important problem in a lot of datacenter networks. It's also great that they've released the source code too. As
by vrv 15y ago
This work is really cool and is attacking an important problem in a lot of datacenter networks. It's also great that they've released the source code too. As someone who's worked on this problem, I encourage those interested in reading up more about this.
Here are a few papers on the problem in chronological order:
1. Original paper briefly talking about the "incast" TCP problem in storage environments: http://portal.acm.org/citation.cfm?id=1049998 http://portal.acm.org/citation.cfm?id=1049998
2. Our follow up work on that problem from a few years back: Measurement paper ( http://portal.acm.org/citation.cfm?id=1364825 http://portal.acm.org/citation.cfm?id=1364825 ) and our initial solution ( http://portal.acm.org/citation.cfm?id=1592604 http://portal.acm.org/citation.cfm?id=1592604 ) with a microsecond retransmission Linux patch here: https://github.com/vrv/linux-microsecondrto https://github.com/vrv/linux-microsecondrto
3. Another paper talking about Incast in Datacenter environments, focusing on a different form of the workload: http://portal.acm.org/citation.cfm?id=1592693 http://portal.acm.org/citation.cfm?id=1592693
4. RAMCloud - a project that briefly talks about the need for low-latency transports: http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf
5. ICTCP - paper from last year that tries to solve the Incast problem using receiver advertised window algorithms: http://conferences.sigcomm.org/co-next/2010/CoNEXT_papers/13-Wu.pdf http://conferences.sigcomm.org/co-next/2010/CoNEXT_papers/13...
DCTCP tries to go beyond solving the Incast problem and focuses on trying to control buffer occupancy in datacenter environments that contain both long flows and bursty flows.
I think these papers all assume lossy link layers (Ethernet), but there are standards and other technologies (Datacenter Ethernet, Myrinet, Infiniband) that aim for lossless link layers to make the transport problem easier, but come with various other drawbacks today (cost, compatibility, etc.). In the meantime, I hope DCTCP or the microsecond TCP patch prove useful in solving some of these problems.
- xtacy 15y agoI would like to hear more from application programmers here, about their experience in dealing with incast. For example, Facebook uses UDP to throttle messages at the application layer to combat incast: https://www.facebook.com/note.php?note_id=39391378919 https://www.facebook.com/note.php?note_id=39391378919. Also, a few catches with DCTCP: 1. It's not completely an end host solution and requires ECN support from switches, which should be widely available. Can someone pitch in about the availability of ECN in their networks? 2. DCTCP and plain TCP don't mix well, and hence not incrementally deployable. DCTCP and TCP+ECN also don't mix well!
- vrv 15y agoI'll let others chime in, but the DCTCP paper mentions "our application reduces the amount of data each worker sends and employs jitter. Facebook, reportedly, has gone to the extent of developing their own UDP-based congestion control [29]." I recently talked with someone on the Facebook memcached team about this problem and they mentioned that moving to UDP has been useful, but if I'm not mistaken, I believe they gave up on 100% in-order reliability in return for low-latency.