9 ms·

The hidden cost of QUIC and TOU (2016)

102 points by luu 8y ago

dylz 8y ago
Here's the problem: other companies ruined it for the good ISPs. They destroyed their trust by screwing with end user transfers, tampering with TLS, tampering with connections, modifying resources in transit, MITMing in horrific fashion, invading privacy, and breaking protocols for profit. Arguably there would be a lot less insane push to make everything a blackbox if companies didn't keep doing dumb crap for an extra buck. See also the TLSv1.3 middlebox arguments. I certainly enjoy the fact that when using an AT&T hotspot I can't reach certain websites over TLS, with Verizon they would append advertising tracking headers to requests on their end, on Sprint if I wget the same image multiple times it'll be in a different quality and not match md5-wise each time, all of the above do DPI and negatively degrade web traffic no matter what port it runs on, TCP connections fail if I try to run non-web traffic over port 80 randomly, TCP connections are randomly killed if long lived.
- CobrastanJorji 8y ago
  Exactly. Would the Internet be way easier to design and use if we could assume that everybody is a good actor? Heck yeah! It'd be great. There are some really useful things you can do when you're allowed to intercept the conversations between two people. Caching, debugging, fixing spelling errors, etc. But we can't assume that the folks between me and you are good, and sharing anything at all with them is a bad idea. Does that mean a whole bunch of really useful techniques are harder or straight up impossible now? Yes, absolutely, and we should plan ahead and make sure to come up with ways to debug issues like this. But stepping back on security and privacy because it makes it harder to benevolently spy? Screw that noise.
- caf 8y ago
  Here's the problem: other companies ruined it for the good ISPs. There's a phrase for that situation: "Poisoning the Well".
  - TeMPOraL 8y ago
    How's it called when some companies are run by good humans, and other by evil reptilians, and the reptilians poison the well with something they can drink, but humans can't?
    - felipemnoa 8y ago
      terraforming
- xyzzy123 8y ago
  The problem I have with TLS 1.3 proposal is it’s gonna stop me having control over the things that go out of my own network :( Realise that this is a multifaceted debate but it’s going to destroy the utility of anything I can’t install a certificate on, and also force me to install certs on everything I own before I can comfortably let those things on the network. It’s gonna screw a lot of enterprise use cases as well. I’m not sure people fully get that the privacy extended to say “dissidents in Syria” is also going to apply to HP printers on their own networks trying to figure out whether to show “toner low” dialogs. Personally I don’t think the “hey the dissidents” value is worth it since those people are pretty screwed anyway - filtering at scale can still work out what you’re connecting to (ips, latency, response size patterns, blah blah) but it really messes up anyone (person or enterprise) who wants to use stuff but also know what’s going on.
  - pilif 8y ago
    >also force me to install certs on everything I own before I can comfortably let those things on the network I'm curious: How is TLS 1.3 different from TLS 1.2 in that regard? How would you implement an intercepting proxy with TLS 1.2 and without installing a CA cert in the clients?
    - ameshkov 8y ago
      I guess the encrypted SNI draft is what's meant here.
      - xyzzy123 8y ago
        Thanks, sorry, you are totally right and I should be more specific. Yes. It’s a metadata leak and generally I am pro the end-to-end principle but encrypted SNI actually forces everyone to MITM. Whether that is good or bad is a value judgement but for people who have been doing “light touch” egress filtering it is a huge PITA. It is actually going to force more invasive surveillance in basically any regulated workplace.
        8y ago
- dstroot 8y ago
  > for the good ISPs... and those are? Do they actually exist?
  - dylz 8y ago
    Some do. Monkeybrains in SFBA and AAISP, Bogons in UK are a few possible examples?
ocdtrekkie 8y ago
You know what a lot of middleboxes do? They block ads/malware. Shocking that the two largest ad companies are trying to push standards which break things that block their ads.
- pilif 8y ago
  This isn’t as much about stopping middleboxes on your perimeter that you fully control, but a lot about middleboxes between you and the destination that are outside of your or the destinations control. Those are often used to add ads rather than block them. If you yourself want to block ads, do so on your machine (where traffic has already been decrypted) or on your router (which will then decrypt traffic for you and re-encrypt it with its certificate that you have added to your machine).
- icebraining 8y ago
  That's news to me, I've never heard of a widespread use of middleboxes for blocking ads. Which boxes are these and how did you know that there are a lot of them deployed?
  - ocdtrekkie 8y ago
    So, an example of a middlebox that's exceedingly common is a "web security gateway", which is your average web filter and logger in a corporate environment. Obviously it logs employee web activity, blocks access to adult websites, and maintains it's own malware definitions to try and block malicious content as well. It's quite often for these to also block domains used by ads and popups by default. When these sorts of devices are configured to inspect HTTPS, this adds a significant additional complexity: Network PCs need to be configured to permit a certificate from the box for all domains, which intercepts, decrypts, and re-encrypts all traffic. Of course, the same type of technology a corporation might use to manage their network could be used by a state actor or a hostile ISP.
    - brazzledazzle 8y ago
      If you’re only using it to block domains you can do that at the DNS level with the added bonus of it being more efficient.
quickben 8y ago
When advertisement companies push for L4 redesign... This is going to be a fun one to watch how it plays out.
nicenewsbc 8y ago
There is no combination between them.
tinus_hn 8y ago
It would be nice if these protocols could be easily decrypted using a key available on the client. Other than that, tough luck for the ‘transparent’ proxies and friends.
- hornetblack 8y ago
  I know that for https in NSS browsers (Chrome and FF) you can dump the secrets to a file that Wireshark can read. Given that Chrome is currently the biggest user i would be supprised if that was easy to update.
  - nouseforaname 8y ago
    I believe chrome currently uses BoringSSL not NSS, but still has some facilities for writing the same keylogfile.
- tialaramex 8y ago
  Of necessity both client and server have the session keys. In principle it would be possible for the client to lack keys needed for the server to read data sent by the client, and vice versa, but in practice this is never done. Under Forward Secrecy a Middlebox must learn fresh session keys for every connection or it can't decrypt it. Both clients (e.g. Firefox) and servers (e.g. using Java or OpenSSL) have facilities to dump the session keys out somewhere, and this is adequate for debugging although obviously you will need to acquire new skills if you're used to being able to get stuff done with a paperclip and a copy of tcpdump. At scale this get hard, arguably that's fine because a minute ago we said we wanted this for "debugging" but people who got their foot in the door with a "debugging" argument often actually want to decrypt everything, always, and so they're unhappy about this. If you don't want Forward Secrecy you have two options. Firstly, when the specification says to think of a random number for the key exchange protocol, you can always pick the same number, or a number chosen in some predictable fashion, the Middlebox can know this number (or method for predicting it) and then it can snoop as normal. This works in TLS 1.3, obviously it weakens your security (if bad guys learn how to predict the numbers you are screwed) but that's your choice. Secondly you could use a key exchange process that doesn't have any Forward Secrecy by design, such as the RSA key exchange from SSL that's grandfathered into TLS 1.0 through 1.2. In this case you just give the server's private RSA key to the middleboxes and they can decrypt everything. As you may notice in all the above scenarios, this is very bad for your security. But if "debugging" is really the problem that's almost certainly acceptable to you.
olliej 8y ago
The other fun bit is the “encryption” of web sockets (outside of tls, etc). Basically (hand waving here) a websocket connection starts with a 64 bit (or 128, I really can’t remember anymore) value, and then essentially just xors that over all subsequent data. This isn’t needed for security of the user or the server. It’s because so many middleware boxes are so poorly build that you could make them crash and/or get code execution if you had sufficient control over enough of the right payload bytes. Java applets exposed the exact same problem, so given they existed a decade before websockets and yet the middleware boxes were still broken enough that this nonsense was required in the websocket spec should tell you everything you need to know.
- Matthias247 8y ago
  That's not only about Middleboxes (which mostly act up to L4). Most likely it's more about preventing JS from attacking arbitrary servers (e.g. broken mail servers, file servers, etc) that might have an open TCP port. The "encryption" at least prevents JS attacks from being deterministic.
  - olliej 8y ago
    no, it was specifically due to broken middleware that had existing well known yet unfixed flaws.
  - aidenn0 8y ago
    It was about Application layer middleboxes (specifically broken http proxies).
- jorangreef 8y ago
  And now, even WebSockets over TLS need servers to waste CPU on XOR masking.
  - jefftk 8y ago
    XOR masking is ridiculously cheap compared to doing anything over the network or running js in general.
denormalfloat 8y ago
Alright, so existing debugging tools don't work with QUIC. We will need to make some new tools that can expose the information we need. If the hops between the source and destination are willing to expose the info, (and we can assume they do, as the author has), then we can figure out what packets go in that never come back out. Instead of say "this won't work because ____", why don't people say "it would work if we could ____"? Someone (or some company) needs to improve the Internet, and it seems like the world just harangues them for their effort.
cm2187 8y ago
Actually it raises an important point I hadn't thought about. By moving the transport protocol from layer 4 to layer 5, google is taking it off the hand of the OS. I understand why they do it (easier than to get all 3 or 4 major OS to implement it), but there is another cost in term of inter-operability. It means that every software consuming QUIC needs to have its own implementation of the protocol. It means every language in which you write that software needs to have libraries available implementing QUIC, or that the libraries you consume must themselves have a QUIC implementation. That's introducing a non insignificant inefficiency if QUIC becomes prevalent.
- tialaramex 8y ago
  The operating system can offer QUIC on top of UDP the same way it offers UDP on top of IP. BSD sockets aren't necessarily the ideal way to use QUIC but there's no reason you couldn't use them. Researchers have even repeatedly built Linux protocol modules that do TLS, either all of it, or the encrypted record layer (so the bulk but not the tricky negotiation decisions at the start). There's just a new TLS protocol you ask for instead of TCP and then the kernel handles encryption and so on.
deleted 8y ago
[deleted]
jsnell 8y ago
There's been an attempt to fix this during the last two years, as the QUIC standardization talks really got going. A bunch of operator people expressed a need for some sort of in-path measurability, while the privacy people have expressed the need to avoid any sort of session linkage or other forms of information leakage. The most viable compromise proposal seems to be the spin bit[0] which gives RTT measurements, but it's not agreed whether a version of that will make it to the first release. [0] https://quicwg.org/base-drafts/draft-ietf-quic-spin-exp.html https://quicwg.org/base-drafts/draft-ietf-quic-spin-exp.html
- pdkl95 8y ago
  > The most viable compromise proposal seems to be the spin bit That's utterly useless. If the "spin bit" becomes widely used, I intend to write a trivial patch that sets it randomly on each packet. If packets could be dropped unless the bit is set to specific values at specific times, too much session information is leaking to middleboxen. More likely, the state of the bit doesn't matter so setting it randomly will discourage wasting a bit in the protocol with this kind of nonsense in the future.
sly010 8y ago
Arguably you wouldn't have to debug bad connections if the middleboxes just did what they were supposed to.
vinay_ys 8y ago
This argument is broken. An ISP engineer debugging need only look at IP packet drops in their part of the network. If their end clients are asking them to debug an issue then they should debug it at the end points, not in the middle. This attitude is what lead them to stick more e and more buffers in the middle and do other shaping devices in the name of problem solving or adding value and ended up ossifying protocols. I think endpoints will evolve to better protocols to solve their own problems if the ISPs stopped putting hacks (like deep buffers) in their networks.
- jsnell 8y ago
  > This argument is broken. An ISP engineer debugging need only look at IP packet drops in their part of the network. That's just not true in practice, pretty much on any level. First, you need to look at a lot more things than just packet drops (e.g. reordering, queue buildup, corruption). Second, even getting full visibility into your own network is highly non-trivial since nobody has active probes on every link. My experience is that arranging for packet captures from an arbitrary point in the network could take a week. And if you guessed wrong about which node was at fault, you'd need to do it again in a binary search pattern. Third, you absolutely do need to know about things other than your own network. Otherwise you don't even know whether there is a problem you can solve. If the bottleneck is in the server, or the client, or the transit links, there is no point in debugging the core or the access network. > If their end clients are asking them to debug an issue then they should debug it at the end points, not in the middle. The endpoints are not going to be available. Do you think that Youtube is going to give an operator some kind of server access or even insight to the traffic? Do you really want to see a world where a customer having a complaint needs to first root their phone and install packet capturing software? What you're really saying here is that no problem should ever be debugged, and we should just hope that the network doesn't break. And hope is not much of a strategy.
ncmncm 8y ago
When you tunnel your protocol in UDP, and control both ends, you can get overwhelmingly better flow control than TCP, which cannot trust the other end, so must rely on packet drops to get a reliable signal. When you can trust the other end, rate of change of packet transfer time (delta packet delay) reveals congestion exactly -- i.e., increasing time means you had better slow down, decreasing, you can go faster. Only problem is, the receiver has the signal, but the sender needs it, and the useful lifetime is less than the packet delay. So, you need a predictor on the sender, fed by corrections from the receiver. This is control theory applied to network flow. This is how all of Hollywood sends reels around the basin to effects houses, and completed movies to digital projection theaters.
nealmueller 8y ago
The author is a middlebox employee (IPS, IDS, Firewall, NAT, WAN optimizers, LBs). Middlebox people want unencrypted transport headers, because they literally profit from unencrypted headers. :) Everyone else, including users, site operators, and software engineers writing network software prefer that middleboxes not be able to see or tamper with transport headers (both for privacy, avoiding bugs, and being able to evolve software). From the original article: "What's wrong with encrypted transport headers? One possible argument is that middleboxes actually serve a critical function in the network, and crippling them isn't a great idea. Do you really want a world where firewalls are unviable? But I work on middleboxes, so of course I'd say that. (Disclaimer: these are my own opinions, not my employer's)." (Credit for this observation goes to my friend NC.)
- zamadatix 8y ago
  Transparent proxy was the wrong way to implement IPS, IDS, FW, NAT, LB, and WAN optimization. For the cases you have a reason to be in the middle these services should have been explicit proxies from the start.

102 points by luu 8y ago

dylz 8y ago

Here's the problem: other companies ruined it for the good ISPs. They destroyed their trust by screwing with end user transfers, tampering with TLS, tampering with connections, modifying resources in transit, MITMing in horrific fashion, invading privacy, and breaking protocols for profit. Arguably there would be a lot less insane push to make everything a blackbox if companies didn't keep doing dumb crap for an extra buck. See also the TLSv1.3 middlebox arguments. I certainly enjoy the fact that when using an AT&T hotspot I can't reach certain websites over TLS, with Verizon they would append advertising tracking headers to requests on their end, on Sprint if I wget the same image multiple times it'll be in a different quality and not match md5-wise each time, all of the above do DPI and negatively degrade web traffic no matter what port it runs on, TCP connections fail if I try to run non-web traffic over port 80 randomly, TCP connections are randomly killed if long lived.

CobrastanJorji 8y ago
Exactly. Would the Internet be way easier to design and use if we could assume that everybody is a good actor? Heck yeah! It'd be great. There are some really useful things you can do when you're allowed to intercept the conversations between two people. Caching, debugging, fixing spelling errors, etc. But we can't assume that the folks between me and you are good, and sharing anything at all with them is a bad idea. Does that mean a whole bunch of really useful techniques are harder or straight up impossible now? Yes, absolutely, and we should plan ahead and make sure to come up with ways to debug issues like this. But stepping back on security and privacy because it makes it harder to benevolently spy? Screw that noise.
caf 8y ago
Here's the problem: other companies ruined it for the good ISPs. There's a phrase for that situation: "Poisoning the Well".
- TeMPOraL 8y ago
  How's it called when some companies are run by good humans, and other by evil reptilians, and the reptilians poison the well with something they can drink, but humans can't?
  - felipemnoa 8y ago
    terraforming
xyzzy123 8y ago
The problem I have with TLS 1.3 proposal is it’s gonna stop me having control over the things that go out of my own network :( Realise that this is a multifaceted debate but it’s going to destroy the utility of anything I can’t install a certificate on, and also force me to install certs on everything I own before I can comfortably let those things on the network. It’s gonna screw a lot of enterprise use cases as well. I’m not sure people fully get that the privacy extended to say “dissidents in Syria” is also going to apply to HP printers on their own networks trying to figure out whether to show “toner low” dialogs. Personally I don’t think the “hey the dissidents” value is worth it since those people are pretty screwed anyway - filtering at scale can still work out what you’re connecting to (ips, latency, response size patterns, blah blah) but it really messes up anyone (person or enterprise) who wants to use stuff but also know what’s going on.
- pilif 8y ago
  >also force me to install certs on everything I own before I can comfortably let those things on the network I'm curious: How is TLS 1.3 different from TLS 1.2 in that regard? How would you implement an intercepting proxy with TLS 1.2 and without installing a CA cert in the clients?
  - ameshkov 8y ago
    I guess the encrypted SNI draft is what's meant here.
    - xyzzy123 8y ago
      Thanks, sorry, you are totally right and I should be more specific. Yes. It’s a metadata leak and generally I am pro the end-to-end principle but encrypted SNI actually forces everyone to MITM. Whether that is good or bad is a value judgement but for people who have been doing “light touch” egress filtering it is a huge PITA. It is actually going to force more invasive surveillance in basically any regulated workplace.
      - 8y ago
dstroot 8y ago
> for the good ISPs... and those are? Do they actually exist?
- dylz 8y ago
  Some do. Monkeybrains in SFBA and AAISP, Bogons in UK are a few possible examples?

ocdtrekkie 8y ago

You know what a lot of middleboxes do? They block ads/malware. Shocking that the two largest ad companies are trying to push standards which break things that block their ads.

pilif 8y ago
This isn’t as much about stopping middleboxes on your perimeter that you fully control, but a lot about middleboxes between you and the destination that are outside of your or the destinations control. Those are often used to add ads rather than block them. If you yourself want to block ads, do so on your machine (where traffic has already been decrypted) or on your router (which will then decrypt traffic for you and re-encrypt it with its certificate that you have added to your machine).
icebraining 8y ago
That's news to me, I've never heard of a widespread use of middleboxes for blocking ads. Which boxes are these and how did you know that there are a lot of them deployed?
- ocdtrekkie 8y ago
  So, an example of a middlebox that's exceedingly common is a "web security gateway", which is your average web filter and logger in a corporate environment. Obviously it logs employee web activity, blocks access to adult websites, and maintains it's own malware definitions to try and block malicious content as well. It's quite often for these to also block domains used by ads and popups by default. When these sorts of devices are configured to inspect HTTPS, this adds a significant additional complexity: Network PCs need to be configured to permit a certificate from the box for all domains, which intercepts, decrypts, and re-encrypts all traffic. Of course, the same type of technology a corporation might use to manage their network could be used by a state actor or a hostile ISP.
  - brazzledazzle 8y ago
    If you’re only using it to block domains you can do that at the DNS level with the added bonus of it being more efficient.

quickben 8y ago

When advertisement companies push for L4 redesign... This is going to be a fun one to watch how it plays out.

nicenewsbc 8y ago

There is no combination between them.

tinus_hn 8y ago

It would be nice if these protocols could be easily decrypted using a key available on the client. Other than that, tough luck for the ‘transparent’ proxies and friends.

hornetblack 8y ago
I know that for https in NSS browsers (Chrome and FF) you can dump the secrets to a file that Wireshark can read. Given that Chrome is currently the biggest user i would be supprised if that was easy to update.
- nouseforaname 8y ago
  I believe chrome currently uses BoringSSL not NSS, but still has some facilities for writing the same keylogfile.
tialaramex 8y ago
Of necessity both client and server have the session keys. In principle it would be possible for the client to lack keys needed for the server to read data sent by the client, and vice versa, but in practice this is never done. Under Forward Secrecy a Middlebox must learn fresh session keys for every connection or it can't decrypt it. Both clients (e.g. Firefox) and servers (e.g. using Java or OpenSSL) have facilities to dump the session keys out somewhere, and this is adequate for debugging although obviously you will need to acquire new skills if you're used to being able to get stuff done with a paperclip and a copy of tcpdump. At scale this get hard, arguably that's fine because a minute ago we said we wanted this for "debugging" but people who got their foot in the door with a "debugging" argument often actually want to decrypt everything, always, and so they're unhappy about this. If you don't want Forward Secrecy you have two options. Firstly, when the specification says to think of a random number for the key exchange protocol, you can always pick the same number, or a number chosen in some predictable fashion, the Middlebox can know this number (or method for predicting it) and then it can snoop as normal. This works in TLS 1.3, obviously it weakens your security (if bad guys learn how to predict the numbers you are screwed) but that's your choice. Secondly you could use a key exchange process that doesn't have any Forward Secrecy by design, such as the RSA key exchange from SSL that's grandfathered into TLS 1.0 through 1.2. In this case you just give the server's private RSA key to the middleboxes and they can decrypt everything. As you may notice in all the above scenarios, this is very bad for your security. But if "debugging" is really the problem that's almost certainly acceptable to you.

olliej 8y ago

The other fun bit is the “encryption” of web sockets (outside of tls, etc). Basically (hand waving here) a websocket connection starts with a 64 bit (or 128, I really can’t remember anymore) value, and then essentially just xors that over all subsequent data. This isn’t needed for security of the user or the server. It’s because so many middleware boxes are so poorly build that you could make them crash and/or get code execution if you had sufficient control over enough of the right payload bytes. Java applets exposed the exact same problem, so given they existed a decade before websockets and yet the middleware boxes were still broken enough that this nonsense was required in the websocket spec should tell you everything you need to know.

Matthias247 8y ago
That's not only about Middleboxes (which mostly act up to L4). Most likely it's more about preventing JS from attacking arbitrary servers (e.g. broken mail servers, file servers, etc) that might have an open TCP port. The "encryption" at least prevents JS attacks from being deterministic.
- olliej 8y ago
  no, it was specifically due to broken middleware that had existing well known yet unfixed flaws.
- aidenn0 8y ago
  It was about Application layer middleboxes (specifically broken http proxies).
jorangreef 8y ago
And now, even WebSockets over TLS need servers to waste CPU on XOR masking.
- jefftk 8y ago
  XOR masking is ridiculously cheap compared to doing anything over the network or running js in general.

denormalfloat 8y ago

Alright, so existing debugging tools don't work with QUIC. We will need to make some new tools that can expose the information we need. If the hops between the source and destination are willing to expose the info, (and we can assume they do, as the author has), then we can figure out what packets go in that never come back out. Instead of say "this won't work because ____", why don't people say "it would work if we could ____"? Someone (or some company) needs to improve the Internet, and it seems like the world just harangues them for their effort.

cm2187 8y ago

Actually it raises an important point I hadn't thought about. By moving the transport protocol from layer 4 to layer 5, google is taking it off the hand of the OS. I understand why they do it (easier than to get all 3 or 4 major OS to implement it), but there is another cost in term of inter-operability. It means that every software consuming QUIC needs to have its own implementation of the protocol. It means every language in which you write that software needs to have libraries available implementing QUIC, or that the libraries you consume must themselves have a QUIC implementation. That's introducing a non insignificant inefficiency if QUIC becomes prevalent.

tialaramex 8y ago
The operating system can offer QUIC on top of UDP the same way it offers UDP on top of IP. BSD sockets aren't necessarily the ideal way to use QUIC but there's no reason you couldn't use them. Researchers have even repeatedly built Linux protocol modules that do TLS, either all of it, or the encrypted record layer (so the bulk but not the tricky negotiation decisions at the start). There's just a new TLS protocol you ask for instead of TCP and then the kernel handles encryption and so on.

deleted 8y ago

[deleted]

jsnell 8y ago

There's been an attempt to fix this during the last two years, as the QUIC standardization talks really got going. A bunch of operator people expressed a need for some sort of in-path measurability, while the privacy people have expressed the need to avoid any sort of session linkage or other forms of information leakage. The most viable compromise proposal seems to be the spin bit[0] which gives RTT measurements, but it's not agreed whether a version of that will make it to the first release. [0] https://quicwg.org/base-drafts/draft-ietf-quic-spin-exp.html https://quicwg.org/base-drafts/draft-ietf-quic-spin-exp.html

pdkl95 8y ago
> The most viable compromise proposal seems to be the spin bit That's utterly useless. If the "spin bit" becomes widely used, I intend to write a trivial patch that sets it randomly on each packet. If packets could be dropped unless the bit is set to specific values at specific times, too much session information is leaking to middleboxen. More likely, the state of the bit doesn't matter so setting it randomly will discourage wasting a bit in the protocol with this kind of nonsense in the future.

sly010 8y ago

Arguably you wouldn't have to debug bad connections if the middleboxes just did what they were supposed to.

vinay_ys 8y ago

This argument is broken. An ISP engineer debugging need only look at IP packet drops in their part of the network. If their end clients are asking them to debug an issue then they should debug it at the end points, not in the middle. This attitude is what lead them to stick more e and more buffers in the middle and do other shaping devices in the name of problem solving or adding value and ended up ossifying protocols. I think endpoints will evolve to better protocols to solve their own problems if the ISPs stopped putting hacks (like deep buffers) in their networks.

jsnell 8y ago
> This argument is broken. An ISP engineer debugging need only look at IP packet drops in their part of the network. That's just not true in practice, pretty much on any level. First, you need to look at a lot more things than just packet drops (e.g. reordering, queue buildup, corruption). Second, even getting full visibility into your own network is highly non-trivial since nobody has active probes on every link. My experience is that arranging for packet captures from an arbitrary point in the network could take a week. And if you guessed wrong about which node was at fault, you'd need to do it again in a binary search pattern. Third, you absolutely do need to know about things other than your own network. Otherwise you don't even know whether there is a problem you can solve. If the bottleneck is in the server, or the client, or the transit links, there is no point in debugging the core or the access network. > If their end clients are asking them to debug an issue then they should debug it at the end points, not in the middle. The endpoints are not going to be available. Do you think that Youtube is going to give an operator some kind of server access or even insight to the traffic? Do you really want to see a world where a customer having a complaint needs to first root their phone and install packet capturing software? What you're really saying here is that no problem should ever be debugged, and we should just hope that the network doesn't break. And hope is not much of a strategy.

ncmncm 8y ago

When you tunnel your protocol in UDP, and control both ends, you can get overwhelmingly better flow control than TCP, which cannot trust the other end, so must rely on packet drops to get a reliable signal. When you can trust the other end, rate of change of packet transfer time (delta packet delay) reveals congestion exactly -- i.e., increasing time means you had better slow down, decreasing, you can go faster. Only problem is, the receiver has the signal, but the sender needs it, and the useful lifetime is less than the packet delay. So, you need a predictor on the sender, fed by corrections from the receiver. This is control theory applied to network flow. This is how all of Hollywood sends reels around the basin to effects houses, and completed movies to digital projection theaters.

nealmueller 8y ago

The author is a middlebox employee (IPS, IDS, Firewall, NAT, WAN optimizers, LBs). Middlebox people want unencrypted transport headers, because they literally profit from unencrypted headers. :) Everyone else, including users, site operators, and software engineers writing network software prefer that middleboxes not be able to see or tamper with transport headers (both for privacy, avoiding bugs, and being able to evolve software). From the original article: "What's wrong with encrypted transport headers? One possible argument is that middleboxes actually serve a critical function in the network, and crippling them isn't a great idea. Do you really want a world where firewalls are unviable? But I work on middleboxes, so of course I'd say that. (Disclaimer: these are my own opinions, not my employer's)." (Credit for this observation goes to my friend NC.)

zamadatix 8y ago
Transparent proxy was the wrong way to implement IPS, IDS, FW, NAT, LB, and WAN optimization. For the cases you have a reason to be in the middle these services should have been explicit proxies from the start.