3 ms·

Their implementation of congestion control (reno) is 85 lines including a big copyright block comment. That's cute but it's completely incomparable to the tran

by kev009 6y ago

Their implementation of congestion control (reno) is 85 lines including a big copyright block comment. That's cute but it's completely incomparable to the transmission intelligence in the Linux TCP stack, especially for the line of business Fastly is engaged in (maybe the technical staff doesn't even know what that is due to poor leadership?) in trying to get goodput to devices across congested and channel lossly links. This article is highly comical. I've yet to be impressed by fastly aside from offloading itself to retail investors with another nonsensical IPO.

vardump 6y ago
They use Reno? I don't know what's their business, but hopefully it doesn't involve lossy links like mobile data, which often drop packets even when uncongested... Reno can work nicely for low packet loss links, though.
- kev009 6y ago
  Just pertinent to this article, take a look at the links to their QUIC implementation. The reason I bring this up is because it's pretty easy to make a simplified fast path TCP stack that doesn't have all the branch heavy code necessary to do what the Linux TCP stack does well. So this is like a Phoronix style "benchmark". An undergrad level of understanding of systems will net you that you can get equal performance in userspace or kernel space with sufficient holistic thinking and accounting of data movement. I would caution that very few people are qualified to do this in userspace though, which is paradoxical because you will find a lot more people showing up claiming they know how. So basically this isn't a competent test; it isn't the production grade QUIC from google, it's a toy and the result is complete nonsense.
  - vardump 6y ago
    Yeah, looked at their code [0]. You're right. I'm no TCP (or even congestion control) expert, but I wouldn't have needed to test their code in real life to tell exponentially reducing send window on a lost packet is just not going work in real life in 2020. They should have at least tested their code with simulated jittery latencies and packet loss typical for modern 4G networks. I bet Linux TCP would have left their implementation in dust. Another thing that caught my eye was their assumption that MTU is same on return path... probably often true, but definitely wouldn't count on it! Huge respect for them for putting their necks out and releasing code! Maybe this is their intention as well, to gain understanding through criticism. Also, perhaps this was just their MVP? It would certainly explain weird choice for congestion control. [0]: https://github.com/h2o/quicly/blob/master/lib/cc-reno.c https://github.com/h2o/quicly/blob/master/lib/cc-reno.c
    - kev009 6y ago
      It's just marketing bullshit, I don't feel bad at all for offering damning criticism. Companies hire people to do this kind of stuff on purpose. Many unfortunate software trends of the last decade have been started by equally flimsy PR jobs.
- esaym 6y ago
  Interestingly, I've found TCP_Illinois[0] to do the best on lossy links. Case in point was a cable modem isp I used for about a year once. The upstream was rated at 10mbs, but it would bounce between 1-5mbs. Using a UDP test, I noticed about a 0.2-1% packetloss on upstream only. I tried all the tcp algos in linux. Illinois did the best. And by "best" I mean it would hold a steady speed between 8-9mbs while others would get quite jumpy at times (or just never go above 5mbs). I'm limiting upstream to 9mbs via fq-codel as well. I've always found it strange Illinois is never brought up anywhere. It even has tuneable parameters too. [0]:https://en.wikipedia.org/wiki/TCP-Illinois https://en.wikipedia.org/wiki/TCP-Illinois
  - kev009 6y ago
    It's a hybrid meaning it may struggle in direct competition with pure loss based algorithms. CUBIC, the Linux default, is somewhat notorious in causing bloat issues but these things tend to play out like an arms race. BBR is another hybrid and has tons of heuristics and tuning to try and adapt to these arms race conditions.
  - vardump 6y ago
    Not questioning your conclusion; out of curiosity, did you also measure latency? You could get very close to maximum possible just by massively delaying & coalescing ACKs. In other words nearing blind transmission without caring about packet loss at all. But latency could get pretty bad!
    - ignoramous 6y ago
      I believe congestion control would be at the application-layer with QUIC, that is, is it left upto L7 protocols like HTTP/3 to implement their own congestion control and packet loss? For HTTP/3, then, would fq-codel + BBR work nicely for foreground traffic like voice or video over WebRTC and, say, LEDBAT for background traffic like HTTP downloads or, say, bit-torrent over WebRTC? (I realise congestion control is long from being solvable: https://news.ycombinator.com/item?id=17455276 https://news.ycombinator.com/item?id=17455276)
    - esaym 6y ago
ignoramous 6y ago
> I've yet to be impressed by fastly aside from offloading itself to retail investors with another nonsensical IPO. Wow, that's quite a strong sentiment, especially given CDNs are your area of expertise [0]. I was (am?) of the opinion that Fastly is a genuine competitor to Cloudflare given their product offering, and that their engineering was more than a match (though, to be honest, I'm mostly going by the presentations their engineers have given over the years). Your statement, to me, makes it seem like Fastly is smokes and mirrors. I'm honestly surprised. [0] https://news.ycombinator.com/item?id=17455940 https://news.ycombinator.com/item?id=17455940
- kev009 6y ago
  Cloudflare and Fastly are in somewhat different industries even though they can do some similar things. Fastly has a small number of large customers, Cloudflare has an astronomical number of customers which makes what they do even harder. From the outside looking in I would rate Cloudflare as one of the most technical companies in systems software industry at the moment. It's probably because CF has a highly technical CTO.
dang 6y ago
It sounds like you know a lot about this field, but can you please post more substantively, and in particular replace name-calling ("highly comical" "nonsensical") with actual details? If you know more than others, that's great, but the thing to do on HN is to share some of what you know so the rest of us can learn. A phrase like "cute but completely incomparable" doesn't teach the reader anything. It would be much better to say what the relevant aspects of "transmission intelligence in the Linux TCP stack" are, and what makes them relevant. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- kev009 6y ago
  It teaches readers not to trust a brand's PR jobs and brands to not mislead. Comedy is intricate, and yes I find the situation having to perform at higher standards than a $2B company in opining on here during my lunch break to be quite funny. A 747 is not comparable to a Cessna. The later is cute but not comparable and it doesn't require a dissertation to comment on the quality among other computing professionals.
  - dang 6y ago
    Attempts to make comedy by putting others and their work down on the internet are rarely funny, and in any case the HN guidelines specifically ask commenters not to do that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting here, we'd be grateful. Note "Don't be snarky." and "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." "A 747 is not comparable to a Cessna" doesn't help because it's really still just calling names. What would help is an explanation of why you're saying that. What are the significant differences and why do they matter? Lots of readers here would find that interesting. If you don't want to spend your lunch break writing an explanation to help people learn, that's totally fine. But at least please don't post empty comments that make HN worse and break the rules. Not posting is always an option. You may be overestimating the extent to which readers here will already understand the details of what you mean. No doubt some do, but I'm pretty sure the majority don't. Even most computing professionals aren't specialists in this area. That's why the opportunity to share information that readers find interesting and can learn from is such a good one. You can certainly (indeed, you should) make a strong critique of inflated and misleading claims, but do it with solid information, not shallow putdowns that any internet snarker can write whether they know anything or not.

by kev009 6y ago

vardump 6y ago

They use Reno? I don't know what's their business, but hopefully it doesn't involve lossy links like mobile data, which often drop packets even when uncongested... Reno can work nicely for low packet loss links, though.

kev009 6y ago
Just pertinent to this article, take a look at the links to their QUIC implementation. The reason I bring this up is because it's pretty easy to make a simplified fast path TCP stack that doesn't have all the branch heavy code necessary to do what the Linux TCP stack does well. So this is like a Phoronix style "benchmark". An undergrad level of understanding of systems will net you that you can get equal performance in userspace or kernel space with sufficient holistic thinking and accounting of data movement. I would caution that very few people are qualified to do this in userspace though, which is paradoxical because you will find a lot more people showing up claiming they know how. So basically this isn't a competent test; it isn't the production grade QUIC from google, it's a toy and the result is complete nonsense.
- vardump 6y ago
  Yeah, looked at their code [0]. You're right. I'm no TCP (or even congestion control) expert, but I wouldn't have needed to test their code in real life to tell exponentially reducing send window on a lost packet is just not going work in real life in 2020. They should have at least tested their code with simulated jittery latencies and packet loss typical for modern 4G networks. I bet Linux TCP would have left their implementation in dust. Another thing that caught my eye was their assumption that MTU is same on return path... probably often true, but definitely wouldn't count on it! Huge respect for them for putting their necks out and releasing code! Maybe this is their intention as well, to gain understanding through criticism. Also, perhaps this was just their MVP? It would certainly explain weird choice for congestion control. [0]: https://github.com/h2o/quicly/blob/master/lib/cc-reno.c https://github.com/h2o/quicly/blob/master/lib/cc-reno.c
  - kev009 6y ago
    It's just marketing bullshit, I don't feel bad at all for offering damning criticism. Companies hire people to do this kind of stuff on purpose. Many unfortunate software trends of the last decade have been started by equally flimsy PR jobs.
esaym 6y ago
Interestingly, I've found TCP_Illinois[0] to do the best on lossy links. Case in point was a cable modem isp I used for about a year once. The upstream was rated at 10mbs, but it would bounce between 1-5mbs. Using a UDP test, I noticed about a 0.2-1% packetloss on upstream only. I tried all the tcp algos in linux. Illinois did the best. And by "best" I mean it would hold a steady speed between 8-9mbs while others would get quite jumpy at times (or just never go above 5mbs). I'm limiting upstream to 9mbs via fq-codel as well. I've always found it strange Illinois is never brought up anywhere. It even has tuneable parameters too. [0]:https://en.wikipedia.org/wiki/TCP-Illinois https://en.wikipedia.org/wiki/TCP-Illinois
- kev009 6y ago
  It's a hybrid meaning it may struggle in direct competition with pure loss based algorithms. CUBIC, the Linux default, is somewhat notorious in causing bloat issues but these things tend to play out like an arms race. BBR is another hybrid and has tons of heuristics and tuning to try and adapt to these arms race conditions.
- vardump 6y ago
  Not questioning your conclusion; out of curiosity, did you also measure latency? You could get very close to maximum possible just by massively delaying & coalescing ACKs. In other words nearing blind transmission without caring about packet loss at all. But latency could get pretty bad!
  - ignoramous 6y ago
    I believe congestion control would be at the application-layer with QUIC, that is, is it left upto L7 protocols like HTTP/3 to implement their own congestion control and packet loss? For HTTP/3, then, would fq-codel + BBR work nicely for foreground traffic like voice or video over WebRTC and, say, LEDBAT for background traffic like HTTP downloads or, say, bit-torrent over WebRTC? (I realise congestion control is long from being solvable: https://news.ycombinator.com/item?id=17455276 https://news.ycombinator.com/item?id=17455276)
  - esaym 6y ago

ignoramous 6y ago

> I've yet to be impressed by fastly aside from offloading itself to retail investors with another nonsensical IPO. Wow, that's quite a strong sentiment, especially given CDNs are your area of expertise [0]. I was (am?) of the opinion that Fastly is a genuine competitor to Cloudflare given their product offering, and that their engineering was more than a match (though, to be honest, I'm mostly going by the presentations their engineers have given over the years). Your statement, to me, makes it seem like Fastly is smokes and mirrors. I'm honestly surprised. [0] https://news.ycombinator.com/item?id=17455940 https://news.ycombinator.com/item?id=17455940

kev009 6y ago
Cloudflare and Fastly are in somewhat different industries even though they can do some similar things. Fastly has a small number of large customers, Cloudflare has an astronomical number of customers which makes what they do even harder. From the outside looking in I would rate Cloudflare as one of the most technical companies in systems software industry at the moment. It's probably because CF has a highly technical CTO.

dang 6y ago

It sounds like you know a lot about this field, but can you please post more substantively, and in particular replace name-calling ("highly comical" "nonsensical") with actual details? If you know more than others, that's great, but the thing to do on HN is to share some of what you know so the rest of us can learn. A phrase like "cute but completely incomparable" doesn't teach the reader anything. It would be much better to say what the relevant aspects of "transmission intelligence in the Linux TCP stack" are, and what makes them relevant. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

kev009 6y ago
It teaches readers not to trust a brand's PR jobs and brands to not mislead. Comedy is intricate, and yes I find the situation having to perform at higher standards than a $2B company in opining on here during my lunch break to be quite funny. A 747 is not comparable to a Cessna. The later is cute but not comparable and it doesn't require a dissertation to comment on the quality among other computing professionals.
- dang 6y ago
  Attempts to make comedy by putting others and their work down on the internet are rarely funny, and in any case the HN guidelines specifically ask commenters not to do that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting here, we'd be grateful. Note "Don't be snarky." and "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." "A 747 is not comparable to a Cessna" doesn't help because it's really still just calling names. What would help is an explanation of why you're saying that. What are the significant differences and why do they matter? Lots of readers here would find that interesting. If you don't want to spend your lunch break writing an explanation to help people learn, that's totally fine. But at least please don't post empty comments that make HN worse and break the rules. Not posting is always an option. You may be overestimating the extent to which readers here will already understand the details of what you mean. No doubt some do, but I'm pretty sure the majority don't. Even most computing professionals aren't specialists in this area. That's why the opportunity to share information that readers find interesting and can learn from is such a good one. You can certainly (indeed, you should) make a strong critique of inflated and misleading claims, but do it with solid information, not shallow putdowns that any internet snarker can write whether they know anything or not.