5 ms·

Hanging S3 downloads

235 points by duggan 9y ago

the_mitsuhiko 9y ago
> "But Juho, what does any of this have to do with S3?", you ask. Well, S3 is one of those rare services that disable timestamps. Interesting. Every once in a while I have issues on my cable provider (UPC in Austria) with S3 and S3 only. Next time that happens I gotta have a closer look at it.
- manmal 9y ago
  I'm also on UPC in Austria, and yes, I'm suspecting the same issue now. I will switch to Bridge mode now because of this.
  - mrkrabo 9y ago
    Maybe it's an issue with the cablemodem? There aren't many different manufacturers of cablemodems. What's yours?
    - manmal 9y ago
      Actually I received a new one yesterday. I have to check again whether the issue persists. The new one is called "Connect Box" and is jointly manufactured by Compal und Arris.
      - the_mitsuhiko 9y ago
        Also using Connect Box. No interest in switching to bridge however as I have no device capable of connecting by itself.
Rjevski 9y ago
Yet another problem caused by shitty consumer-grade network equipment.
- mjw1007 9y ago
  Also: yet another problem caused by NAT.
  - the_mitsuhiko 9y ago
    And yet NAT solves more problems than it causes.
    - Rjevski 9y ago
      I disagree. What problem does NAT solve, other than allowing people to cut corners and not deploy IPv6?
      - kingraoul3 9y ago
        Well, it allowed us to keep growing the internet before we had IPv6, for one thing.
        Rjevski 9y ago
        Wasn't IPv6 already out before we ran out of IPv4 addresses? In which case NAT is just a kluge for the lazy who didn't want to deploy IPv6.
        wbl 9y ago
        You might want to look at when IPv6 was designed. It's quite a bit older than you might think.
        sulam 9y ago
        Designed is very different from available in any form. If I couldn't buy a router that supports it with any amount of money, NAT was my only answer. This was the existing state of the world for a long time.
      - diamondlovesyou 9y ago
coldcode 9y ago
We take network transmissions for granted sometimes, TCP/IP has been around so long we assume nothing strange can still happen. But implementations still matter, it only take one strange piece of software or hardware and you wind up with craziness like this.
- moduspwnens14 9y ago
  Yep. Honestly I probably would have assumed it's not a protocol issue if both HTTP and HTTPS downloads from S3 behaved the same way. Both of those protocols are on TCP, though, so that would have thrown me off!
smudgymcscmudge 9y ago
Way to put a spoiler right in the title. I'm happy to have read this before seeing the post here.
- JCharante 9y ago
  but the title of the HN post has the same content as the title of the post?
  - jsnell 9y ago
    It's a bit unfortunate that there's no history of the title edits on HN, as it makes it hard to understand the context of any complaints about titles. The post was originally submitted with the same title as the original post, "The mystery of the hanging S3 downloads". That was apparently judged to be clickbait, so the title was changed to something along the lines of "Hanging S3 downloads due to NAT router", and the story was also very heavily penalized at the same time. This title was indeed giving away most of the ending. But luckily it was a bit later edited again, to the current form of "Hanging S3 downloads".
- 8draco8 9y ago
  This is not supposed to be thrilling story. It's a news.
fumigator 9y ago
Your cloud service does everything except for file backups, load balancing and can't handle huge file copies. And for reliability and responsiveness, requires a hybrid private-public cloud. Which begs the question as to what's the point of moving to the cloud in the first place.
- Rjevski 9y ago
  Is this a troll or you are really blaming Amazon for a fault in a shitty router?
  - CupOfJava 9y ago
    I don't think he read the article, just the title.
  - deleted 9y ago
    [deleted]
  - joshribakoff 9y ago
    I had the same reaction. If I need to support users with shitty routers should I stay away from amazon? My users will just blame me not their router, or switch to my competition who is not using Amazon since it works. My users aren't going to read a long explanation of TCP/IP. Some users have no choice like a school or work firewall/router. If my website doesn't work and the competitions does what do you think happens next if I don't stop using s3? The real question is how many users have a "shitty router".
    - Rjevski 9y ago
      Well S3 is popular enough (you don't realise it but a lot of sites use it under the hood for static assets, file uploads, etc) and this is the first time I've heard of this issue, so the number of routers so shitty that they break S3 (and only S3) would be pretty insignificant. There will be more routers that break the general web (no matter which site) than those only breaking S3.
deathanatos 9y ago
Huh. The Factorio community also ran into hanging downloads, and it also seemed to the devs to be caused by checksums[1][2]. I think they had much the same solution, mix a random number into the packet. (They use UDP, so they didn't have the benefit of TCP timestamps.) [1]: https://www.factorio.com/blog/post/fff-176 https://www.factorio.com/blog/post/fff-176 [2]: https://www.reddit.com/r/factorio/comments/5rvkyp/friday_facts_176_belts_optimization_for_015/ https://www.reddit.com/r/factorio/comments/5rvkyp/friday_fac...

235 points by duggan 9y ago

> "But Juho, what does any of this have to do with S3?", you ask. Well, S3 is one of those rare services that disable timestamps. Interesting. Every once in a while I have issues on my cable provider (UPC in Austria) with S3 and S3 only. Next time that happens I gotta have a closer look at it.

manmal 9y ago
I'm also on UPC in Austria, and yes, I'm suspecting the same issue now. I will switch to Bridge mode now because of this.
- mrkrabo 9y ago
  Maybe it's an issue with the cablemodem? There aren't many different manufacturers of cablemodems. What's yours?
  - manmal 9y ago
    Actually I received a new one yesterday. I have to check again whether the issue persists. The new one is called "Connect Box" and is jointly manufactured by Compal und Arris.
    - the_mitsuhiko 9y ago
      Also using Connect Box. No interest in switching to bridge however as I have no device capable of connecting by itself.

Rjevski 9y ago

Yet another problem caused by shitty consumer-grade network equipment.

mjw1007 9y ago
Also: yet another problem caused by NAT.
- the_mitsuhiko 9y ago
  And yet NAT solves more problems than it causes.
  - Rjevski 9y ago
    I disagree. What problem does NAT solve, other than allowing people to cut corners and not deploy IPv6?
    - kingraoul3 9y ago
      Well, it allowed us to keep growing the internet before we had IPv6, for one thing.
      - Rjevski 9y ago
        Wasn't IPv6 already out before we ran out of IPv4 addresses? In which case NAT is just a kluge for the lazy who didn't want to deploy IPv6.
      - wbl 9y ago
        You might want to look at when IPv6 was designed. It's quite a bit older than you might think.
        sulam 9y ago
        Designed is very different from available in any form. If I couldn't buy a router that supports it with any amount of money, NAT was my only answer. This was the existing state of the world for a long time.
    - diamondlovesyou 9y ago

coldcode 9y ago

We take network transmissions for granted sometimes, TCP/IP has been around so long we assume nothing strange can still happen. But implementations still matter, it only take one strange piece of software or hardware and you wind up with craziness like this.

moduspwnens14 9y ago
Yep. Honestly I probably would have assumed it's not a protocol issue if both HTTP and HTTPS downloads from S3 behaved the same way. Both of those protocols are on TCP, though, so that would have thrown me off!

smudgymcscmudge 9y ago

Way to put a spoiler right in the title. I'm happy to have read this before seeing the post here.

JCharante 9y ago
but the title of the HN post has the same content as the title of the post?
- jsnell 9y ago
  It's a bit unfortunate that there's no history of the title edits on HN, as it makes it hard to understand the context of any complaints about titles. The post was originally submitted with the same title as the original post, "The mystery of the hanging S3 downloads". That was apparently judged to be clickbait, so the title was changed to something along the lines of "Hanging S3 downloads due to NAT router", and the story was also very heavily penalized at the same time. This title was indeed giving away most of the ending. But luckily it was a bit later edited again, to the current form of "Hanging S3 downloads".
8draco8 9y ago
This is not supposed to be thrilling story. It's a news.

fumigator 9y ago

Your cloud service does everything except for file backups, load balancing and can't handle huge file copies. And for reliability and responsiveness, requires a hybrid private-public cloud. Which begs the question as to what's the point of moving to the cloud in the first place.

Rjevski 9y ago
Is this a troll or you are really blaming Amazon for a fault in a shitty router?
- CupOfJava 9y ago
  I don't think he read the article, just the title.
- deleted 9y ago
  [deleted]
- joshribakoff 9y ago
  I had the same reaction. If I need to support users with shitty routers should I stay away from amazon? My users will just blame me not their router, or switch to my competition who is not using Amazon since it works. My users aren't going to read a long explanation of TCP/IP. Some users have no choice like a school or work firewall/router. If my website doesn't work and the competitions does what do you think happens next if I don't stop using s3? The real question is how many users have a "shitty router".
  - Rjevski 9y ago
    Well S3 is popular enough (you don't realise it but a lot of sites use it under the hood for static assets, file uploads, etc) and this is the first time I've heard of this issue, so the number of routers so shitty that they break S3 (and only S3) would be pretty insignificant. There will be more routers that break the general web (no matter which site) than those only breaking S3.

deathanatos 9y ago

Huh. The Factorio community also ran into hanging downloads, and it also seemed to the devs to be caused by checksums[1][2]. I think they had much the same solution, mix a random number into the packet. (They use UDP, so they didn't have the benefit of TCP timestamps.) [1]: https://www.factorio.com/blog/post/fff-176 https://www.factorio.com/blog/post/fff-176 [2]: https://www.reddit.com/r/factorio/comments/5rvkyp/friday_facts_176_belts_optimization_for_015/ https://www.reddit.com/r/factorio/comments/5rvkyp/friday_fac...