5 ms·

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something li

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

Thev00d00 3y ago
Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.
- axus 3y ago
  They should go to http://169.150.211.2 http://169.150.211.2 and see if that gets blocked. I get a "Welcome to nginx!" page there.
  - WirelessGigabit 2y ago
    That's weird. I get nothing there.
js2 3y ago
But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.
- Existing4190 3y ago
  You are right. My comment can't solve the whole story. Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.
  - xnyan 3y ago
    Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.
    - deleted 3y ago
      [deleted]
      - rstuart4133 3y ago
        Similar happened to me not so long ago. One day a junior admin asked me to diagnose who ssh to a box started hanging. After a bit of diagnosis it became evident TCP's 3 way handshake got through, and then it all stopped. No normal network behaves like this. My answer was "some network admin is having fun with a middleware box", you will have to speak with them. They did, and the response that came back is "we are moving to a network with real security". Access was restored. It was a Palo Alto NGFW as others have mentioned. IMHO the industry looking for way to move high priced gear, and they convince someone with a sparrow problem with barely knows how to handle an air rifle to buy bazooka's. A bit of collateral damage is to be expected....
  - js2 3y ago
    Filtering + established connection tracking: https://news.ycombinator.com/item?id=39822214 https://news.ycombinator.com/item?id=39822214
- xyst 3y ago
  Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.
  - js2 3y ago
    That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.
    - dunham 3y ago
      It's been a very long time since I've diagnosed something like this, but I've had problems in the past when the MTU is smaller than the default and ICMP is blocked (interfering with path MTU discovery). Often IPSec or some other tunneling was involved. The initial packets got through but as soon as a full packet was sent it was dropped. EDIT - I've now scrolled down in HN and saw that this was ruled out.
- xnyan 3y ago
  ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.
  - whirlwin 3y ago
    > Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. Are you referring to local networks only? It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.
    - pajko 3y ago
      Fragmented packets won't work without ICMP. Edit: here's a good page about the effects of disabling ICMP: https://www.rimscout.com/why-you-should-not-block-icmp/ https://www.rimscout.com/why-you-should-not-block-icmp/ Also there's some blackhole detection or how is it called. However it's OK to block _parts_ of the ICMP protocol for security reasons, like echo and reply.
    - fragmede 3y ago
      That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.
      - Hikikomori 3y ago
        AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).
        immibis 3y ago
        This explains why some people have problems with IPv6 - if you block IPv6 Control Messages, then it will only work sometimes.
dunham 3y ago
My employer did something like that once, and it took out access to github.

Thev00d00 3y ago

Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.

axus 3y ago
They should go to http://169.150.211.2 http://169.150.211.2 and see if that gets blocked. I get a "Welcome to nginx!" page there.
- WirelessGigabit 2y ago
  That's weird. I get nothing there.

js2 3y ago

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

Existing4190 3y ago
You are right. My comment can't solve the whole story. Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.
- xnyan 3y ago
  Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.
  - deleted 3y ago
    [deleted]
    - rstuart4133 3y ago
      Similar happened to me not so long ago. One day a junior admin asked me to diagnose who ssh to a box started hanging. After a bit of diagnosis it became evident TCP's 3 way handshake got through, and then it all stopped. No normal network behaves like this. My answer was "some network admin is having fun with a middleware box", you will have to speak with them. They did, and the response that came back is "we are moving to a network with real security". Access was restored. It was a Palo Alto NGFW as others have mentioned. IMHO the industry looking for way to move high priced gear, and they convince someone with a sparrow problem with barely knows how to handle an air rifle to buy bazooka's. A bit of collateral damage is to be expected....
- js2 3y ago
  Filtering + established connection tracking: https://news.ycombinator.com/item?id=39822214 https://news.ycombinator.com/item?id=39822214
xyst 3y ago
Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.
- js2 3y ago
  That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.
  - dunham 3y ago
    It's been a very long time since I've diagnosed something like this, but I've had problems in the past when the MTU is smaller than the default and ICMP is blocked (interfering with path MTU discovery). Often IPSec or some other tunneling was involved. The initial packets got through but as soon as a full packet was sent it was dropped. EDIT - I've now scrolled down in HN and saw that this was ruled out.
xnyan 3y ago
ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.
- whirlwin 3y ago
  > Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. Are you referring to local networks only? It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.
  - pajko 3y ago
    Fragmented packets won't work without ICMP. Edit: here's a good page about the effects of disabling ICMP: https://www.rimscout.com/why-you-should-not-block-icmp/ https://www.rimscout.com/why-you-should-not-block-icmp/ Also there's some blackhole detection or how is it called. However it's OK to block _parts_ of the ICMP protocol for security reasons, like echo and reply.
  - fragmede 3y ago
    That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.
    - Hikikomori 3y ago
      AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).
      - immibis 3y ago
        This explains why some people have problems with IPv6 - if you block IPv6 Control Messages, then it will only work sometimes.

dunham 3y ago

My employer did something like that once, and it took out access to github.