8 ms·
Filter all ICMP and watch the world burn
- taspeotis 11y agoThis is a well known problem. Windows (since at least 2000) can detect this scenario and mitigate it [1]. [1] https://technet.microsoft.com/en-us/library/cc960465.aspx https://technet.microsoft.com/en-us/library/cc960465.aspx
- js2 11y agoEh, seen this so many times... My favorite related problem was about 12 years ago when I had a Mac and a Linux box side-by-side and the Mac could connect to a Verizon site (a paging gateway) while the Linux box never even got a response to its SYN. I eventually figured out the Linux box had ECN enabled. Probably an out-of-date firewall at Verizon's end didn't like such exotic TCP options. Disabling ECN on the Linux box fixed the issue. (I believe I was working on an email-to-page script at the time.)
- mcguire 11y agoECN was a bit of a special case, and a major pain in the ass: it uses a previously-unused bit in the header and "security conscious" network hardware developers forgot the "liberal in what you accept", set-it-to-zero-when-you-create-a-packet-and-ignore-it-otherwise proper default behavior for unused bits.
- deleted 11y ago[deleted]
- znep 11y agoBrings back memories from my past... http://znep.com/~marcs/mtu/ http://znep.com/~marcs/mtu/ Woefully out of date and wrong, but helped some folks out.
- cperciva 11y agoLast time I checked, EC2 defaulted to filtering ICMP packets, with the predictable bad results: http://www.daemonology.net/blog/2012-11-28-broken-EC2-firewall.html http://www.daemonology.net/blog/2012-11-28-broken-EC2-firewa...
- acdha 11y agoAs a bonus, you can't enable them for things like ELBs where you don't control the box.
- jsmthrowaway 11y agoYou can in VPC just fine. The ELB has a regular security group.
- X-Istence 11y agoThe lowest MTU on IPv6 is 1280, which means that even if we go with the minimal MTU just to get traffic to flow, it's not as terrible as IPv4's minimum MTU: 576.
- e12e 11y agoThat just means you'll have to encapsulate ipv6 in ipv4 to get across those really poorly configured routers...
- vacri 11y agoWould those routers support 6 in the first place?
- mobiplayer 11y agoIf it's encapsulated it doesn't matter :)
- deleted 11y ago[deleted]
- dap 11y agoWhen a client tries to send packets too big for the network (as when the client is configured with jumbo frames but the network isn't), this can be really painful to debug. The worst is that many things will work because small packets get through. For example, an "scp" connection may successfully connect, and then just hang when it starts transferring real data.
- contingencies 11y agoFor those who are new to this issue: ICMP is the sort of 'signalling' protocol related to IP itself. ICMP provides a few services (eg. classic ping), but critically when issues within the IP layer occur elsewhere on the internet while attempting the delivery of a packet, ICMP messages are usually sent in response. However, recently (~last 15-20 years) badly configured firewalls block all ICMP or certain types of ICMP, which can result in difficulties in communication. In this case, type 3 code 4 was blocked. More info @ http://en.wikipedia.org/wiki/Internet_Control_Message_Protocol http://en.wikipedia.org/wiki/Internet_Control_Message_Protoc...
- avodonosov 11y agoThanks. Why does the source host sets the Don't Fragment flag?
- contingencies 11y agoAccording to the original IPv4 RFC @ http://tools.ietf.org/html/rfc791 http://tools.ietf.org/html/rfc791 page #25... If the Don't Fragment flag (DF) bit is set, then internet fragmentation of this datagram is NOT permitted, although it may be discarded. This can be used to prohibit fragmentation in cases where the receiving host does not have sufficient resources to reassemble internet fragments. One example of use of the Don't Fragment feature is to down line load a small host. A small host could have a boot strap program that accepts a datagram stores it in memory and then executes it. What it seems to mean is things like PXE[1] or BOOTP[2]. Basically, DF was built to allow the sender to optionally force zero fragmentation by intermediate hosts en-route that are connected to networks with a smaller MTU than the packet size originally emitted by the sender. This was originally intended to be of use because the sender was somehow made aware of limitations in the recipient's network stack. Probably 20 or more years ago it was used for awhile as a latency-related hack for certain applications (VOIP, video, low-latency finance, certain scientific experiments generating vast amounts of data, etc.) mostly on UDP[3], though we have better methods for those now that operate through other IP headers (QoS). Theoretically, intermediate nodes could also use it as part of path selection during routing, though I have no idea if this has been done or is encouraged - eg. a packet from node A reaches node B en-route to node C. Node B has two routes to node C. The lower-cost route is available with a smaller MTU than the packet size, and a higher-cost route is available with a large enough MTU to accommodate the packet size. The DF flag could be used by routing logic at node B to automatically shuffle the packet across that higher-cost route. I believe the Path MTU discovery[4] feature of modern Linux kernels also probably uses this mechanism combined with short TTLs (ie. maximum hop counts, which also cause ICMP errors to be returned on their failure, and are the basis of traceroute[5], itself an unintended hack based on unrecognized capabilities of the IP protocol's specification) to optimize long-lived traffic flows. There must be other edge-case or optimization-related wishy-washy reasons to set it, too. The main thing to remember is: none of these were really intended by the authors of IPv4. This whole historic pile of what-if edge-case hackyness has been thrown out in favor of a better system in IPv6[6]. [1] http://en.wikipedia.org/wiki/Preboot_Execution_Environment http://en.wikipedia.org/wiki/Preboot_Execution_Environment [2] http://en.wikipedia.org/wiki/Bootstrap_Protocol http://en.wikipedia.org/wiki/Bootstrap_Protocol [3] http://en.wikipedia.org/wiki/User_Datagram_Protocol http://en.wikipedia.org/wiki/User_Datagram_Protocol [4] http://en.wikipedia.org/wiki/Path_MTU_Discovery http://en.wikipedia.org/wiki/Path_MTU_Discovery [5] http://en.wikipedia.org/wiki/Traceroute http://en.wikipedia.org/wiki/Traceroute [6] http://en.wikipedia.org/wiki/IPv6#Simplified_processing_by_routers http://en.wikipedia.org/wiki/IPv6#Simplified_processing_by_r...
- KaiserPro 11y agoYup, I've had this: "ICMP is a security hole" lets turn it off. such a tedious conversation to have with the networks(!) team
- droopyEyelids 11y agoTCP and UDP are the real culprits. As a network admin with unquestionable authority I always advocate for banning them on my networks.
- georgerobinson 11y agoCan you explain how ICMP is insecure? Is it just ping being exploited?
- MichaelCrawford 11y agoThe kernel doesn't pass ICMP packets through to userspace.
- georgerobinson 11y agoI see. But why is this a security threat?
- MichaelCrawford 11y ago
- ChuckMcM 11y agoWhen people filter ICMP is really really annoys me. Sure I get that some people don't like to respond to pings, or that you can ddos some routers by flooding them with 64 byte packets (old routers btw) but hey ICMP is a critical part of making the network work correctly.
- Twirrim 11y agoUnfortunately almost every time I've dealt with PCI-DSS compliance auditors they almost always raise the fact that I haven't got ICMP completely filtered. It's always an annoying, long argument with them about why ICMP exists, why it shouldn't be completely filtered, and what the potential side effects of filtering it are.
- OSButler 11y agoMost annoying instance I've experienced with PCI compliance was with an auditor who didn't understand the concept of patching. Even after providing the full list of backported CVEs, which clearly showed the one he insisted was missing on the system, he still refused to pass it. Only after escalating the issue was it finally marked as passed.
- vidarh 11y agoI've had that too - the only thing the auditor in question cared about was the version number that their run of nmap had guessed.
- vidarh 11y agoNot PCI-DSS, but I dealt with security auditors for a client that insisted that we shouldn't allow ping... To the address of the public website... Because we might reveal there was something there. My passive aggressive response was to point out, while copying the client, that a number of more prominent security auditors sites responded to ping, as well as the websites of any number of intelligence organizations, banks and similar.
- MichaelGG 11y agoJust had an audit with one of the biggest telcos, done by a large "security" firm. They insisted that the public website not respond unless the right Host header was there. Stupid, but OK, I can see it on a checklist for intranet apps. But the real kicker: The site was TLS only, so connecting to the IP will still leak the hostname, from the cert. Edit: This was a really big security firm, too. Totally worthless audit. They actually complained that a site admin could "include iframes in the HTML, which could be a malware vector" when uploading content. Ignoring that they could also upload scripts and arbitrary binaries. After their weeklong "penetration test" concluded, I found some serious XSS (public user->admin, which could easily turn into system takeover) with about 5 minutes of looking. Are most audits this useless?
- jvdh 11y agoDiscovering Path MTU black holes on the Internet using RIPE Atlas: https://www.os3.nl/_media/2011-2012/courses/rp2/p57_report.pdf https://www.os3.nl/_media/2011-2012/courses/rp2/p57_report.p... A Master thesis research report from 2012 which examined this very problem on a global scale, using the RIPE Atlas monitoring network.
- lloeki 11y ago> ipv6 > someone blackholed the very important packets which say "fragmentation needed but DF set" IIRC IPv6 never fragments and uses MTU path discovery (via ICMPv6)
- feld 11y agoIPv6 does have fragments. Routers will not do the fragmentation. They just drop the packet and force the client to do it. It can be horrible if you're using a tunnel broker and don't lower your MTU.
- feld 11y agoRead the title and immediately knew it would be PMTU.
- Swannie 11y agoDitto. Yet I still have this conversation with self styled "network architects" who want to blanket block all ICMP. It's depressing.
- benjojo12 11y agoWe got hit massively by this at CloudFlare (though we were not explicitly filtering ICMP, but changes we made to our infra meant that PMTU packets got lost) We wrote a blog post about this too: https://blog.cloudflare.com/path-mtu-discovery-in-practice/ https://blog.cloudflare.com/path-mtu-discovery-in-practice/ and the solution to our change: https://github.com/cloudflare/pmtud https://github.com/cloudflare/pmtud
- michh 11y agoInteresting! I'd think, in theory the ECMP router could keep track of the MTU on a per IP basis (rather than per TCP connection) based on it having received the ICM unreachable packet. And from that moment on, sending a spoofed ICMP packet back whenever one of the servers it's routing for sends a packet the router knows won't reach the host. But even if that works, I'm by no means a network engineer, your solution of simply broadcasting the packets is probably more efficient in the real world.
- bgilroy26 11y agoEducational Stack Overflow from the Google results for 'mtu': http://serverfault.com/questions/43866/whats-the-best-mtu-setting-for-a-web-server http://serverfault.com/questions/43866/whats-the-best-mtu-se...
- junto 11y agoWe have what I think is a long running MTU problem on Rackspace hosting for a customer. We are losing parts of HTTP requests between the H5 load balancer and the customer's web servers (which are running IIS). The header of the request reaches IIS and then the content body of the request fails to turn up causing a 500 error on the server. Issue is mostly seen on POST requests where the content of the request is going to be split over more packets. It's been driving us nuts. I wonder if we should also check the ICMP blocking too?
- mobiplayer 11y agoHey! Ex-Racker (NetSec) here. The truth is on the wire, get captures and check where the packets are dropped. Do you have the F5 in front of the webservers or on a different interface in one-arm mode? Second case you're going through the firewall, so I guess that's your scenario.
- simon_vetter 11y agoicmp in ipv6 does much more than its ipv4 counterpart and most importantly: 1) L2 address resolution (neighbor discovery), which ARP used to do in ipv4, 2) full network autoconfiguration (global scope addresses, default route(s), DNS resolver), which DHCP used to do in ipv4 (although DHCPv6 is still an option), 3) multicast group management (MLD), which igmp used to do in ipv4, 4) path mtu discovery (through 'packet too big' messages this article references). Routers fragment packets exceeding the link MTU in ipv4, they notify the source of the lower mtu in ipv6. ping, TTL exceeded, destination (host, route or port) unreachable and parameter problem were mostly carried over from ipv4. Blocking 1, 2 (and to some extent 3) on a local network will most likely break ipv6 connectivity entirely while blocking the others will only break it in subtle, hard to debug ways (especially with ECMP and traffic engineering where multiple routes for a given destination can be used). I've found that explaining this before asking network admins to unblock icmpv6 filters is a good way to succeed (although it can be hard, i'll give you that). People aren't used to filter ARP or link local broadcast in ipv4 (which DHCP uses), so telling them that they need to allow icmpv6 to let stations merely configure themselves is a bit of a mentality change. At the same time, developers of firewall management tools like ufw understood this problem a while ago and insert a working, good, tried and tested icmpv6 accept list as first rule which you can't mess with. Telling people to use ufw is usually much better than teaching them ip[6]tables.
- smkelly 11y agoWe use a load balancer product that is Linux-based. It defaulted to blocking all IPv6 ICMP, including neighbor discovery. This made IPv6 not work at all. It was a struggle to get them to fix it. And I don't think they've released the update with the fix yet either.
- tzakrajs 11y agoSome application protocols require ICMP and another TCP or UDP port and won't send their TCP or UDP packets until the ICMP ping packet has successfully been responded to.
- lamontcg 11y agoIts really kind of depressing that this is 'news' enough that it gets 245+ upvotes here. It can't be older than the internet itself, obviously, but its damn close...
- scurvy 11y agoI'd probably be saying "Eureka!" if this were 1997. But it's 2015. This is super basic stuff. Do people these days just blindly assume a MSS of 1460 is going to work on the Internet? Or do they think that the Internet is comprised entirely of Ethernet links? Or has the use of cloud providers and reliance on higher level programming languages produced a generation of ops people who don't understand the mechanics of how things work?
- scurvy 11y agoEveryone who downvoted my comment needs to pick up and read a copy of Comer's "Internetworking with TCP/IP". That and Stevens' TCP/IP Illustrated are the best sources for networking out there. You won't find the info on any blog. It won't be in something on ServerFault or StackOverflow and definitely not HN. Buy, read, learn. Yes, I know the US publisher has messed with the pricing for current versions but you can find the previous ones used pretty cheap. Other than dropping the IPng chapter, I doubt much has changed.
- sliken 11y agoHeh, back when google was just a white page with a search bar I noticed that the front page would come up, but he results didn't. Turns out I was on a home network connection with PPPoE which slightly decreases the maximum MTU. I opened a ticket with google and a SRE called me back (to my surprise) and we tracked it down. Google had a new firewall that was blocking MTU negotiation.
- mjankowski 11y agoDamn, I had this issue a while ago but I was not able to figure out the cause. I posted to stackoverflow but nobody pointed me in this direction. now I went back and answered my own question :) thanks! @jonchang