4 ms·
Some interesting stuff I found on IX LANs
- lazide 1y agoI wish they didn’t defacto sort least-interesting-first :s
- protocolture 1y agoOne of my past employers had a crazy simple networking philosophy. It killed me. He once had his provider shut the port facing his core switch down, due to STP BPDUs being transmitted. To remedy this, instead of disabling the feature on the switch, he replaced it with an unmanaged switch that doesnt speak STP.
- ThePowerOfFuet 1y agoHe had no business managing a network.
- protocolture 1y agoHe had a particular niche that had few requirements on complex networking, and a business plan that was so brain dead simple it could be achieved easily. But yeah, really nice guy but unable to expand beyond that paradigm. I once helped him deal with a fibre ring, where there were 5 sites in the ring, but only 2 internet services. So sites 3, 4 and 5 had to communicate via sites 1 and 2. His team had put a sophos firewall, with NAT, and no routing, on every fibre connection and called me when it didnt work.
- api 1y agoThat's not great, but it's better than the opposite. I've been in networking for ages and have observed that most networking people will, as a rule, make networks as complicated as possible. Why have one layer of NAT when you can have four or five? Why not invent a bespoke addressing scheme? Why not cargo cult the documentation and/or scripts and config files from Stack Overflow or ChatGPT? Under-engineering is an easier problem to solve than over-engineering.
- bombcar 1y agoSeriously, the number of setups where "they gave us 255 VLANs, use them all!" I've seen is amazing.
- mleonhard 1y agoI took an "Architecting on AWS" class and half of the content was how to replicate complicated physical networking architectures on AWS's software-defined network: layers of VPCs, VPC peering, gateways, NATs, and impossible-to-debug firewall rules. AWS knows their customers tho. Without this, a lot of network engineers would block migrations from on-prem to AWS.
- protocolture 1y agoAges ago I deployed a sophos virtual appliance in AWS, so I could centrally enforce some basic firewall rules, in a way that my management could understand. There was only 1 server behind it, the same thing could have been achieved simply using the standard built in security rules. I think about it often. I do find Azures implementation of this stuff pretty baffling. Just in, networking concepts being digested by software engineers, and then regurgitated into a hierarchy that makes sense to them. Not impermeable, just weird.
- p_l 1y agoMain source of issues leading to overcomplex networking that I ever seen was "every VPC gets a 10./8" like approach replicated, so suddenly you have complex time trying to interconnect the networks later.
- api 1y agoIPv6 solves this but people are still afraid of it for stupid reasons. It's not hard, but it is a little bit different and there is a small learning curve to deploying it in non-trivial environments.
- p_l 1y agoAnother issue (also driving some of the opposition to v6) is the pervasive use of numerical IPs everywhere instead of setting up DNS proper.
- kc3 1y agoIt has been a while, but I have seen some switches that don't "speak" STP but still forward the BPDU messages (so other switches receive them and still detect loops through the "non-STP switch") and other switches that drop the BPDUs (so other switches/bridges cannot see them, possibly leading to a Layer 2 forwarding loop through the "non-STP switch").
- protocolture 1y agoYeah I had a TP Link in my homelab that I broke in half that did stuff like this.
- piggg 1y agoI remember in the 2000s a large-ish Telco network in the US was running ospf on an IX. A few of us on IRC did the what if? And one of us brought up the adjacency and it worked. Same network also had all their network links in MRTG public too with no auth - if you only knew the hostname/URL you could see it all (which their staff would sometimes drop in Noc communication when linking a graph and you attempted to go there to poke around).
- sevensor 1y agoI’m still a bit unclear about how an IX is situated relative to the internet and to end users. Per the article, it’s not meant to have desktops or print queues plugged into it, but it’s also a LAN. What sort of computer is meant to participate in an IX?
- tecleandor 1y agoIn an IX you have mostly routers, firewalls and servers. But sometimes people will need to connect a "personal" device to the internet or the internal servers network, and instead of using a NATed/firewalled zone, they will use a public internet port (mostly by mistake), and then that kind of stuff can happen.
- Palomides 1y agoit's a network for ISPs and organizations with big networks to hook together to let traffic flow more directly, mostly only routers talking BGP
- pumplekin 1y agoThe idea of an IX, or IX peering LAN is simple in concept. It is a LAN (a flat, layer2 network), to which multiple ISP's can plug in routers. Like your home LAN might have 192.168.0.1 = router, 192.168.0.2 = laptop, 192.168.0.3 = phone etc, a peering LAN will have things like 195.66.224.21 = HurricaneElectric, 195.66.224.22 = NTLI, 195.66.224.31 = Akamai, 195.66.224.48 = Arelion etc ... So instead of all these ISP's that want to exchange traffic with each other having to assign ports and run cables in a full mesh (which quickly would get out of control), everyone connects to the "big switch in the middle" with that peering LAN on it, and they use that. Back in the day, that might have been an actual single big switch, or a stack of switches. Now IXP infrastructures are much more complex, but the presentation to the end user is usually still a cable (or bundle of cables) that goes into something that looks to them like a "big switch". There is a LOT more to know about this space (Peering vs Transit, PNI's, L3 internet exchanges, what Google are doing by withdrawing from IXP's), but I wanted to write a comment that didn't turn into an essay.
- q3k 1y agoIt's a big peering LAN (not dissimilar to a LAN party, but you know, with equipment costing more than a nice car), scaling from a switch in a rack somewhere (eg. FCIX [1]) up to hundreds of switches in dozen of locations not even in the same city anymore (eg. DECIX, AMSIX). You connect to the IX either over a cross connect cable from some equipment nearby (in the same DC or same building) or across town via some leased line / dark fiber / lambda / etc. But usually what's at the other end of a connection to an IX is your edge router, on which you will then run BGP to all the other folks on the IX. However, it's not always this simple - for example, you might have another switch-like device in between the router and the IX, to maintain some level of flexibility for your own services, or because you're actually being bundled together alongside multiple customers into IX access by some provider. You might actually go into a whole MPLS backbone first, because that's how your provider is selling you transport to the IX. Or you might've set up some peering LAN bridging on your router to set up some hot standby and then plugged it into some switch for convenience to run it across the office LAN to your desk and then... What ends up happening, is with this amount of complex network devices along the way, and with how network equipment is generally provisioned (you SSH into it and then you do some mutable changes, then you remember the update the orga docs), mistakes happen. In a critical case of misconfiguration (also stale configuration and VLAN identifier reuse) it's not unimaginable for an IX peering LAN to then get accidentally bridged into some VLAN that actually reaches an old Windows laptop that was at some point plagged into another VLAN for troubleshooting. This is especially likely for customers that co-locate their office equipment, AD servers, web servers and edge router in the same building across the same infrastructure. [1] - FCIX is also a good source of how this sort of stuff actually looks like IRL, instead of the bullshit marketing 'rack of neat racks' renders on providers' websites: https://pbs.twimg.com/media/FIsd-JsVgAQLUh9.jpg?name=orig https://pbs.twimg.com/media/FIsd-JsVgAQLUh9.jpg?name=orig https://pbs.twimg.com/media/FP72xaiWQAsyPUM.jpg?name=orig https://pbs.twimg.com/media/FP72xaiWQAsyPUM.jpg?name=orig
- liotier 1y agoI wish everyone used LLDP everywhere: it is harmless and immensely helps in finding the correct spaghetti in the plate.
- madsushi 1y agoAll surface area carries risk, and LLDP is additional surface area when exposed on a public IX.
- unit149 1y ago[dead]
- wmf 1y agoLLDP and STP are between the host and the switch and switches don't forward them. Why would you see that at an IX?
- bcrl 1y agoThis is why IX like Torix lock down switch ports to a single MAC address and limited protocols (ARP, IPv4, IPv6). It's a bit of a pain when first setting up peering, but it reduces the exposure for all those connected to the IX. The reliability track record clearly demonstrates how worthwhile the strategy is.
- egberts1 1y agoCosmic Analogy of an Internet Exchange (IX) - The IX as a Galactic Core: The IX is like a gravitational hub at the center of a galactic cluster. It doesn’t create stars (data) itself, but it bends spacetime (network topology) in such a way that galaxies (ISPs) can orbit and interact efficiently without needing to sling data across the intergalactic void (expensive transit). - ISPs as Stars/Planets: Each ISP is like a stellar system with its own mass (bandwidth capacity, user base). They orbit into the IX “gravitational well” by sending out fiber-optic “space-time bridges” (links) to dock into the exchange. - Fiber Links as Wormholes: The optical fibers into the IX are essentially wormholes: direct, high-speed tunnels across spacetime that collapse distance. Instead of traveling light-years through transit (Tier-1 providers), local traffic jumps instantly between ISPs through the IX’s wormhole endpoints. - Switch Fabric as the Interstellar Medium: The IX switching fabric is like a plasma medium in the galactic core — it doesn’t decide where stars go, it just provides the environment where matter (packets) can move around without collision. It’s neutral, like the intergalactic medium, just enabling the exchange. - BGP as Gravity’s Equation: Just like Einstein’s field equations determine how mass curves spacetime, BGP policies determine how traffic paths curve through the IX. Each ISP “publishes its mass” (advertises prefixes), and neighboring ISPs decide how to route packets based on that gravitational influence. - Local vs. Transit: Without an IX, traffic between two galaxies (ISPs) might have to pass through a supercluster-scale filament (Tier-1 backbone). With an IX, they just exchange starlight (packets) directly within the cluster, conserving energy and reducing latency — much like galaxies trading radiation locally instead of sending it across megaparsecs. So in astrophysics terms: an IX is a galactic core enabling local wormhole trade routes, where BGP is the gravity law and fibers are Einstein-Rosen bridges connecting stellar systems (ISPs) into a dynamic, orbiting cluster.
- egberts1 1y agoInternet Exchange (IX) for an 8th Grader - Imagine a big hangout spot in the middle of town where all the school buses from different schools meet. - Each bus is an Internet company (ISP), carrying kids (data) who need to visit kids from other schools. - Without the hangout spot, the buses would have to drive all the way across the city through downtown traffic (expensive, slow routes through bigger Internet companies). - But at the hangout spot (the IX), the buses can just pull up next to each other and drop kids off directly -- quick, cheap, and easy. - The IX itself doesn’t tell kids where to go, it’s just the place with lots of parking spots and connections. - The ISPs (buses) still decide who gets on and off by making agreements with each other. That’s like them saying, “Okay, I’ll let your kids ride with me if you let mine ride with you.” So, an IX is basically a meeting place where Internet companies connect their networks directly, instead of going the long way around through someone else.