12 ms·
Heartbreaking: 6509 PSUs Fail After a Brief 22 Years of Service
- jrrtokenring 5y agoRESEARCH TRIANGLE PARK, NC, CISCO SYSTEMS INC 25-Dec-2021 After only 22 years of service, a 6509 chassis deployed at a customer location has reportedly reached its final end of life. Taken from us in its prime, lon1-gandalf01, known simply as “Gandalf” by those close to it, put in its final notice after the power outage at Interexion’s LON1 facility killed off the last working PSU. Network Engineer Rich Ikhanda, the technician tasked with placing gandalf into its final resting box, stated “it’s a real shame to see this router taken out of service while it still had its youth. I mean, it was barely out of its teens! A true tragedy.” Rich reminisced that “once we disabled that bastard protocol, IPv6, we could easily fit the entire v4 table into Gandalf’s TCAM with plenty of room to spare, until everyone else had the same idea and even the “real” internet table grew too large, referencing the IPv4 default free zone. Luckily we were able to slice, dice, and julienne using as-paths, like real network engineers.”. Gandalf will be replaced by a UniFi USG-Pro nicknamed “Gandalf the White”. When asked if there were any concerns with Gandalf’s replacement, Rich stated that “I am not sure about this whole app thing to run a network, but it has gigabit interfaces, it should be good to go.” Gandalf is survived by nearly one million IPv4 routes and a 6513-E named Frodo, all of whom will feel the impact from this loss (as they continue to reconverge).
- deleted 5y ago[deleted]
- wmf 5y agoThe really sad part is that Cisco keeps selling 10-year-old obsolete chassis when newer systems have so much lower cost per bit.
- jrrtokenring 5y agoWhy innovate, when you can stagnate!
- smarx007 5y agoCisco stopped selling them all: https://www.cisco.com/c/en/us/obsolete/switches/cisco-catalyst-6509-switch.html https://www.cisco.com/c/en/us/obsolete/switches/cisco-cataly... and https://www.cisco.com/c/en/us/support/switches/catalyst-6500-series-switches/series.html https://www.cisco.com/c/en/us/support/switches/catalyst-6500... (2006 and 2020 respectively). The post is the "onion".
- testrun 5y agoI absolutely love your username.
- jrrtokenring 5y agoThank you :)
- faeyanpiraat 5y agoMy friend asks what is it to love about it, he doesn’t get it?
- iso1631 5y agoJRR Tolkin wrote Lord of the Rings Token Ring was an old networking technology (before ethernet became ubiquitous)
- barbazoo 5y agoPSU is the power adapter?
- jrrtokenring 5y agoYes, PSU is the Power Supply Unit. They are known for failing when they are turned back on after a power is lost.
- tialaramex 5y agoThis is why your 6509 should have dual PSUs.
- richardfey 5y agoIn my case also after turning off and on the building main power switch, after a brief 12 years of service.
- geocrasher 5y agoMy first thought was "MAN... that's a lot of PSUs"
- spicybright 5y agoEveryone knows you measure PSU life in dog years. 154 years is a damn good run!
- par 5y agoHow many servers out there named gandalf? I've got one in my own house.
- nine_k 5y agoI had one in our org back in the day. We had a number of boxes named following this theme. For instance, the firewall box was named pelennor.
- k_sze 5y agoAll of my network devices have JAV actress names, lol.
- vincnetas 5y agoIs this story by any chance about you? Every variable and every method name was given a female first name. "Each of these is named for one of Sten's girlfriends." Given the number of names required, it was improbable that these were real girlfriends, but Sten gave no hint about this being fiction. https://thedailywtf.com/articles/my-many-girlfriends https://thedailywtf.com/articles/my-many-girlfriends
- kQq9oHeAz6wLLS 5y agoMy first ISP used female names for their Sun servers, reportedly ex-girlfriends of the owner.
- dendriti 5y agowell, that's gross
- na85 5y agoCringe intensifies.
- no_time 5y agoMay their filesystems get the fsck'ing they need
- TradingPlaces 5y agoThis whole blog is a hoot. https://www.jumboframeinternet.com/post/2/ https://www.jumboframeinternet.com/post/2/
- mplabelspace 5y agoI suspect there may be semi-regular updates =)
- ezconnect 5y agoThe home page have redirect problems on Firefox and cant load.
- jrrtokenring 5y agoThank you :)
- 6LLvveMx2koXfwn 5y agoSatire is fine but this 'nice' thing is really galling.
- jrrtokenring 5y agoBreaking: Anonymous HN reader really shows anonymous shitposter how to live their life
- 5y ago
- RantyDave 5y agoTaken so young, the poor parents.
- jrrtokenring 5y ago"The router that routes twice as fast routes half as long"
- wmf 5y agoHas it seen packets you wouldn't believe?
- thrtythreeforty 5y agoTCP streams glittering in the dark near the Tanenbaum gate
- DonHopkins 5y agoNOTE: This packet is sold by net wait, not by volume. Packed as full as practicable by modern automatic equipment, it was delayed the full net wait indicated. If it does not appear full when opened, it is because contents have been compressed during shipping and handling.
- lostlogin 5y agoIf Gandalf the White is anything like my UDM-Pro, I hope the backups are good. Mine self erases the config periodically, most recently 10 hours ago.
- octoberfranklin 5y agoUbiquiti's Octeon hardware is amazing -- but only after you expunge the horrid "bonus software" that they throw in there for free.
- jrrtokenring 5y agoBut I mean... gigabit interfaces it must be good!
- mise_en_place 5y agoI wish I still had gigabit interfaces on my workstation. dmesg is constantly complaining about my 10 GigE interfaces
- lostlogin 5y agoI use the 10gb, and it’s on a UPS. Tempted to move back to an edgerouter. Or go Pfsense.
- ornornor 5y ago> Pfsense Check opnsense out. It’s like pfsense but better and not toxic.
- topspin 5y agoIronically there is a good chance your problem is also power supply related.
- lostlogin 5y agoMaybe, it’s on a UPS, but that doesn’t mean it isn’t the power.
- ssklash 5y agoWhat's crazy to me is that a massive 6509 could be replaced by a tiny UniFi USG-Pro.
- belfalas 5y agoDear OP, my username bows to yours.
- rlt 5y ago"J.R.R. TokenRing"
- azalemeth 5y agoFun fact: in the University of Oxford (former home of one JRR Tolkien) each building has a router and switch cabinet where JANET comes into the building and is distributed. What is this called? Well, it's the Front Door to the internet, or Frodo for short...
- _8j50 5y agoThe 6509 and 2811 and other older models are amazing. I use catalyst switches and routers I bought from ebay and craigslist almost 10 years agi that were probably resold several times after production use. I am no longer in networking but I was always impressed when I ssh into some arbitrary switch or router to troubleshoot and see 10+ year uptime. I bet 20+ is common these days. I am sure others have stories in this regard. Someone told me their remote router died after 12 years uptime but only because of a lightning strike. From a software perspective, the Cisco IOS has been doing "unikernel" before it was hyped. Can't say that's a factor but having a standalone binary as the entire OS might perhaps make testing and shipping a sturdy stack easier?
- netfl0 5y agoCame here looking for reminiscing. You could do wild things with the right 6500 setup. It was like driving a manual transmission.
- iso1631 5y agoI'm embarrased to say I have some cisco 3750 edge switches with 10+ year uptimes. They are isolated at a management level but really should have their firmware upgraded. Arranging downtime with 30 different stakeholders who don't subscribe to the concept of multiple services capable of failure is never easy.
- manbart 5y agoAFAIK all the newer Cisco IOS boxes are running Linux now, i.e. IOS XE
- _8j50 5y agoThey are. All the Nexus/nxos line as well. You don't hear people rave about their reliability as much as IOS.
- octoberfranklin 5y ago> Gandalf will be replaced by a UniFi USG-Pro Octeons are truly amazing chips. Get them while you still can.
- raverbashing 5y agoI'm surprised no one with a white beard pulled out a soldering iron out of their drawer and tried to fix it. Or maybe they did...
- peterkelly 5y agoI love how many of the comments here think this story is real, and not just the first article published by what appears to be a newly-launched satire tech news site inspired by The Onion. Follow the "prev page" links at the bottom for other articles, you won't be disappointed :)
- tempodox 5y agoAnd we miraculously see that site although “Jumbo frames don't work on the internet!” :)
- deleted 5y ago[deleted]
- xbar 5y agoIt's a good site and I'm not disappointed. But 6509 longevity is real and the best satire mirrors our real experiences. Was there ever a gandalf 6509? Yeah. Probably 10,000 of them and many are 22 years old. My 22 year old gandalf/bilbo pair were actually VA Linux hosts. But no doubt lots of 6509s were so named.
- deleted 5y ago[deleted]
- newaccount74 5y agoNot totally on topic, but can someone explain why switched mode power supplies don't seem to have any type of inrush current limiting? Often when I turn on power right after a power loss, the circuit breaker immediately trips. I assume it's because all those power supplies have such a big inrush current that they overload the circuit. All these devices use less than an amp in total on average, but the inrush current is enough to immediately trip a 13A circuit breaker.
- fake-name 5y agoBecause that's another part that adds to the BOM cost and assembly cost. SMPSes all have input stages that effectively directly rectify the incoming mains to provide the SMPS internal bus voltage. This is also a great way to basically design an inrush current simulator (diode(s) + big capacitor). Adding stuff like active PFC correction in your common wall-wart is a non-starter. Those things are already massively cost-optimized.
- namibj 5y agoNo they have enough that they don't vote a breaker when you plug them in the first time. Also toroidal cores are worse than modern active PFC SMPSs. Also note that there are different breaker classes for different loads, with the more-forgiving ones having tighter demands on the insurance of the downstream wiring, because they have to still reliably trip instantly from a short.
- Ballas 5y agoThey should have an NTC for inrush limiting.
- newaccount74 5y agoHow long do those NTCs take to reset? Maybe that explains why the breaker trips after I turn it back on after a power failure -- if those NTCs are still warm, they would not limit the inrush current.
- formerly_proven 5y ago
- Zenst 5y agoReminds me of the fun of naming hosts before they became all formalized. Recall in the late '90s at a large Medical distributor, how we set the user/client DHCP pool names from a virus database (there are more virus's than stars in the Universe apparently). Was a nice in-house admin chuckle that AFAIK, nobody in upper management ever found out about. Our in-joke was that virus's kill hosts and the thinking was that without users, every server/service ran just 100% fine.
- hnarn 5y agoI know servers are supposed to be "cattle" but I can't get away from the feeling that having at least a part of the name being somewhat sentimental is a nice thing. The default naming scheme for many containers, for example, will combine an adjective and a noun to create a unique name. I myself wrote a pretty silly script once that randomly returned the name of a British or German naval warship. It seems to me that "formalized" naming doesn't have to be exclusive with sentimentality, you can just add it at the end, like: geotag-srvtype[#]-prod-[pokemon_name_or_whatever] That way, if your environment is completely fluid, it becomes obvious when a node has re-deployed, and if it's not, you can just re-use the old name when re-deploying instead of generating a new one. You can of course include hash-looking things in hostnames as well, but I'm a strong believer that hashes are for computers, not for humans to read (most of the time).
- tomjen3 5y agoYou totally should name the container classes after battleships. Then you can truthfull say you deleted/sunk the Bismark. Anyway I think there is nothing wrong with naming classes of things after known things - so you are deploying two more Victor class frontend servers or Dreadnaught monitoring services. The trouble becomes when you care about Dreadnaught-oux3z and treat it as not disposable.
- DonHopkins 5y agoYou could name your servers after cuts of beef, beef recipes, and hamburgers, like the WOPR Computer and Big Mac: https://www.youtube.com/watch?v=iRsycWRQrc8 https://www.youtube.com/watch?v=iRsycWRQrc8 https://en.wikipedia.org/wiki/Big_Mac_(supercomputer) https://en.wikipedia.org/wiki/Big_Mac_(supercomputer)
- infologics 5y ago[dead]
- ChrisMarshallNY 5y agoI like the screen name. There's probably a whole lot of folks, that have no idea what a "token ring" is.
- xbar 5y agoFor reference https://dilbert.com/strip/1996-05-02 https://dilbert.com/strip/1996-05-02
- franga2000 5y agoI teach computer networking in high school and it's still a part of the curriculum. I have no idea why, but it had absolutely no difference in importance between Token Ring and Ethernet, despite the latest revision of the curriculum being written in 2017.
- kloch 5y agoThere is a known hardware bug affecting every PSU and cards in the 6500/7600 series. It manifests as a significant chance of permanent failure on power-up after being on for several years. As long as the chassis stays powered everything is fine. If you reboot or power cycle you may end up with some random dead cards or power supplies. The use of the "enterprise" 6509 as a backbone ISP router was the worst kept secret in the industry in the 2003-2007 timeframe. During that time you would see about about 60% of the routers in a carrier hotel were 6500/7600's. Most of the rest were Juniper MX or M with very few Cisco 12k, ASR, or CRS. The 12k used an inefficient fixed cell size architecture and focused on ATM line cards and the CRS1 was far too expensive for most operators. The Ethernet-based 6000/6500 series was originally developed by the enterprise vertical at Cisco as an L2 product, but it gradually evolved to full route scale L3 features at a much lower price point than the "real" ISP routers. This price advantage was especially dramatic when 10GE interfaces for the platform came out in 2003. Cisco replaced the 12k with the more popluar Ethernet focused ASR series but the very low cost of the 6500 kept them in use much longer than they should have been. While dirt cheap due being officially marketed at enterprises and a robust greymarket for parts, the 6509 was far from an ideal platform for ISP's. It generally performed well but there were a number of quirks from an internal architecture that gradually evolved from L2 only->full scale L3. RP cpu/ram was woefully under powered and control plane policing was marginal at best. Any significant packet flow to the RP instead of being forwarded by the ASIC hardware would instantly cripple the router and cause it to drop all BGP/OSPF/IS-IS sessions. In addition to broken control plane policing there were more scenarios where packets could be punted to the RP unexpectedly compared to other platforms. For example if any of the TCAM partitions filled up then ASIC forwarding was immediately disabled and ALL packets were sent to the RP! You also had to watch traffic levels when round-tripping on adjacent 10GE ports on the most popular cards to ensure it didn't exceed about 6 gbps or those ports would start dropping packets. Real ISP routers can often do full line rate in+out on all ports. The monolithic IOS for the platform was notoriously buggy, partly due to the architecture and partly due to an enterprise feature set that most ISP's didn't even need. It was wise to avoid any version where the release notes claim to have just fixed a catastrophic bug (you wanted to see only boring fixes to cosmetic bugs or features you would never use) and to search the c-nsp mailing list for other's experiences with the version. Otherwise you were playing "IOS Roulette". Because of this it was common to run a known stable version much longer than normal, running the same version for 5+ years in some cases. After the fail-on-power-up bug became known most left them running without even a reboot until they could be replaced. In more recent years Arista has made a similar play by evolving their switches from originally focused on minimum L2 latency to full route scale L3, and more recently adding full MPLS support at a lower price point than the equivalent C/J products.
- sandos 5y agoOh, I read that 6502 and thought it was about Commodore 64. Was cofused that they would replace it with a router thing :)
- camb416 5y agoI understood some of these words