4 ms·
What I find really funny about Solarwinds - is that in the medium to large ISP sector it's always been an absolute joke. Nobody of any consequence or real size
by walrus01 6y ago
What I find really funny about Solarwinds - is that in the medium to large ISP sector it's always been an absolute joke. Nobody of any consequence or real size has ever used it for network monitoring.
Once you get to the scale of ISPs that have 50,000+ customers, or are supporting more than that through other smaller ISPs that are downstream of them - the monitoring and network automation tools are almost entirely open source, and some combination of GPL/LGPL/BSD/Apache/MIT license. Combined with custom things written in house to tie together different tools for a company's specific business needs.
What you'll have typically is a collection of network equipment that may have closed-source operating systems (cisco, juniper routers and switches and similar, optical transport platforms from vendors like Infinera, Ciena), but everything managing and monitoring them is open source and runs on a *nix platform.
If you have the in-house Linux/BSD knowledge to run the world's most powerful and popular open source networking tools, there is no need to ever touch solarwinds.
My job interacts on a regular basis with all of the different pieces of the puzzle that make up solutions which are, in my opinion, vastly superior to Solarwinds.
In the serious ISP business, if you ask the persons who admin the monitoring tools what they think of solarwinds, the answer you'll almost universally get back is "Windows GUI button pushing tools for enterprise end users who don't have the knowledge or motivation to really understand what's going on under the hood of their network".
- nikisweeting 6y agoOut of curiosity, can you give some examples of open-source monitoring tools that large-scale ISPs use?
- walrus01 6y agothe first and most important thing is to have the correct network architecture and engineering to make effective use of the tools, and not have a network that needs a lot of babysitting in the first place. after that's taken care of as an over-arching and continual business process: there is no one single god box piece of software that is the be-all and do-all of network management/monitoring for an ISP. Some things come close, such as LibreNMS when used as the sole tool for a small ISP. But most often it is a patchwork quilt of many different things, each used for a discrete purpose. in no particular order: opennms a combination of (influxdb + telegraf + grafana) librenms provisioning and automation tools like ansible various in house things built on traditional RRA files and rrdtool tools like netbox for keeping track of datacenter customers/hosting environments phpipam or nipap for IP address management various self-hostable wiki software packages for internal documentation various types of self-hostable ticketing systems, monitoring systems that integrate with a customized asterisk system for NOC phone workflow 4 or 5 different tools that fill the same role as smokeping wireshark lots of different things for analyzing netflow data (Elastiflow or other) ELK stack stuff, elasticsearch/logstash/kibana, customized as needed. in house setups for openstreetmap tile servers and map presentation, to pull data from back-end mariadb databases and present them on monitoring displays. GIS software like QGIS and a PostGIS backend lots of different possible things done with custom code and postgresql, mysql/mariadb, or similar if you go through the PDF slideshows for the powerpoint decks at the last 4-5 years of the NANOG, RIPE and APNIC conferences you'll see discussion of some of the most popular network automation and monitoring tools.
- gerdesj 6y agoI'm not a large scale ISP but I do have to monitor quite a lot of stuff. You'd be amazed at how much you can monitor with Nagios/Icinga(1,2). They are written in C but call a lot of external stuff written in whatever you fancy and that's the power, right there. Bodge upon bodge! There's no single technology in these beasties. The interface between the system and the plugins is very basic to say the least, so you can throw whatever at it as you require. I'm a sysadmin not a programmer and need to get jobs done. We currently use Icinga 1 with a dash of Netdisco and I intend to migrate to Icinga 2 with DIrector etc. That said, I have dallied with OpenNMS many times ever since the project began - it's too good to ignore. Zabbix also turns my head quite often.
- walrus01 6y agoopennms is simultaneously arcane weirdly laid out a massive java memory hog (thankfully, RAM is cheap, giving an opennms VM 16GB of memory isn't a big deal anymore) extremely powerful something that has 450+ pages of documentation totally open source extensible to support monitoring of massive international-scale networks
- slt2021 6y agoa lot of CIOs are very cheap and prefer to keep their IT teams "lean" (understaffed, underpaid, underskilled) as they see them as cost center, and rather hand over couple mills to a vendor like SolarWinds to install their "automation/AI" pixie dust. So instead of investing into own employees who have the best interest of a company in mind (because you know job satisfaction and job security) -> they prefer investing into third party vendor whose interest is only to keep renewing multi-mill contract year after year while keep delivering barely above what's required to keep things afloat
- deleted 6y ago[deleted]
- detaro 6y agoThat's true with pretty much any enterprise-aimed tool, isn't it? With enterprise IT generally expected to run a very mixed pile of different stuff, without having the resources to specialize much on each of these things. Whereas an ISP, SaaS, specialized hosting company, ... puts much more emphasis on specific stacks and mastery of them, and often more recognition that investing in these things is not just a cost center.
- walrus01 6y agoYes - though a big enough ISP with 40, 50 staff or more also needs a very wide range of common enterprise software tools, which the people running must have full mastery of. You've got stuff going on like billing/accounting systems, call centres, GIS systems for outside plant fiber construction and aerial+underground utilities work, HR software, VoIP systems, IDS and NAC systems. Lots of things that support the ordinary office-worker environment of the ISP in addition to all of the tools that automate and monitor the network.
- Spooky23 6y agoI don’t know what you consider a “serious ISP”, or what serious people do, but I’m confident that 10/10 of the big ISPs have Solarwinds on their corporate network. The customer facing stuff may be different, but once you own the LAN, you own the company. Their play was a cheaper, easier, multi-vendor toolset for enterprise networks. You’d pay half of whatever the Cisco dreck costs, and not need an army of consultants to tend it. I’d argue that the vast majority of network people do not demonstrate strong Unix skill sets. Windows tools FTW in most enterprises, as dumb as that may be.
- walrus01 6y ago> I don’t know what you consider a “serious ISP” Something that's big enough and has a wide enough enough reach that other network operators with presence at major IX points know its AS number by sight - the same way people will recognize AS174 as Cogent or AS1299 as Telia, for instance. Or an ISP that is big enough that its wholly-owned/controlled fiber network spans most of a state, or several states, and has other major ISPs riding on it (whether as lit 10/100G customers, or dark fiber IRUs, or whatever). Something big enough to have a whole team of guys with bucket trucks and fiber equipment running around building the physical internet, while at the same time there's an office/work-from-home environment with 4 or 5 people whose job title has some form of "network engineer" in it, building the network at OSI layers 2/3. Or for an ISP that is not middle-mile/last-mile focused, and is rather a hosting/colocation company, something with significant datacenter presence at or near major IX points, as measured in square feet of space leased, kW of electrical power and cooling. > I’m confident that 10/10 of the big ISPs Which ASes would those be? If you can find a documented instance of a top-50 (by CAIDA ASRANK size) ISP using solarwinds to run its core stuff, please provide a reference to it... https://asrank.caida.org/asns https://asrank.caida.org/asns
- vitus 6y agoThe most likely source of said claims would be something like SolarWinds's customer list, which they took offline in December. http://web.archive.org/web/20201214030038/https://www.solarwinds.com/company/customers http://web.archive.org/web/20201214030038/https://www.solarw... "Our customer list includes: ... All ten of the top ten US telecommunications companies" I see... AT&T, Sprint, Comcast, Level 3 (now CenturyLink, still AS3356) for US-based ISPs. Telecom Italia made the shortlist, too. (And an honorable mention for Cisco, which was also apparently explicitly targeted: https://tools.cisco.com/security/center/resources/solarwinds_orion_event_response; https://tools.cisco.com/security/center/resources/solarwinds... "While Cisco does not generally use SolarWinds for its enterprise network management or monitoring, we have isolated and removed the Orion installations from a small number of Cisco assets.") That said, it doesn't say anything about _how_ the ISPs were using SolarWinds, just that they were in some capacity. But with any infiltration, it doesn't matter if it's widely used, so long as it's used somewhere that can be used as a launching pad for a follow-up attack. edit: apparently CenturyLink rebranded as Lumen last year. They're still AS3356 (and its subsidiary networks) to me.
- Sylamore 6y ago>Nobody of any consequence or real size has ever used it for network monitoring. At least one of the big 3 telcos uses it very extensively for network monitoring, inventory, device configuration enforcement and alert generation.
- walrus01 6y agoI am much less surprised to hear that in the context of a company that is a "baby bell" / ILEC such as Verizon, Frontier, Centurylink (Former embarq/uswest/qwest/whatever), than I would be if I heard that they were using solarwinds inside NTT or Telia. Without going into a whole lot of personal opinion and detail, the business practices and management methodologies in a ILEC are very different from other ISPs.