5 ms·

Hoo boy, most of these things don't worry me, but this one does. I'm semi-responsible for some Juniper gear, thankfully all Junos (BSD) based, but I no longer

Hoo boy, most of these things don't worry me, but this one does. I'm semi-responsible for some Juniper gear, thankfully all Junos (BSD) based, but I no longer trust any of it if this is malicious injection vs. a bad review. However, what the hell can I do? I can't audit the code. I trusted Juniper, and now I'm stuck with that trust being burned. Running to any other proprietary network vendor is just as uncertain. If Junos gets a bulletin, I have a lot of work on my hands very soon, as do a good chunk of service providers. I remember there being rumors of a certain three-letter agency saying they had some type of exploit for the Cisco ASA as well; I wonder if it was something this deep, vs. just a run of the mill RCE vuln. This is one more reason to use open-source products for actually security-sensitive systems, maintain a good amount of defense in depth, and do a little bit of auditing of the code you're using yourself. More often than not these days, it sure pays to be paranoid. EDIT: At the same time, this also really makes me respect Juniper more than I have previously. A company that finds this internally, on their own audit, could have patched it silently and said nothing about it to anybody. It probably would have been better for them PR-wise. The honesty is worth me not jumping ship to another (probably compromised) proprietary vendor, but you betcha if I can get away with it, I'll run something open-source and community audited when I can.

jlgaddis 11y ago
If JunOS gets a bulletin, the whole Internet has a lot of work on its hands. If JunOS gets a bulletin, shit, that's really, really bad. I feel it's very likely that the NSA had something to do with the recent Cisco ROMMON "discoveries", so it would not surprise me one iota if they were involved here as well (although it's obviously pretty early to speculate on something like that -- and impossible to {dis}prove).
- adrtessier 11y ago
  I am eagerly awaiting the incident report on this, although I find it unlikely we'll ever hear anything more than this from JTAC and friends. If this is the work of an intelligence agency, it will likely be gagged under the guise of "national security" unless there's a press-worthy indictment coming out of it.
x5n1 11y ago
Software based networking on BSD. Can't trust American vendors for anything. Is this what you wanted Mr. NSA? Good job at sabotaging your own business interests. It's not that you're spying, it's that you're so promiscuous about it.
- adrtessier 11y ago
  Don't sell JunOS short. It is far more complex than "software networking on BSD" and has a lot of proprietary bits. Junos (FreeBSD) is the Routing Engine; Juniper hardware also contains an ASIC-based Packet Forwarding Engine, which loads microcode from the Routing Engine upon boot. Not everything's in Junos all the time, but since the PFE loads its embedded OS from the Routing Engine kernel, you could just pwn the Routing Engine and then also have some sense of persistence in the PFE on reboot, probably. I don't know much about how the PFEs work internally. I'm certainly no FreeBSD/JunOS expert. I am an unabashed fanboy of JunOS's *nix-y structure, though, vs. the monolithic binary that is IOS. (There was a great Blackhat 2011 talk on IOS reverse engineering, if you are interested in that sort of thing. [1]) [1] [PDF Warning] https://media.blackhat.com/bh-eu-11/Sebastian_Muniz/BlackHat_EU_2011_MunizOrtega_Cisco_iOS-WP.pdf https://media.blackhat.com/bh-eu-11/Sebastian_Muniz/BlackHat...
  - qb45 11y ago
    I think your parent didn't mean that "JunOS is just software networking on BSD" but that "software networking on BSD is all that can be trusted because NSA screws commercial products". And of course it doesn't have to be NSA. Maybe some foreign spies or pretty much anybody interested in spying on some Juniper's customers. Or even a bored employee doing it for bragging rights. FWIW, I once worked for a (reasonably big) corp making software which has to run as root and I'm pretty sure I'd have been able to slip some small privilege escalation backdoor in there if I felt like doing so. But I have to admit that their products weren't as security critical (and, actually, already had some vulns), so one could hope that Juniper and Cisco are better than that.
lmm 11y ago
I've always regarded networking equipment as outside my security boundary. All it does is forward packets to the right places; an attacker can deny service by shutting that down or sending them to the wrong place, but nothing else. All my connections are encrypted and authenticated at a higher level. Do you terminate SSL or something on yours? Or have open unauthenticated services running on your internal network? If not, what's the actual threat here?
- ghshephard 11y ago
  Lots of enterprises use JunOS to terminate VPNs.
  - ra1n85 11y ago
    Nit: ScreenOS is the impacted network operating system, not JunOS.
- technion 11y ago
  - Plenty of services that can't adequately be secured any other way are often mitigated by restricting access to VPN users coming in over a network device. I appreciate that "just secure the service" is supposed to be the best practice, but when you're talking about things like IPMI interfaces or SCADA devices the alternatives approache zero - Controlling the networking equipment can open you up to things like sslstrip
  - rsync 11y ago
    "things like IPMI interfaces or SCADA devices the alternatives approache zero" My strategy with IPMI has been to assign IPMI non-routable, private IP addresses, then block that address space at the interior of the network (which is sort of redundant) and then require folks to SSH onto an interior host and connect to IPMI that way. I would be very interested in, and receptive to, criticisms of this model ...
    - technion 11y ago
deleted 11y ago
[deleted]
chei0aiV 11y ago
Here is a Linux (Debian) based and more open similar thing: https://cumulusnetworks.com/ https://cumulusnetworks.com/
- darkr 11y ago
  It's not really much more open. Similar to some other network hardware (e.g extreme networks) it runs linux for the management interface. The switching/routing ASICs that does the actual work are still proprietary. It allows for a bit more flexibility in that it supports apt, but IMHO an image-based update system is preferable on network hardware, as you are not vulnerable to apt-breakage or anything like that. You know, whatever happens it will boot, always.
rodgerd 11y ago
> some type of exploit for the Cisco ASA as well Given ASAs run a 2.6 kernel that's not hard. From my Kiwicon 8 notes on Alec Stuart-Muirk's talk: * Literally every protocol handler has CVEs against it. * Every time Cisco add a new one it gets at least a DOS CVE. (There are some proofs of concept for pivoting these into real exploits on other Cisco products.) * The ASA’s high availability protocols are unauthenticated and unencrypted. This is bad. Like, “will accept any packet claiming to be a management packet as valid” bad. * Some authentication is optionally available, but if you enable it, the ASA will still accept unauthenticated protocols.
- BuildTheRobots 11y ago
  I this this might be the slides [1] -though if anyone has a video of the talk I'd love to watch it. [1] https://ruxcon.org.au/assets/2014/slides/Breaking%20Bricks%20Ruxcon%202014.pdf https://ruxcon.org.au/assets/2014/slides/Breaking%20Bricks%2...
- jlgaddis 11y ago
  Because they can (allegedly) survive software upgrades (on the ASAs and IOS routers), I've always believed that these "infections" are done at a lower level than the OS, such as in the ROMMON on the IOS routers. After hearing about "SYNful Knock" recently, I'm inclined to believe this even more.
dingo_bat 11y ago
Have a look at http://openswitch.net/ http://openswitch.net/. Full disclosure: I work at HPE, though not in the Openswitch team.
- deleted 11y ago
  [deleted]
- tw04 11y ago
  That just means the exploits are pushed to the firmware blobs. Sure, it's some level better than something like JunOS, but just barely. What's the point of locking your front door and leaving the window open?
  - kijiki 11y ago
    They don't even need to go after the firmware: Openswitch has a binary-only userspace process that talks to the hardware. That is where a bad actor would hide something nefarious.
    - dingo_bat 11y ago
      But you can take the OS and write your own driver for your own hardware. Also, you can take the OS as-is, run it on a VM, and examine the packets coming out for signs of any telemetry or any such nefariousness. I think it is a big step ahead of any other switch/router OS.
      - jlgaddis 11y ago
        That's not really any different than putting a tap on both sides of one of these NetScreens, for example, and looking at the traffic. It's still impossible to "verify" a device isn't compromised that way. Only when the public can view (ALL) the code, rebuild it, and install it on their own hardware can we be reasonably confident it has not been tampered with.
      - kijiki 11y ago
        None of the current forwarding ASIC vendors publish enough information to write a driver for their chips. So no, you can't write your own driver. The reason OpenSwitch's OpenNSL binary is binary-only is because it uses Broadcom's proprietary SDK, and programs registers that are only available under Broadcom's NDA. If you're just going to run it in a VM, you'd be better off with more established things like OpenBSD, OpenWRT, pfSense, Cumulus VX, or even just a Debian VM with Quagga. <disclosure>I'm CTO of Cumulus, whose network OS is exactly as open-source as OpenSwitch: everything but the single user-space program that links to Broadcom's SDK.</disclosure>

jlgaddis 11y ago

If JunOS gets a bulletin, the whole Internet has a lot of work on its hands. If JunOS gets a bulletin, shit, that's really, really bad. I feel it's very likely that the NSA had something to do with the recent Cisco ROMMON "discoveries", so it would not surprise me one iota if they were involved here as well (although it's obviously pretty early to speculate on something like that -- and impossible to {dis}prove).

adrtessier 11y ago
I am eagerly awaiting the incident report on this, although I find it unlikely we'll ever hear anything more than this from JTAC and friends. If this is the work of an intelligence agency, it will likely be gagged under the guise of "national security" unless there's a press-worthy indictment coming out of it.

x5n1 11y ago

Software based networking on BSD. Can't trust American vendors for anything. Is this what you wanted Mr. NSA? Good job at sabotaging your own business interests. It's not that you're spying, it's that you're so promiscuous about it.

adrtessier 11y ago
Don't sell JunOS short. It is far more complex than "software networking on BSD" and has a lot of proprietary bits. Junos (FreeBSD) is the Routing Engine; Juniper hardware also contains an ASIC-based Packet Forwarding Engine, which loads microcode from the Routing Engine upon boot. Not everything's in Junos all the time, but since the PFE loads its embedded OS from the Routing Engine kernel, you could just pwn the Routing Engine and then also have some sense of persistence in the PFE on reboot, probably. I don't know much about how the PFEs work internally. I'm certainly no FreeBSD/JunOS expert. I am an unabashed fanboy of JunOS's *nix-y structure, though, vs. the monolithic binary that is IOS. (There was a great Blackhat 2011 talk on IOS reverse engineering, if you are interested in that sort of thing. [1]) [1] [PDF Warning] https://media.blackhat.com/bh-eu-11/Sebastian_Muniz/BlackHat_EU_2011_MunizOrtega_Cisco_iOS-WP.pdf https://media.blackhat.com/bh-eu-11/Sebastian_Muniz/BlackHat...
- qb45 11y ago
  I think your parent didn't mean that "JunOS is just software networking on BSD" but that "software networking on BSD is all that can be trusted because NSA screws commercial products". And of course it doesn't have to be NSA. Maybe some foreign spies or pretty much anybody interested in spying on some Juniper's customers. Or even a bored employee doing it for bragging rights. FWIW, I once worked for a (reasonably big) corp making software which has to run as root and I'm pretty sure I'd have been able to slip some small privilege escalation backdoor in there if I felt like doing so. But I have to admit that their products weren't as security critical (and, actually, already had some vulns), so one could hope that Juniper and Cisco are better than that.

lmm 11y ago

I've always regarded networking equipment as outside my security boundary. All it does is forward packets to the right places; an attacker can deny service by shutting that down or sending them to the wrong place, but nothing else. All my connections are encrypted and authenticated at a higher level. Do you terminate SSL or something on yours? Or have open unauthenticated services running on your internal network? If not, what's the actual threat here?

ghshephard 11y ago
Lots of enterprises use JunOS to terminate VPNs.
- ra1n85 11y ago
  Nit: ScreenOS is the impacted network operating system, not JunOS.
technion 11y ago
- Plenty of services that can't adequately be secured any other way are often mitigated by restricting access to VPN users coming in over a network device. I appreciate that "just secure the service" is supposed to be the best practice, but when you're talking about things like IPMI interfaces or SCADA devices the alternatives approache zero - Controlling the networking equipment can open you up to things like sslstrip
- rsync 11y ago
  "things like IPMI interfaces or SCADA devices the alternatives approache zero" My strategy with IPMI has been to assign IPMI non-routable, private IP addresses, then block that address space at the interior of the network (which is sort of redundant) and then require folks to SSH onto an interior host and connect to IPMI that way. I would be very interested in, and receptive to, criticisms of this model ...
  - technion 11y ago

deleted 11y ago

[deleted]

chei0aiV 11y ago

Here is a Linux (Debian) based and more open similar thing: https://cumulusnetworks.com/ https://cumulusnetworks.com/

darkr 11y ago
It's not really much more open. Similar to some other network hardware (e.g extreme networks) it runs linux for the management interface. The switching/routing ASICs that does the actual work are still proprietary. It allows for a bit more flexibility in that it supports apt, but IMHO an image-based update system is preferable on network hardware, as you are not vulnerable to apt-breakage or anything like that. You know, whatever happens it will boot, always.

rodgerd 11y ago

> some type of exploit for the Cisco ASA as well Given ASAs run a 2.6 kernel that's not hard. From my Kiwicon 8 notes on Alec Stuart-Muirk's talk: * Literally every protocol handler has CVEs against it. * Every time Cisco add a new one it gets at least a DOS CVE. (There are some proofs of concept for pivoting these into real exploits on other Cisco products.) * The ASA’s high availability protocols are unauthenticated and unencrypted. This is bad. Like, “will accept any packet claiming to be a management packet as valid” bad. * Some authentication is optionally available, but if you enable it, the ASA will still accept unauthenticated protocols.

BuildTheRobots 11y ago
I this this might be the slides [1] -though if anyone has a video of the talk I'd love to watch it. [1] https://ruxcon.org.au/assets/2014/slides/Breaking%20Bricks%20Ruxcon%202014.pdf https://ruxcon.org.au/assets/2014/slides/Breaking%20Bricks%2...
jlgaddis 11y ago
Because they can (allegedly) survive software upgrades (on the ASAs and IOS routers), I've always believed that these "infections" are done at a lower level than the OS, such as in the ROMMON on the IOS routers. After hearing about "SYNful Knock" recently, I'm inclined to believe this even more.

dingo_bat 11y ago

Have a look at http://openswitch.net/ http://openswitch.net/. Full disclosure: I work at HPE, though not in the Openswitch team.

deleted 11y ago
[deleted]
tw04 11y ago
That just means the exploits are pushed to the firmware blobs. Sure, it's some level better than something like JunOS, but just barely. What's the point of locking your front door and leaving the window open?
- kijiki 11y ago
  They don't even need to go after the firmware: Openswitch has a binary-only userspace process that talks to the hardware. That is where a bad actor would hide something nefarious.
  - dingo_bat 11y ago
    But you can take the OS and write your own driver for your own hardware. Also, you can take the OS as-is, run it on a VM, and examine the packets coming out for signs of any telemetry or any such nefariousness. I think it is a big step ahead of any other switch/router OS.
    - jlgaddis 11y ago
      That's not really any different than putting a tap on both sides of one of these NetScreens, for example, and looking at the traffic. It's still impossible to "verify" a device isn't compromised that way. Only when the public can view (ALL) the code, rebuild it, and install it on their own hardware can we be reasonably confident it has not been tampered with.
    - kijiki 11y ago
      None of the current forwarding ASIC vendors publish enough information to write a driver for their chips. So no, you can't write your own driver. The reason OpenSwitch's OpenNSL binary is binary-only is because it uses Broadcom's proprietary SDK, and programs registers that are only available under Broadcom's NDA. If you're just going to run it in a VM, you'd be better off with more established things like OpenBSD, OpenWRT, pfSense, Cumulus VX, or even just a Debian VM with Quagga. <disclosure>I'm CTO of Cumulus, whose network OS is exactly as open-source as OpenSwitch: everything but the single user-space program that links to Broadcom's SDK.</disclosure>