7 ms·
We tried to make an internal IoT device using Ubuntu Core and snaps because the capabilities of it were very promising. We started a PoC and about halfway throu
by beckler 6y ago
We tried to make an internal IoT device using Ubuntu Core and snaps because the capabilities of it were very promising. We started a PoC and about halfway through we hit a major roadblock. Our enterprise network does certificate substitution, and Ubuntu Core absolutely does not allow you to install your own certificates globally, so our devices would never receive updates. We tried EVERY hack we could think up, short of making our own core snap. We talked to Canonical about it, and they didn't seem interested in our fixing our complaints without a massive amount of money, so our PoC died, and we dropped Ubuntu entirely because of it.
- comboy 6y agoWhat did you end up using?
- beckler 6y agoWe ultimately used a custom yocto image.
- adrianN 6y agoSounds like a solid decision from Canonical tbh. Certificate substitution is terrible.
- kanox 6y agoMITM should be completely illegal. Why does this even need to be stated?
- ccmcarey 6y agoSeriously, you can think of no reasons? Many competent companies MITM employee traffic to scan for malware, leaking confidential data, etc.
- kazinator 6y agoGrandparent comment by beckler says they were trying to make some IoT product. That will be deployed in situations where that happens; if your customer has a MITM set up, you just nod your head and sell them something that works in that setup. You can't say, "MITM should be illegal, please buy my non-auto-updating solution anyway and stop it with your MITM." Good thing beckler found this while eating their own dogfood due to their own network being that way. Imagine that everything worked fine in their environment and then so customers came back with this issue. Then they would be beavering away hacking up their own core snap or whatever.
- cortesoft 6y agoShould you not be able to MITM your own machines?
- kanox 6y agoOnly in dev environments. Not on machines used by employees.
- diffeomorphism 6y agoNot if other people are using them. There are different value tradeoffs in different countries. The US says it is okay to spy on employees for no reason at all as long as you use company equipment. The EU says that employees like every other human being have rights and you better have a good reason and do so in a respectful way and be clear about it.
- MPSimmons 6y agoFor some situations, it's called for, but it's a huge pain in the ass. I am in a similar situation, and I need to patch every docker image I use. It's terrible to deal with, as an engineer, but the information security team does catch and eliminate a lot of content-based attacks.
- compuguy 6y agoI agree its a pain. It also makes things like working with other private certificate authorities (DoD Cert authority, other private certs) a pain. I spent a decent amount of time trying to get certain work/project related sites whitelisted from our MITM proxy because it didn't recognize the certificate chain...
- GekkePrutser 6y agoIn your own company you're free to do what you want. I can understand the reason for this. Now that most suppliers treat their devices as 'black boxes' and call home to install updates whenever they want, the security team no longer has visibility nor control over this. So much stuff runs Linux which we don't manage but still has to have full access to our network. And public repositories have been compromised and spread malware in the past. So yeah I totally understand this, even though as an enterprise Admin it's a total PITA to manage the root CAs.
- kanox 6y ago> In your own company you're free to do what you want. No, it's corporate MITM specifically which should be illegal.
- deleted 6y ago[deleted]
- beckler 6y agoNot gonna disagree at all, but I don't see any widespread adoption from enterprises because of it. It's disappointing because Ubuntu Core is actually quite secure, and we were really impressed with it... we just couldn't use it.
- Spooky23 6y agoFor a consumer, sure. In a business setting, it seems irresponsible to be to allow every random server to have an un-inspectable VPN.
- GekkePrutser 6y agoYep I couldn't imagine our Fortune 500 company ever allowing access without it.
- lukeschlather 6y agoIt seems irresponsible to inject devices into your network that that indiscriminately MITM all traffic and can easily be configured to log passwords and auth cookies, no matter what setting you're in.
- bdamm 6y agoIt's basically the TSA of corporate networks. They need to inspect traffic because they can't control what devices show up in their environments and what malware might ride along side legitimate traffic.
- compuguy 6y agoActually most of these corporations have plenty of controls on their networks preventing the random plugging in of devices into networks. Most of the time they are using something that involves 802.1X.
- lucb1e 6y agoPlus which, it allows me to check what black box software is doing. Certificate pinning is great and all, but it also makes it way harder to know what data "huawei mobile services", "google play services", or a random mobile game for that matter, is phoning home about. I'm not a big fan of these corporate MITM boxes that contain the keys to the TLS traffic of the whole company (which additionally often double as employees' private phones and laptops), but I do like to look at my own device's traffic.
- 6y ago
- GordonS 6y agoA colleague of mine was also looking at Ubuntu Core for an IoT project recently, but Ubuntu wanted $15k/y to run a private, branded Snap store - erm.. no. If they really want snaps to succeed, there should be an open source snap store protocol, and 3rd parties should be allowed to run their own stores, just like you can add 3rd part apt repos, for example. We decided on Photon OS, BTW. It's tiny, and perfect for use as a Docker host.
- eikenberry 6y agoIsn't there some old adage about how if you can't afford something you aren't the target audience. At the larger companies I've worked at you didn't even need approval for 15k/y.
- GordonS 6y agoThis was for a company with just under 300k employees - you need approval from multiple people for everything. From the marketing, blogs etc, Ubuntu Core does seem to be targeted at everyone, not just people that would drop $15k/y like it was nothing. It's almost like a trap - it sounds perfect for IoT, so you start wasting your time building a PoC, and then much later you find out about the costs. And as another commenter mentioned, they also charge you for doing updates on top!
- kevindong 6y agoAt the two larger companies I've worked at (~1,000 and ~50,000), I've been explicitly told that I cannot sign any contracts without getting it approved by the legal department. Furthermore, all software purchases must go through the approval process.
- beckler 6y agoWow, it's gone up! When we talked, it was $10k/year, and then there was an additional cost for every update we pushed out. It depended on the speed, size and number of devices receiving the update.
- 6y ago
- eikenberry 6y ago> [..] didn't seem interested in our fixing our complaints without a massive amount of money [..] Why is asking for money wrong? You want the feature, why shouldn't you pay for it.
- beckler 6y agoNothing wrong with it, but we couldn't get any funding approved until the PoC was successful, and we couldn't be successful without the feature.
- lucb1e 6y ago> We tried EVERY hack we could think up Just to be sure, installing the CA from that MITM box didn't work? Because that should be the generally recommended solution and I can't see why snap would have a hardcoded CA list separate from the system. If that didn't work, it's indeed a bug, but a rather weird one; definitely worth posting to the bug tracker.
- beckler 6y agoThe CAs are hard embedded in the core snap. They're pulled from some specific package when built, but snaps themselves are immutable. We attempted to overwrite it in several different ways, but the OS is just simply mounting these folders from the core snap (which is immutable), and then marking those mounted paths them as immutable. That being said, there are a list of paths you can write to, and they're listed here (for core18): https://github.com/snapcore/core18/blob/master/static/etc/system-image/writable-paths https://github.com/snapcore/core18/blob/master/static/etc/sy...
- lucb1e 6y ago> The CAs are hard embedded in the core snap. That is ridiculous. Thanks for the info, yet another reason to stay far away from snap...