5 ms·
There's no way an "enterprise grade" cloud vendor like AWS would allow co-tenancy of containers (for ECS, Lambda etc) from different customers within a single V
by NovemberWhiskey 3y ago
There's no way an "enterprise grade" cloud vendor like AWS would allow co-tenancy of containers (for ECS, Lambda etc) from different customers within a single VM - it's the reason Firecracker exists.
- deleted 3y ago[deleted]
- monocasa 3y agoDoes google? I know they use gvisor in production, which is ultimately enforced by a normal kernel (with a ton of sandboxing on top of it).
- eddythompson80 3y agoGoogle is moving away from gvisor as well. The "process sandbox" wars are over. Everybody lost, hypervisors won. That's it. It feels incredibly wasteful after all. Hypervisors don't share mm, scheduler, etc. It's a lot of wasted resources. Google came in with gvisor at the last minute to try to say "no, sandboxes aren't dead. Look at our approach with gvisor". They lost too and are now moving away from it.
- lima 3y ago> Google is moving away from gvisor as well. I've been wondering about this - are they really?
- beardedwizard 3y agoI have seen zero evidence of this; but if it's true I would love to learn more. The real action is in side channel vulnerabilities bypassing all manner of protections.
- eddythompson80 3y agosee https://cloud.google.com/blog/products/serverless/cloud-run-jobs-and-second-generation-execution-environment-ga https://cloud.google.com/blog/products/serverless/cloud-run-...
- beardedwizard 3y agoBut this is because the workloads they execute changed, right? Http only before, to more general code today. I didn't see anything there that said gvisor was inferior, only that a new requirement was full kernel api access. For latency sensitive ephemeral and constrained workloads gvisor/seccomp can make a lot of sense and in the case of google handle multi-tenancy. Now if workloads become less ephemeral and more general purpose, tolerance for startup latency goes up, annd probability of bespoke needs goes up making VM more palatable.
- pjmlp 3y agoIn a way, it feels like a sweet revenge for microkernels.
- RainbowFriends 3y agoCitation needed. gvisor seems to be under active development and just added support for the systrap platform, deprecating ptrace: https://gvisor.dev/blog/2023/04/28/systrap-release/ https://gvisor.dev/blog/2023/04/28/systrap-release/
- znpy 3y ago[flagged]
- eddythompson80 3y agoCloud run has abandoned gvisor in their "second generation" execution environment for containers https://cloud.google.com/blog/products/serverless/cloud-run-jobs-and-second-generation-execution-environment-ga https://cloud.google.com/blog/products/serverless/cloud-run-... Obviously there might be many reasons for that, but as someone who worked on a similar gvisor tech for another company, it's dead in the water. No security expert or consultant will ever sign off on a process isolation model. Despite of architecture, audits, reviews, etc. There is just too much surface area for anyone to feel comfortable signing off on hostile multi-tenants with process isolation regardless of the sandboxing tech. Not saying that there are no bugs in hypervisors, but the surface area is so so much smaller.
- tptacek 3y agogvisor isn't simply a process isolation model. Security experts will certainly sign off on gvisor for some multitenant workloads. The reason Google is moving from it, to the extent they are, is that hypervisors are more performant for more common workloads.
- yencabulator 3y agoI read "we got tired of reimplementing Linux kernel syscalls and functionality" as the reason. Like network file systems. The Cloud Run client base kept asking for more and more features, and they punted to just running the Linux kernel.
- 3y ago
- Rapzid 3y agoReally? Has gvisor ever been popped? Has there ever even been a single high-profile compromise caused by a container escape? Shared hosting was a thing and considered "safe enough" for decades and that's all process isolation. Can't help but feel the security concerns are overblown. To support my claim; Well, Google IS using gvisor as part of their GKE sandboxing security..
- tptacek 3y agoI don't know what "popped" means here, but so far as I know there's never been a major incident caused by a flaw in gvisor. But gvisor is a much more intricate and carefully controlled system than standard Linux containers. Obviously, there have been tons of container escape compromises.
- eddythompson80 3y agoshared this link on another reply, but google moved away from gvisor to hypervisor for cloud run. It won't be long before they do for GKE as well https://cloud.google.com/blog/products/serverless/cloud-run-jobs-and-second-generation-execution-environment-ga https://cloud.google.com/blog/products/serverless/cloud-run-...
- mikehotel 3y agoIt doesn’t look like the moved away from gVisor due to security reasons. “We were able to achieve these improvements because the second generation execution environment is based on a micro VM. This means that unlike the first generation execution environment, which uses gVisor, a container running in the second generation execution environment has access to a full Linux kernel.”
- eddythompson80 3y agoThe reason you go with process isolation over VM isolation is performance. If you share a kernel, you share memory managers and pages, scheduler, limits, groups, etc. If you get better performance running VMs vs running processes, then what was even your isolation layer for? But at the end of the day, there is a line in the sand around hypervisors vs proc/kernel isolation models. I challenge you to go to a financial or medical institute and tell their CTO "yeah, we have this super bullet proof shared-kernel-inproc isolation model" The first question you'd get is "Why is this not just part of upstream linux?" Answer that question and realize why you should just use a hypervisor.
- intelVISA 3y agoTbf gvisor was pretty much DOA by design. Hypervisors are alright, but nowadays security expectations go much lower than ring-1.
- tptacek 3y agoCan you expand on this? What do you mean "security expectations go lower than ring-1", and how does that relate to gvisor?
- pjmlp 3y agoFor example what Microsoft is doing at firmware level for Azure.
- monocasa 3y agoWhat is Microsoft doing at the firmware level for azure?
- pjmlp 3y agoFor example, https://techcommunity.microsoft.com/t5/microsoft-defender-for-iot-blog/analyze-iot-ot-device-firmware-with-microsoft-defender-for-iot/ba-p/3853474 https://techcommunity.microsoft.com/t5/microsoft-defender-fo... https://techcommunity.microsoft.com/t5/azure-infrastructure-blog/introducing-microsoft-azure-boost-preview/ba-p/3876742 https://techcommunity.microsoft.com/t5/azure-infrastructure-...
- yencabulator 3y agogVisor uses KVM or ptrace as its sandbox layer, and there's some indications that Google's internal fork uses an unpublished kernel mechanism, perhaps by extending seccomp (EDIT: It seems this has made its way to the outside world since I last looked. `systrap` is now default: https://gvisor.dev/docs/architecture_guide/platforms/ https://gvisor.dev/docs/architecture_guide/platforms/ ). It's fake-kernel-in-userspace then sandboxed by seccomp. Saying gVisor is "ultimately enforced by a normal kernel" is about as misleading & accurate as "KVM is enforced by a normal kernel" -- it is, but it's a very narrow boundary, not the usual syscall ABI.
- basique 3y agoAren't Cloudflare Workers multitenant? Although, if you want to be cynical, that could be a reason they aren't 'enterprise grade™'.
- tyingq 3y agoThey are using v8 isolates, which is maybe easier to do in a sound way than the whole broad space of containers. Previous discussion: https://news.ycombinator.com/item?id=31740885 https://news.ycombinator.com/item?id=31740885
- nilptr 3y ago> There's no way an "enterprise grade" cloud vendor like AWS would allow co-tenancy of containers (for ECS, Lambda etc) from different customers within a single VM - it's the reason Firecracker exists. I won't speak for AWS, but your assumption about what "enterprise grade" cloud vendors do is dead wrong. I know, because I'm working on maintaining one of these systems.
- lima 3y agoLots of enterprise grade cloud vendors trust the Linux kernel boundary WAY too much...
- mochomocha 3y agoBoth Lambda firecracker VMs and t2 instances are multi-tenant and oversubscribed.
- tptacek 3y agoI take them to mean "multiple tenants sharing a kernel"; I think everyone understands that AWS and GCP have multitenant hypervisor hosts.
- rewmie 3y ago> There's no way an "enterprise grade" cloud vendor like AWS would allow co-tenancy of containers (...) I don't think your beliefs are well founded. AWS's EC2 by default only supoprts shared tenancy, and dedicated instances are a premium service.
- sharts 3y agoI think bryan cantrill founded a company (joyent? or triton?) to do just that several years ago. It may have been based on solaris/smartos zones which is that exact use case w/ very secure/isolated containers.
- abwizz 3y agoalthou it came with linux binary compat (of unknown quality) i think the solaris thing was just too off putting for most customers and the company did not do very well
- NexRebular 3y agoTriton is now being developed by MNX Solutions and seems to be doing quite well. We run Triton and SmartOS in production and the linux compatibility works via lx-zones just fine. Only some of the linux-locked software, which usually means docker, needs to go inside a bhyve VM.