7 ms·

Having sold both containers and VMs to untrusted users? I disagree. Now, my container experience is old, and containerization technology is improving all the

by lsc 13y ago

Having sold both containers and VMs to untrusted users? I disagree. Now, my container experience is old, and containerization technology is improving all the time, so over time, what I say here becomes less true, but that's the thing; I'm willing to pay a fairly high cost in (cheap) ram for relatively small gains in compartmentalization. (which is to say, I'm willing to spend a fair bit of money to make my pager go off less.) See, the expensive part? it's not the ram, it's sysadmin time. That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box. Same goes for all the containerization technologies I've tried; it's easier to just throw xen/kvm on there and pay for the extra ram than to manage one user stepping on resources required by another user. For me? that was the beautiful part of switching from jails to Xen; suddenly, I could just ignore customers who tried to burn all the cpu and ram they could. Now, if I can figure out how to do the same with disk? I'll be a very happy man. That, at least, is where I'm putting my money; in more ram.

jauer 13y ago
SmartOS has disk IO throttling for KVM guests.
- lsc 13y ago
  and you can throttle with ionice on xen running a Linux dom0[1]. It works pretty great if you have some user who is going overboard and you want to make it suck for just them and suck less for everyone else. It doesn't work like the CPU scheduler; which is to say, I can't set ionice on all the guests and expect good performance on the light-use guests while a heavy-use guest is thrashing the disk. (I can expect the good performance for the light-use guests when the resource in contention is CPU time.) It's possible that either ionice isn't a good enough tool, or that I don't know enough about it... but I think it's far more likely that the variable nature of disk throughput makes balancing requests harder... (e.g. how many IOPS a second I get varies a lot from second to second, based on where the sectors I want to operate on are in relation to the read/write head. How much disk bandwidth I get varies even more.) I am pretty curious about how much this situation will change with SSD storage. [1]http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting_Untrusted_Users_Under_Xen:_Lessons_from_the_Trenches#Regulating_Disk_Access_with_ionice http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting...
  - nitrogen 13y ago
    Maybe with spinning disks, instead of/in addition to allocating IO operations or bandwidth, you could allocate seek distance.
    - davidstrauss 13y ago
      Especially once you start using hardware RAID controllers, the physical overhead of running I/O commands is pretty abstracted away from the kernel.
    - lsc 13y ago
      My understanding is that the cyl/head/track info the operating system gets is... pretty far abstracted from reality by the disk firmware, making that difficult to do in any but a very rough way. By 'very rough' I mean that in the normal case, what the OS sees as sequential sectors are sequential on disk, modulo the bad block remapping... but, from what I understand, going further, e.g. Is sector 5557 closer to 6669 or 3331? is all but impossible to answer from the OS. I mean, most of the time we assume that 6669 is closer as the difference between 6669 and 5557 is smaller than the difference between 3331 and 5557, but because the slow part of the disk is usually the rotation and not the head seek, this is not always or even usually the case.
porker 13y ago
> See, the expensive part? it's not the ram, it's sysadmin time. And that's why I like containers, because the sysadmin time of updating and patching all VMs I own (and manage for friends/clients) is huge. Yes, I should probably use chef/puppet. The potential incompatibility issues with automated VM upgrades are probably the same as upgrading the container's OS. But mentally it feels more risky.
- dspillett 13y ago
  > because the sysadmin time of updating and patching all VMs I own There is a key difference between your circumstance the person you are replying to: you are managing a farm of your own VMs so that admin time is your's to find. If the VMs are sold to thers so you are just hosting them, then the VM updates are someone else's problem and the admin time that becomes your problem is dealing with people whose VMs that are (accidentally usually, but not always) abusing shared resources. This is something you don't have as much hassle with when all the VMs are your's: you have presumably planned the likely resource needs carefuly and if something out of the ordinary happens you won't answer back to yourself like someone who has bought a cheap VM would answer back to you when you tell them they seem to be doing something wrong. Selling containers/VMs to others is a very different kettle of fish to running you own private farm.
  - davidgerard 13y ago
    If you buy a VM on Amazon, you get to do your apt-get updates yourself. And yes, managing it is a PITA.
    - susi22 13y ago
      Shouldn't be a PITA. Look into Vagrant, ansible, salt stack, chef, puppet etc.
      - porker 13y ago
        Does vagrant apply in this situation? I thought it was only for dev machines
davidstrauss 13y ago
> Now, if I can figure out how to do the same with disk? Hi, I'm the author of the article. You should check out cgroups. You can isolate disk access between services and containers using block I/O shares with cgroups. The way shares work in cgroups is that there's unlimited access until the resource comes under contention. Then, the kernel schedules activity based on which cgroups have shares remaining. This allows scheduling to be "fair" in the sense that no cgroup always wins; access just becomes proportional. We use block I/O shares (and other cgroup methods) for isolating containerized MySQL instances on Pantheon, the platform I work on as CTO.
- boryas 13y ago
  Doesn't the blkio cgroup not know anything about buffered writes? Or do the shares work differently and that limitation is only for accounting?
  - davidstrauss 13y ago
    Yes: "All the buffered writes are still system wide and not per group. Hence we will not see service differentiation between buffered writes between groups." [1] [1] https://www.kernel.org/doc/Documentation/cgroups/blkio-controller.txt https://www.kernel.org/doc/Documentation/cgroups/blkio-contr...
    - lsc 13y ago
      huh. interesting. this was the primary reason jails sucked so hard for me. The heavy users would push the light users pagecache out of ram. However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with. I wonder if then using cgroups for the remaining writes would work well? seems like the 'shares' approach might work better than the 'priority' approach that ionice uses.
      - davidstrauss 13y ago
        > However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with. It gets hard to say whether a shared page cache is a good thing or not, even though it may be unfair. I say this because I/O bandwidth getting exhausted is a huge issue. For example, using cgroups to limit memory allocations for groups or processes seemed like a great way to fairly distribute memory. But, doing so forced such cgroups into swapping when they tried to exceed their limits, even when there was available memory on the host system. The swapping was so bad in terms of saturating disk I/O that we had two choices to maintain quality of service: (1) set the hard limit and OOM kill (or equivalent) within the cgroup when it gets exceeded or (2) not treat it as a hard limit and monitor usage separately. We chose the latter. So, I honestly wonder, is it better to enforce separate page caches in the cause of fairness, even if it results in less efficient disk I/O? Or is it better to have a unified page cache and dedicate system-wide resources to increasing effective disk I/O bandwidth? (Do we focus on slicing the pie more fairly or increasing the size the pie while cutting sloppily?)
ChuckMcM 13y ago
So one of the projects I got to particpate in at Google before I left were efforts at containerizing disk I/O such that each container could get a strong guarantee of x% of the disks available bandwidth. That resulted in a number of really useful changes to the disk scheduler code. I expect that the code has been contributed back by this time but I've not been following it closely. Like the author I think containers are a much better answer than VMs. That is particularly true in a "web 2.0" world. I wrote up a bit about that on http://ops.blekko.com http://ops.blekko.com. In an architecture where the "app" is effectively a schema across a bunch of platform services you can achieve much higher machine utilization with containers. At some point I hope the team at Google writes up their results for publication.
csense 13y ago
Parent's business provides virtual servers to the general public (disclosure: I'm a customer). With Xen, guests can run any distro kernel, self-compiled kernel with custom patches, or even BSD -- any guest kernel with Xen patches should work. The guests can update their kernel on their own, and the only the guest that's updating their kernel needs a reboot -- the host, and guests which don't change kernels except during major upgrades, can have months or years of uptime. So if you have a business where some of your customers want custom, bleeding-edge kernels, and others mainly value stable systems with high uptimes, and everyone wants to run a different distribution's kernel, then containers aren't for you. I guess I'm trying to say that VPS will always have a niche in the market. The parent lives in that niche. Containers offer fundamental resource-efficiency advantages and may capture large markets from Xen and Xen-like technologies over the next few years, but I disagree with the article's implication that Xen will eventually become extinct. As the parent notes, RAM is cheap, and the difference in costs between container and Xen will give the former an advantage, but never be big enough to drive the latter completely out of business, since the difference is small (on an absolute scale) and Xen can do a few things that containers can't.
jacques_chester 13y ago
> That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box I came to the same conclusion nearly 5 years ago: http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed-and-i-have-the-graphs-to-prove-it/ http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed... I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.
- lsc 13y ago
  Yeah. I saw that at the time, I think. >I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc. I think the thing that keeps shared hosting alive is that most people don't want to be SysAdmins. VPSs don't remove the need for a sysadmin's services, they just push it down to the customer. From that perspective, shared php hosting is not only more expensive to do well (vs. selling VPSs) it provides more value to the customer (when done well) - unfortunately, market expectations right now make people expect to pay more for a VPS; the net effect being that most shared hosting services have underfunded admin departments and are thus, kinda crap. I mean, even as someone who sells VPSs? I want the shared hosting market to be stronger. Selling a VPS to someone who is not able or willing to sysadmin a linux box is not fun or profitable. It's a bad day for the customer, and it's a bad day for me. And it's reasonable to want to not sysadmin your own server; it's a lot of work. We need some place for those people to go.
  - jacques_chester 13y ago
    That's why at the time I pointed out that a new breed of apps, written to the VPS as a target, could take the administrative burden off both host and user and move it upstream to the app.
    - lsc 13y ago
      Maybe. We have all sorts of tools that are reducing the administrative overhead of running a linux box, but they are far from eliminating that burden. Further, most of the tools I've seen? webmin and the like? The thing is, it works great... Usually, for a while. Then sometimes it breaks, and it's a huge emergency, as the customer had it working just fine for a while; the customer was tricked into thinking they knew what they were doing, and now they've got an emergency on their hands, and they don't even have the training that installing the apps by hand forces on you. So yeah; the thing I see displacing shared hosting? not VPSs, but another kind of fully managed service. See, as you point out, the difficulty of managing a shared hosting host goes up with the number of customers on that host in a kindof exponential way. What if you made something like shared hosting (where the user doesn't have root access, and the host is managed by full-time sysadmins) but ran one user per host, and run each host in a VM or container or something that is managed in a largely automated way? Spend the ram/cpu overhead to better compartmentalize users and take the exponential increase in difficulty out of adding more customers to a particular bit of hardware? It's something I've thought about for a while. Problem is, the temptation to give users root on their VM would be very strong, and once that happens? well, managing thousands of hosts isn't that big of a deal... managing thousands of sysadmins? yeah. Not a job I want. Either I manage the host or you manage the host. Sharing root is a difficult, high-touch thing that requires a lot of communication. Edit: I believe that many of the platform as a service providers operate like this (e.g. your stuff runs in one or several VMs that are not shared with other users, but that you don't have root on.) Considering how cheap platform as a service is for the light users, I wonder how that model hasn't destroyed the flat-rate 'shared hosting' model? Perhaps one of the platform as a service providers needs to offer a flat monthly plan (perhaps that has the option of degrading /or/ interrupting service when usage exceeds the agreed-upon threshold, sort of in emulation of the "unlimited" providers.)

by lsc 13y ago

jauer 13y ago

SmartOS has disk IO throttling for KVM guests.

lsc 13y ago
and you can throttle with ionice on xen running a Linux dom0[1]. It works pretty great if you have some user who is going overboard and you want to make it suck for just them and suck less for everyone else. It doesn't work like the CPU scheduler; which is to say, I can't set ionice on all the guests and expect good performance on the light-use guests while a heavy-use guest is thrashing the disk. (I can expect the good performance for the light-use guests when the resource in contention is CPU time.) It's possible that either ionice isn't a good enough tool, or that I don't know enough about it... but I think it's far more likely that the variable nature of disk throughput makes balancing requests harder... (e.g. how many IOPS a second I get varies a lot from second to second, based on where the sectors I want to operate on are in relation to the read/write head. How much disk bandwidth I get varies even more.) I am pretty curious about how much this situation will change with SSD storage. [1]http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting_Untrusted_Users_Under_Xen:_Lessons_from_the_Trenches#Regulating_Disk_Access_with_ionice http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting...
- nitrogen 13y ago
  Maybe with spinning disks, instead of/in addition to allocating IO operations or bandwidth, you could allocate seek distance.
  - davidstrauss 13y ago
    Especially once you start using hardware RAID controllers, the physical overhead of running I/O commands is pretty abstracted away from the kernel.
  - lsc 13y ago
    My understanding is that the cyl/head/track info the operating system gets is... pretty far abstracted from reality by the disk firmware, making that difficult to do in any but a very rough way. By 'very rough' I mean that in the normal case, what the OS sees as sequential sectors are sequential on disk, modulo the bad block remapping... but, from what I understand, going further, e.g. Is sector 5557 closer to 6669 or 3331? is all but impossible to answer from the OS. I mean, most of the time we assume that 6669 is closer as the difference between 6669 and 5557 is smaller than the difference between 3331 and 5557, but because the slow part of the disk is usually the rotation and not the head seek, this is not always or even usually the case.

porker 13y ago

> See, the expensive part? it's not the ram, it's sysadmin time. And that's why I like containers, because the sysadmin time of updating and patching all VMs I own (and manage for friends/clients) is huge. Yes, I should probably use chef/puppet. The potential incompatibility issues with automated VM upgrades are probably the same as upgrading the container's OS. But mentally it feels more risky.

dspillett 13y ago
> because the sysadmin time of updating and patching all VMs I own There is a key difference between your circumstance the person you are replying to: you are managing a farm of your own VMs so that admin time is your's to find. If the VMs are sold to thers so you are just hosting them, then the VM updates are someone else's problem and the admin time that becomes your problem is dealing with people whose VMs that are (accidentally usually, but not always) abusing shared resources. This is something you don't have as much hassle with when all the VMs are your's: you have presumably planned the likely resource needs carefuly and if something out of the ordinary happens you won't answer back to yourself like someone who has bought a cheap VM would answer back to you when you tell them they seem to be doing something wrong. Selling containers/VMs to others is a very different kettle of fish to running you own private farm.
- davidgerard 13y ago
  If you buy a VM on Amazon, you get to do your apt-get updates yourself. And yes, managing it is a PITA.
  - susi22 13y ago
    Shouldn't be a PITA. Look into Vagrant, ansible, salt stack, chef, puppet etc.
    - porker 13y ago
      Does vagrant apply in this situation? I thought it was only for dev machines

davidstrauss 13y ago

> Now, if I can figure out how to do the same with disk? Hi, I'm the author of the article. You should check out cgroups. You can isolate disk access between services and containers using block I/O shares with cgroups. The way shares work in cgroups is that there's unlimited access until the resource comes under contention. Then, the kernel schedules activity based on which cgroups have shares remaining. This allows scheduling to be "fair" in the sense that no cgroup always wins; access just becomes proportional. We use block I/O shares (and other cgroup methods) for isolating containerized MySQL instances on Pantheon, the platform I work on as CTO.

boryas 13y ago
Doesn't the blkio cgroup not know anything about buffered writes? Or do the shares work differently and that limitation is only for accounting?
- davidstrauss 13y ago
  Yes: "All the buffered writes are still system wide and not per group. Hence we will not see service differentiation between buffered writes between groups." [1] [1] https://www.kernel.org/doc/Documentation/cgroups/blkio-controller.txt https://www.kernel.org/doc/Documentation/cgroups/blkio-contr...
  - lsc 13y ago
    huh. interesting. this was the primary reason jails sucked so hard for me. The heavy users would push the light users pagecache out of ram. However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with. I wonder if then using cgroups for the remaining writes would work well? seems like the 'shares' approach might work better than the 'priority' approach that ionice uses.
    - davidstrauss 13y ago
      > However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with. It gets hard to say whether a shared page cache is a good thing or not, even though it may be unfair. I say this because I/O bandwidth getting exhausted is a huge issue. For example, using cgroups to limit memory allocations for groups or processes seemed like a great way to fairly distribute memory. But, doing so forced such cgroups into swapping when they tried to exceed their limits, even when there was available memory on the host system. The swapping was so bad in terms of saturating disk I/O that we had two choices to maintain quality of service: (1) set the hard limit and OOM kill (or equivalent) within the cgroup when it gets exceeded or (2) not treat it as a hard limit and monitor usage separately. We chose the latter. So, I honestly wonder, is it better to enforce separate page caches in the cause of fairness, even if it results in less efficient disk I/O? Or is it better to have a unified page cache and dedicate system-wide resources to increasing effective disk I/O bandwidth? (Do we focus on slicing the pie more fairly or increasing the size the pie while cutting sloppily?)

ChuckMcM 13y ago

So one of the projects I got to particpate in at Google before I left were efforts at containerizing disk I/O such that each container could get a strong guarantee of x% of the disks available bandwidth. That resulted in a number of really useful changes to the disk scheduler code. I expect that the code has been contributed back by this time but I've not been following it closely. Like the author I think containers are a much better answer than VMs. That is particularly true in a "web 2.0" world. I wrote up a bit about that on http://ops.blekko.com http://ops.blekko.com. In an architecture where the "app" is effectively a schema across a bunch of platform services you can achieve much higher machine utilization with containers. At some point I hope the team at Google writes up their results for publication.

csense 13y ago

Parent's business provides virtual servers to the general public (disclosure: I'm a customer). With Xen, guests can run any distro kernel, self-compiled kernel with custom patches, or even BSD -- any guest kernel with Xen patches should work. The guests can update their kernel on their own, and the only the guest that's updating their kernel needs a reboot -- the host, and guests which don't change kernels except during major upgrades, can have months or years of uptime. So if you have a business where some of your customers want custom, bleeding-edge kernels, and others mainly value stable systems with high uptimes, and everyone wants to run a different distribution's kernel, then containers aren't for you. I guess I'm trying to say that VPS will always have a niche in the market. The parent lives in that niche. Containers offer fundamental resource-efficiency advantages and may capture large markets from Xen and Xen-like technologies over the next few years, but I disagree with the article's implication that Xen will eventually become extinct. As the parent notes, RAM is cheap, and the difference in costs between container and Xen will give the former an advantage, but never be big enough to drive the latter completely out of business, since the difference is small (on an absolute scale) and Xen can do a few things that containers can't.

jacques_chester 13y ago

> That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box I came to the same conclusion nearly 5 years ago: http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed-and-i-have-the-graphs-to-prove-it/ http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed... I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.

lsc 13y ago
Yeah. I saw that at the time, I think. >I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc. I think the thing that keeps shared hosting alive is that most people don't want to be SysAdmins. VPSs don't remove the need for a sysadmin's services, they just push it down to the customer. From that perspective, shared php hosting is not only more expensive to do well (vs. selling VPSs) it provides more value to the customer (when done well) - unfortunately, market expectations right now make people expect to pay more for a VPS; the net effect being that most shared hosting services have underfunded admin departments and are thus, kinda crap. I mean, even as someone who sells VPSs? I want the shared hosting market to be stronger. Selling a VPS to someone who is not able or willing to sysadmin a linux box is not fun or profitable. It's a bad day for the customer, and it's a bad day for me. And it's reasonable to want to not sysadmin your own server; it's a lot of work. We need some place for those people to go.
- jacques_chester 13y ago
  That's why at the time I pointed out that a new breed of apps, written to the VPS as a target, could take the administrative burden off both host and user and move it upstream to the app.
  - lsc 13y ago
    Maybe. We have all sorts of tools that are reducing the administrative overhead of running a linux box, but they are far from eliminating that burden. Further, most of the tools I've seen? webmin and the like? The thing is, it works great... Usually, for a while. Then sometimes it breaks, and it's a huge emergency, as the customer had it working just fine for a while; the customer was tricked into thinking they knew what they were doing, and now they've got an emergency on their hands, and they don't even have the training that installing the apps by hand forces on you. So yeah; the thing I see displacing shared hosting? not VPSs, but another kind of fully managed service. See, as you point out, the difficulty of managing a shared hosting host goes up with the number of customers on that host in a kindof exponential way. What if you made something like shared hosting (where the user doesn't have root access, and the host is managed by full-time sysadmins) but ran one user per host, and run each host in a VM or container or something that is managed in a largely automated way? Spend the ram/cpu overhead to better compartmentalize users and take the exponential increase in difficulty out of adding more customers to a particular bit of hardware? It's something I've thought about for a while. Problem is, the temptation to give users root on their VM would be very strong, and once that happens? well, managing thousands of hosts isn't that big of a deal... managing thousands of sysadmins? yeah. Not a job I want. Either I manage the host or you manage the host. Sharing root is a difficult, high-touch thing that requires a lot of communication. Edit: I believe that many of the platform as a service providers operate like this (e.g. your stuff runs in one or several VMs that are not shared with other users, but that you don't have root on.) Considering how cheap platform as a service is for the light users, I wonder how that model hasn't destroyed the flat-rate 'shared hosting' model? Perhaps one of the platform as a service providers needs to offer a flat monthly plan (perhaps that has the option of degrading /or/ interrupting service when usage exceeds the agreed-upon threshold, sort of in emulation of the "unlimited" providers.)