5 ms·

This is a pretty good lesson: when you have that little niggling feeling in the back of your head about something security-related, take care of it. Otherwise,

by jarin 16y ago

This is a pretty good lesson: when you have that little niggling feeling in the back of your head about something security-related, take care of it. Otherwise, someone WILL exploit it. Seems like they were using the load balancer as a way to obfuscate the existence of the individual EC2 instances. Also, that has gotta be really expensive to have an EC2 instance-per-customer.

X-Istence 16y ago
Depends on the type of instance they spin up, but I would definitely tend to agree with you! Security in shared hosting is extremely hard (I used to be a sys admin for a hosting company in a prior life), especially since there is no good way to separate everyone from each other without making performance suck completely, FreeBSD jails alleviate some of it, but you start having scalability issues, PHP running in php-fpm works, but uses up a lot of resources keeping spare instances around, there are a whole bunch of other ones as well. Individual virtual machines per user isn't such a crazy idea but it is really expensive. What I would really like to know is how Google has accomplished it, at scale, with AppEngine. How are they able to do their security separation so well that at this point I am not even aware of any security breaches. There has to be a better way to do it, and securely, but it may require rethinking how the entire architecture fits together, PHP, a web server, and the database engine.
- sriramk 16y ago
  Heroku has done it reasonably well too and they seem to do it only with POSIX permissions for the most part.
  - X-Istence 16y ago
    That is definitely interesting, although for Ruby that is slightly simpler because there is no real way like PHP to have a single instance of Ruby deal with all of the requests (talking about mod_php and PHP-fastcgi, not php-cgi which spawns a process per request).
  - moe 16y ago
    Ahem. There have been multiple exploits for heroku, some of which enabled access to code and data of other heroku customers (google for "heroku vulnerability"). From what I read about their virtualization (which may not be up-to-date) they seem to rely on the security of chroot(). If that is still the case then there is a big problem in their future.
    - sriramk 16y ago
      Could you expand on the last sentence? Why is relying on chroot+file permissions inherently bad?
      - baq 16y ago
        chroot has not been designed as a security feature but as a system testing tool. you only need a local root exploit to get out of chroot. you need additional protection to have a proper jail; freebsd does this, openbsd used to, not sure how it is now.
- tomjen3 16y ago
  >What I would really like to know is how Google has accomplished it, at scale, with AppEngine. How are they able to do their security separation so well that at this point I am not even aware of any security breaches. Easy (well it takes a lot of work, but it isn't difficult): they only allow languages that run on VMs, they then rewrite the VMs to remove/limit the file system access, network access and whatever else they don't want you to access, prevent access to unknown C code (since you can always fudge with the stack and other fun stuff at that point) and count the memory used and instructions run (so they can bill you and so denial of service gets too expensive). This kind of thing is easy to do if you throw 100 good developers on it. My guess is that phpfrog didn't have that kind of manpower.
jpetazzo 16y ago
I don't think they are obfuscating it. If I remember correctly, their pricing page made perfectly clear that they used dedicated EC2 instances. Just for the record - the cheapest EC2 instance type is t1.micro, and amounts to ~15 USD/month (+EBS and IP costs). I didn't see their business plan so I can't tell what is their big picture about that :-)
- jarin 16y ago
  Oh that's not bad at all, maybe they added that since the last time I looked or maybe I just missed it.
- ridruejo 16y ago
  If they use reserved instances, it should be even less than that. They still need to control abuse in terms of bandwidth, etc. and that is the difficult part.
  - ceejayoz 16y ago
    It should be easy enough to throttle bandwidth on the instances.
    - ridruejo 16y ago
      As far as I know that cannot be done with standard Amazon tools
      - ceejayoz 16y ago
        You don't need it on Amazon's side - plenty of solutions exist that can be installed on the instances themselves. Apache throttling, for example.

by jarin 16y ago

X-Istence 16y ago

Depends on the type of instance they spin up, but I would definitely tend to agree with you! Security in shared hosting is extremely hard (I used to be a sys admin for a hosting company in a prior life), especially since there is no good way to separate everyone from each other without making performance suck completely, FreeBSD jails alleviate some of it, but you start having scalability issues, PHP running in php-fpm works, but uses up a lot of resources keeping spare instances around, there are a whole bunch of other ones as well. Individual virtual machines per user isn't such a crazy idea but it is really expensive. What I would really like to know is how Google has accomplished it, at scale, with AppEngine. How are they able to do their security separation so well that at this point I am not even aware of any security breaches. There has to be a better way to do it, and securely, but it may require rethinking how the entire architecture fits together, PHP, a web server, and the database engine.

sriramk 16y ago
Heroku has done it reasonably well too and they seem to do it only with POSIX permissions for the most part.
- X-Istence 16y ago
  That is definitely interesting, although for Ruby that is slightly simpler because there is no real way like PHP to have a single instance of Ruby deal with all of the requests (talking about mod_php and PHP-fastcgi, not php-cgi which spawns a process per request).
- moe 16y ago
  Ahem. There have been multiple exploits for heroku, some of which enabled access to code and data of other heroku customers (google for "heroku vulnerability"). From what I read about their virtualization (which may not be up-to-date) they seem to rely on the security of chroot(). If that is still the case then there is a big problem in their future.
  - sriramk 16y ago
    Could you expand on the last sentence? Why is relying on chroot+file permissions inherently bad?
    - baq 16y ago
      chroot has not been designed as a security feature but as a system testing tool. you only need a local root exploit to get out of chroot. you need additional protection to have a proper jail; freebsd does this, openbsd used to, not sure how it is now.
tomjen3 16y ago
>What I would really like to know is how Google has accomplished it, at scale, with AppEngine. How are they able to do their security separation so well that at this point I am not even aware of any security breaches. Easy (well it takes a lot of work, but it isn't difficult): they only allow languages that run on VMs, they then rewrite the VMs to remove/limit the file system access, network access and whatever else they don't want you to access, prevent access to unknown C code (since you can always fudge with the stack and other fun stuff at that point) and count the memory used and instructions run (so they can bill you and so denial of service gets too expensive). This kind of thing is easy to do if you throw 100 good developers on it. My guess is that phpfrog didn't have that kind of manpower.

jpetazzo 16y ago

I don't think they are obfuscating it. If I remember correctly, their pricing page made perfectly clear that they used dedicated EC2 instances. Just for the record - the cheapest EC2 instance type is t1.micro, and amounts to ~15 USD/month (+EBS and IP costs). I didn't see their business plan so I can't tell what is their big picture about that :-)

jarin 16y ago
Oh that's not bad at all, maybe they added that since the last time I looked or maybe I just missed it.
ridruejo 16y ago
If they use reserved instances, it should be even less than that. They still need to control abuse in terms of bandwidth, etc. and that is the difficult part.
- ceejayoz 16y ago
  It should be easy enough to throttle bandwidth on the instances.
  - ridruejo 16y ago
    As far as I know that cannot be done with standard Amazon tools
    - ceejayoz 16y ago
      You don't need it on Amazon's side - plenty of solutions exist that can be installed on the instances themselves. Apache throttling, for example.