7 ms·

I Hacked Google App Engine: Anatomy of a Java Bytecode Exploit

281 points by arkadiyt 5y ago

exabrial 5y ago
In process sandboxing was definitely one of the worst ideas attempted in the early days of Java. None of my jobs in the last 10 years have ran with the security manager enabled, as it's just useless. I'm suspicious of any modern technology attempting to do the same thing, as it's just hard to get right. When IBM developed Websphere for the 1996 Atlanta Olympics, they wanted a mainframe, but you know, web stuff. The model was doomed from the start, as they designed the JVM to run as a process. Sun attempted tried to get deeper kernel support for the JVM with Solaris, but ultimately just ended up developing the precursor to LXC Containers called Solaris Zones. What we have today with containers is pretty awesome but it was definitely a bumpy road to figure it all out.
- pjmlp 5y ago
  .NET Core doesn't support the .NET version of security manager (CAS), and the is actually a JEP for deprecating the security manager for removal. The JAAS idea was quite good, however it was too complex for most developers to implement properly. I see a big failure for .NET and Java not having been a JIT/AOT stack since the beginning, and now we are going through their reboots while trying to keep the existing ecosystems to fall apart in the process.
  - _old_dude_ 5y ago
    I believe it's JEP 411 [1], good ridance, there is nothing worst than a false sense of security. [1] https://openjdk.java.net/jeps/411 https://openjdk.java.net/jeps/411
- jfrunyon 5y ago
  > In process sandboxing was definitely one of the worst ideas attempted in the early days of Java. None of my jobs in the last 10 years have ran with the security manager enabled, as it's just useless. I'm suspicious of any modern technology attempting to do the same thing, as it's just hard to get right. I think Android has done a very good job with their permissions, which seem quite similar to me. Am I missing something? Yes, properly limiting permissions is difficult, and there can be bugs, but that's true regardless of what layer the security controls are applied at.
  - gene91 5y ago
    The comment from @pron in the sibling thread explains the difference. Android applications, in general, don't share process.
    - dfox 5y ago
      The Android API gives me the feeling that the original idea was to do in-process sandboxing in mostly same way as it is done in Java and run the whole thing as one giant JVM instance and only later it was changed to the process-for-app model (which probably was not that much of breaking change because lot of things that cross the process boundary had to be serializable/parcelable already because of how android activity lifecycle handles memory pressure).
    - jfrunyon 5y ago
      That's true, but largely irrelevant. > Yes, properly limiting permissions is difficult, and there can be bugs, but that's true regardless of what layer the security controls are applied at.
  - pjmlp 5y ago
    Yes, the fact that each application is usually a process, and a mix of Linux kernel features are used for sandboxing, namely LinuxSE, seccomp and eBPF. Also despite being based on Linux kernel, since Project Treble Android has doubled down on its Binder mechanism for IPC across drivers, kernel and activities with multiple processes, which in the end makes it look like microkernel based architecture.
- momothereal 5y ago
  What's the link between WebSphere and the Olympic games?
kasperni 5y ago
In-process sandboxing is falling out of favour. Code Access Security has been removed from the .NET framework [1]. And Java's SecurityManager is deprecated for removal in future versions of Java [2]. [1] https://docs.microsoft.com/en-us/dotnet/framework/misc/code-access-security https://docs.microsoft.com/en-us/dotnet/framework/misc/code-... [2] https://openjdk.java.net/jeps/411 https://openjdk.java.net/jeps/411
- pjmlp 5y ago
  Yeah, the way we adopted threading and dynamic libraries for plugins is on the way out and we are back into multiple process with IPC, for more stability and security. The downside is they are more memory hungry.
- CodesInChaos 5y ago
  > In-process sandboxing is falling out of favour. Javascript and webassembly still aim for for in process sandboxing, though browsers add process level security as defense-in-depth.
  - pron 5y ago
    They do not, at least not in the way "in-process sandboxing" means in the context of Java's Security Manager, which is now being considered for deprecation and removal. Java's Security Manager does not create a single set of permissions for the entire Java program, but rather assigns different permissions to different Java classes running in the same program and even on the same thread, through a mechanism that assigns different permissions to different subroutines on the call-stack, composing them in some elaborate way. The same operation can be allowed or blocked depending on the call-stack by which it is reached. For example, writing to a specific file or socket could be allowed if the call stack contains a trusted subroutine that is known to ensure the access is done "safely" but blocked otherwise. The reason this is now being considered for removal is that even though this mechanism is very flexible and powerful in theory, it is too elaborate to be used correctly by most programmers in practice.
    - 5y ago
eternalban 5y ago
This is a beautiful, impressive, hack. The blog post itself is a minor tour de force.
frew 5y ago
This is impressive! For folks who are worried about what this says about App Engine's security - historically the JVM level sandboxing was one part of an overall sandboxing story (not surprisingly, very similar to Chrome's setup) and my understanding, although haven't been on the team for a decade at this point, is that the JVM-level restrictions have been effectively replaced by a gVisor-based solution.
taklamunda 5y ago
I also wanna learn the technical, plz share with me
nyanpasu64 5y ago
Would capability-based security be more effective than what Java attempted for in-process isolation?
- hderms 5y ago
  Probably easier to reason about as a programmer, at least in my opinion
flakiness 5y ago
I think GAE isolation nowadays is done by gVisor[1]? Looking at the doc [2] however, it seems like the old "proprietary" mechanism is still used for old (first generation) runtime. Probably this post is about that version. [1] https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 [2] https://cloud.google.com/appengine/docs/standard/runtimes
- prattmic 5y ago
  Even the first-generation runtimes no longer use this mechanism. Note https://cloud.google.com/appengine/docs/standard/java/migrating-to-java8#taking_full_advantage_of_java_8_runtime_features https://cloud.google.com/appengine/docs/standard/java/migrat...: "All standard Java classes are now available, and there is no class allowlist."
  - flakiness 5y ago
    That's good to know!
scottlu2 5y ago
+1 impressive independent of anything else. +1 for doing this as an intern.
- deleted 5y ago
  [deleted]
iamgopal 5y ago
Security was the prime reason app engine couldn’t add features fast enough, and hence couldn’t gain the market fast enough, else it could have been the money maker for google.
- pjmlp 5y ago
  Security for similar offerings in AWS and Azure is quite good, I guess that wasn't the reason, after all Google only hires creme de la crème.
twistedpair 5y ago
Note that GAE has two flavors: - Standard (constrained, more shared bits) - Flex (supply your own container, get your own instance)
- thesandlord 5y ago
  These days, Standard uses gVisor so there are fewer and fewer constraints. Most things just work. Flex really is for the edge cases where you need really big instances or really long execution times. (I used to work at GCP)
  - elyobo 5y ago
    Or, annoyingly, websockets.
    - thesandlord 5y ago
      Yeah, it seems like all the new infrastructure features (websockets, gRPC, GBLB, VPC, etc) are coming to Cloud Run instead of App Engine.
      - elyobo 5y ago
        Websockets are fine on flex, but yeah, Cloud Run is getting the love these days.
f430 5y ago
too long. lost interest halfway through the article
segmondy 5y ago
very nice write up. wondering if they kept detailed notes or could recall most of this from memory after 8yrs.
EE84M3i 5y ago
Looks like the fix was: https://gitlab.ow2.org/asm/asm/-/issues/316506 https://gitlab.ow2.org/asm/asm/-/issues/316506 12 October 2013: ASM 4.2 (tag ASM_4_2) 316506 ByteVector doesn't validate byteLength in putUTF8(). https://asm.ow2.io/versions.html https://asm.ow2.io/versions.html Couldn't find a CVE
cjohansson 5y ago
Impressive work, Google must have been interested in your skills after this?
- jere 5y ago
  This was done in one week during a college internship. Seriously what the hell.
aarchi 5y ago
A logic programming language like Prolog seems like it would have been a good fit for solving the bytecode packing with constraints.

281 points by arkadiyt 5y ago

In process sandboxing was definitely one of the worst ideas attempted in the early days of Java. None of my jobs in the last 10 years have ran with the security manager enabled, as it's just useless. I'm suspicious of any modern technology attempting to do the same thing, as it's just hard to get right. When IBM developed Websphere for the 1996 Atlanta Olympics, they wanted a mainframe, but you know, web stuff. The model was doomed from the start, as they designed the JVM to run as a process. Sun attempted tried to get deeper kernel support for the JVM with Solaris, but ultimately just ended up developing the precursor to LXC Containers called Solaris Zones. What we have today with containers is pretty awesome but it was definitely a bumpy road to figure it all out.

pjmlp 5y ago
.NET Core doesn't support the .NET version of security manager (CAS), and the is actually a JEP for deprecating the security manager for removal. The JAAS idea was quite good, however it was too complex for most developers to implement properly. I see a big failure for .NET and Java not having been a JIT/AOT stack since the beginning, and now we are going through their reboots while trying to keep the existing ecosystems to fall apart in the process.
- _old_dude_ 5y ago
  I believe it's JEP 411 [1], good ridance, there is nothing worst than a false sense of security. [1] https://openjdk.java.net/jeps/411 https://openjdk.java.net/jeps/411
jfrunyon 5y ago
> In process sandboxing was definitely one of the worst ideas attempted in the early days of Java. None of my jobs in the last 10 years have ran with the security manager enabled, as it's just useless. I'm suspicious of any modern technology attempting to do the same thing, as it's just hard to get right. I think Android has done a very good job with their permissions, which seem quite similar to me. Am I missing something? Yes, properly limiting permissions is difficult, and there can be bugs, but that's true regardless of what layer the security controls are applied at.
- gene91 5y ago
  The comment from @pron in the sibling thread explains the difference. Android applications, in general, don't share process.
  - dfox 5y ago
    The Android API gives me the feeling that the original idea was to do in-process sandboxing in mostly same way as it is done in Java and run the whole thing as one giant JVM instance and only later it was changed to the process-for-app model (which probably was not that much of breaking change because lot of things that cross the process boundary had to be serializable/parcelable already because of how android activity lifecycle handles memory pressure).
  - jfrunyon 5y ago
    That's true, but largely irrelevant. > Yes, properly limiting permissions is difficult, and there can be bugs, but that's true regardless of what layer the security controls are applied at.
- pjmlp 5y ago
  Yes, the fact that each application is usually a process, and a mix of Linux kernel features are used for sandboxing, namely LinuxSE, seccomp and eBPF. Also despite being based on Linux kernel, since Project Treble Android has doubled down on its Binder mechanism for IPC across drivers, kernel and activities with multiple processes, which in the end makes it look like microkernel based architecture.
momothereal 5y ago
What's the link between WebSphere and the Olympic games?

kasperni 5y ago

In-process sandboxing is falling out of favour. Code Access Security has been removed from the .NET framework [1]. And Java's SecurityManager is deprecated for removal in future versions of Java [2]. [1] https://docs.microsoft.com/en-us/dotnet/framework/misc/code-access-security https://docs.microsoft.com/en-us/dotnet/framework/misc/code-... [2] https://openjdk.java.net/jeps/411 https://openjdk.java.net/jeps/411

pjmlp 5y ago
Yeah, the way we adopted threading and dynamic libraries for plugins is on the way out and we are back into multiple process with IPC, for more stability and security. The downside is they are more memory hungry.
CodesInChaos 5y ago
> In-process sandboxing is falling out of favour. Javascript and webassembly still aim for for in process sandboxing, though browsers add process level security as defense-in-depth.
- pron 5y ago
  They do not, at least not in the way "in-process sandboxing" means in the context of Java's Security Manager, which is now being considered for deprecation and removal. Java's Security Manager does not create a single set of permissions for the entire Java program, but rather assigns different permissions to different Java classes running in the same program and even on the same thread, through a mechanism that assigns different permissions to different subroutines on the call-stack, composing them in some elaborate way. The same operation can be allowed or blocked depending on the call-stack by which it is reached. For example, writing to a specific file or socket could be allowed if the call stack contains a trusted subroutine that is known to ensure the access is done "safely" but blocked otherwise. The reason this is now being considered for removal is that even though this mechanism is very flexible and powerful in theory, it is too elaborate to be used correctly by most programmers in practice.
  - 5y ago

eternalban 5y ago

This is a beautiful, impressive, hack. The blog post itself is a minor tour de force.

frew 5y ago

This is impressive! For folks who are worried about what this says about App Engine's security - historically the JVM level sandboxing was one part of an overall sandboxing story (not surprisingly, very similar to Chrome's setup) and my understanding, although haven't been on the team for a decade at this point, is that the JVM-level restrictions have been effectively replaced by a gVisor-based solution.

taklamunda 5y ago

I also wanna learn the technical, plz share with me

nyanpasu64 5y ago

Would capability-based security be more effective than what Java attempted for in-process isolation?

hderms 5y ago
Probably easier to reason about as a programmer, at least in my opinion

flakiness 5y ago

I think GAE isolation nowadays is done by gVisor[1]? Looking at the doc [2] however, it seems like the old "proprietary" mechanism is still used for old (first generation) runtime. Probably this post is about that version. [1] https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 [2] https://cloud.google.com/appengine/docs/standard/runtimes

prattmic 5y ago
Even the first-generation runtimes no longer use this mechanism. Note https://cloud.google.com/appengine/docs/standard/java/migrating-to-java8#taking_full_advantage_of_java_8_runtime_features https://cloud.google.com/appengine/docs/standard/java/migrat...: "All standard Java classes are now available, and there is no class allowlist."
- flakiness 5y ago
  That's good to know!

scottlu2 5y ago

+1 impressive independent of anything else. +1 for doing this as an intern.

deleted 5y ago
[deleted]

iamgopal 5y ago

Security was the prime reason app engine couldn’t add features fast enough, and hence couldn’t gain the market fast enough, else it could have been the money maker for google.

pjmlp 5y ago
Security for similar offerings in AWS and Azure is quite good, I guess that wasn't the reason, after all Google only hires creme de la crème.

twistedpair 5y ago

Note that GAE has two flavors: - Standard (constrained, more shared bits) - Flex (supply your own container, get your own instance)

thesandlord 5y ago
These days, Standard uses gVisor so there are fewer and fewer constraints. Most things just work. Flex really is for the edge cases where you need really big instances or really long execution times. (I used to work at GCP)
- elyobo 5y ago
  Or, annoyingly, websockets.
  - thesandlord 5y ago
    Yeah, it seems like all the new infrastructure features (websockets, gRPC, GBLB, VPC, etc) are coming to Cloud Run instead of App Engine.
    - elyobo 5y ago
      Websockets are fine on flex, but yeah, Cloud Run is getting the love these days.

f430 5y ago

too long. lost interest halfway through the article

segmondy 5y ago

very nice write up. wondering if they kept detailed notes or could recall most of this from memory after 8yrs.

EE84M3i 5y ago

Looks like the fix was: https://gitlab.ow2.org/asm/asm/-/issues/316506 https://gitlab.ow2.org/asm/asm/-/issues/316506 12 October 2013: ASM 4.2 (tag ASM_4_2) 316506 ByteVector doesn't validate byteLength in putUTF8(). https://asm.ow2.io/versions.html https://asm.ow2.io/versions.html Couldn't find a CVE

cjohansson 5y ago

Impressive work, Google must have been interested in your skills after this?

jere 5y ago
This was done in one week during a college internship. Seriously what the hell.

aarchi 5y ago

A logic programming language like Prolog seems like it would have been a good fit for solving the bytecode packing with constraints.