13 ms·
Oracle plans to dump risky Java serialization
- hprotagonist 8y agocool. can we kill pickle next?
- fpoling 8y agoOne of the attack vector in Java was classes storing native pointers as integer fields calling free on the above pointers in the finalizer. So the moment one can force deserialiazation of such classes one ends up with corrupted heap and trivially weponized exploits. Does pickle in Python suffer from the same problem?
- rjzzleep 8y agowell theres this comment in the docs: > Warning > The pickle module is not secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.
- icebraining 8y agoArbitrary code execution: https://www2.cs.uic.edu/~s/musings/pickle/ https://www2.cs.uic.edu/~s/musings/pickle/
- bitL 8y agoThat only works because Python is an interpreter. Won't cause a thing on Java.
- joshuamorton 8y agoDeserialization in Java requires reflection, which puts you firmly in the land of "I'm basically interpreted now". Javas serialization has basically the same set of vulnerability.
- dwaite 8y agoThe JRE has had arbitrary code execution attacks on serialization. The leaked classes eventually invoke a class loader and instantiate your binary code as a new java class.
- bitL 8y agoDoes the serialization work with bytecode though? Doesn't it stream just data members, not method implementations?
- icebraining 8y agoThe JVM is a Java Bytecode interpreter; don't be so sure you can't make it run things.
- the_new_guy_29 8y agoThx fpr the good read
- jerf 8y agoThe primary difference between pickle and all the other problematic serialization mechanisms that have been causing security issues is that pickle has been labeled for decades now as being unsuitable for any use with public data. No sarcasm, I'm quite serious. It hasn't been 100% successful at preventing issues, there have been a couple, but it does seem to have been enough to keep it from being the star of it's own massive disaster, even though the potential is theoretically there. As with many of the dynamic language deserializers, such as the Ruby YAML one, for pickle it's not even an exploit or something... it's a feature of the code that it can call methods, and getting to arbitrary methods isn't that hard.
- bonzini 8y agoWhy the heck are those classes marked as `implements Serializable`?...
- dwaite 8y agoPossibly because a superclass or interface are marked, and it is inherited? The problem is that (excluding the new filtering mechanism) deserialization is unscoped - any class in your code, a third party library, or the core VM is fair game in an attack.
- dionian 8y agoIt is a marker interface. You are supposed to consciously opt-in.
- bonzini 8y agoExactly, so I am not sure why wrappers to native pointers are Serializable.
- ben509 8y agoPickle is an entirely reasonable way to move data between processes when you're using Python for distributed processing.
- ataturk 8y agoIt's funny that serialization is noted as a "horrible mistake" from 1997 but no mention of RMI, the even worse mistake. I guess Java has a lot of bad mistakes. I wish Oracle would end Java so I could move on to something else. Or maybe I'll just move on to something else anyways as I'm really rather sick of writing these syntactically crippled lambdas. The streaming stuff is almost good.
- dwaite 8y agoGood news, removing serialization will irrevocably break RMI. And yeah, it would be nice if eventually a spring cleaning could remove some of the mistakes - Enumeration, Date/Calendar, Hashtable, ClassLoader, etc.
- chopin 8y agoWhat's the problem with ClassLoader?
- adrianmsmith 8y agoCould you point me to some information on why RMI is a bad mistake? (Genuine question.)
- willtim 8y agoIn OOP, objects have identity and state, having a remote endpoint with both these attributes is undesirable. IIRC, RMI also encourages one to abstract over the network, which can lead to very fine-grained, brittle and possibly insecure communication. This is somewhat ironic given that Sun itself authored "The fallacies of distributed computing".
- jnwatson 8y agoRMI (and similar technologies like CORBA) is a “mistake” in today’s world because it makes every public method an attack surface. It also hides important details that the client really can’t not deal with, namely network issues. I put mistake in quotes because there are situations where RMI (and Java serialization) work fine: trusted, reliable networks like cluster or grid computing.
- Hupriene 8y agoHaving used java serialization a few times for POC level work, I'll be sorry to see it go. I wish they would just rename it something sufficiently ominous sounding that people wouldn't think about using it on untrusted data sources. Maybe AribitraryCodeAndDataSerialization
- crooked-v 8y agoComing from a Javascript context, would this be something roughly analogous to expecting JSON.parse and actually getting eval?
- teraflop 8y agoPretty much. It's slightly less terrible than that -- the deserializer doesn't evaluate arbitrary Java expressions, but it can call no-argument constructors, run deserializer methods, and set field values (including private fields) on any loaded class that's marked Serializable. As you can imagine, any sufficiently large codebase is likely to have a million different ways this can be leveraged to get arbitrary code execution.
- askvictor 8y agoMight catch some cases, but just from a couple of weeks ago, by an org that supposedly puts security very high up the priority list we had this: https://news.ycombinator.com/item?id=17096022 https://news.ycombinator.com/item?id=17096022 (Signal's security hole created by using dangerouslySetInnerHTML)
- ta5244626777 8y agoYes it was a nice/quick/dirty way to persist an object graph, I bet it's been abused richly in a lot of codebases out there, it'll be curious to see how this is handled in terms of breakage i.e. the JDK that introduces this might be the Python 2/3 showdown (if that's still a thing). Be curious to do a Github wide grep for ObjectOutputStream or something similar and see what it's like in open source land.
- 8y ago
- exabrial 8y agoThere are some valid uses for it, but the interface is unsafe by default. How about instead, requiring a MAC key in the serialization protocol?
- dwaite 8y agoYou can package up the data so that there is authentication to evaluate before you hit the serialization layer, and integrity behind that authentication. This might be your approach if say your session cookie is based on serialized Java. (However, most people give up on this approach - java serialization is also very inefficient space-wise, and the cookie will get too big for the browser to honor)
- gmueckl 8y agoI would hate to see serialization go completely. It has its uses. Maybe Java should copy the attributes that the C# data contract serializer uses to mark the subset of classes that may appear in serialized form. It coukd be retrofitted and would be a less drastic change at the same time.
- bitmapbrother 8y agoIs it really going away, though? The article mentions that it's just being replaced with a safer implementation.
- gmueckl 8y agoI may be wrong, but to me it seems like they will replace the current implementation with something fundamentally incompatible. My impression is that dependencies like RMI will go away, too. But that would be a huge breaking change.
- bitmapbrother 8y agoSure it's going to be incompatible and break old code. The title, however, makes it seem like Oracle is getting rid of serialization when it's just replacing it with a safer method.
- gmueckl 8y agoJust for the sake of arguing semantics: "Just replacing" is incorrect in my view. The new implementation is designed specifically to not be one. It is a different mechanism mit a different interface and a different design goal.
- tannhaeuser 8y agoJava has had the "transient" keyword for that since 1.0.
- 8y ago
- jeffdavis 8y agoCan someone briefly explain the problems and how general they are to other serialization interfaces?
- blattimwind 8y agoMany runtimes include object serialization capabilities, which allow serialization of essentially arbitrary object structures with little code. Unpacking that kind of structure always means that you're constructing object instances of potentially any object you can construct, which generally means you can run arbitrary code. Examples: Java Serialization, Python marshal and pickle, Ruby marshal, Perl Data::Dumper.
- jeffdavis 8y agoSo the problems don't apply to, say, rust because rust doesn't run arbitrary initialization when building structures?
- curun1r 8y agoThis is a key point. The problem is significantly worse because of Java's "everything is an object" philosophy. In languages where data is just data and not a combination of data and behavior, you don't get this kind of problem.
- pron 8y agoYou also don't get this behavior if you restrict the scope of serialization. The problem here was that Java wanted to make virtually any object potentially serializable. This required all sorts of deep hacks that circumvent various language guarantees and open the door to vulnerabilities (as well as encumber new features, as serialization could interact with anything). A new serialization mechanism would be restricted to classes for which serialization makes sense, basically classes that, as you say, represent data rather than data with complex behavior.
- steveklabnik 8y ago
- rukuu001 8y agoIt's less useful now than it used to be. In an age of ubiquitous JSON there's much less need for sending Java objects over the wire.
- paulddraper 8y agoAnd even more seamless...protobuf.
- rukuu001 8y agoBlushing because this is the first I've heard of it - I owe you one :)
- quickthrower2 8y agoWhere have you been hiding.
- iainmerrick 8y agoI’m beginning to think there really is an xkcd for everything: https://www.xkcd.com/1053/ https://www.xkcd.com/1053/ (Everything that tends to come up on HB, anyway)
- quickthrower2 8y agoI meant it in a friendly way, knowing that there is a tonne of shit I don't know. But ASCII can't convey that. Maybe Unicode can. Should of put a smiley.
- echelon 8y agoProtos are amazing and have so many advantages over json. You'll love them!
- skocznymroczny 8y ago
- kpcyrd 8y agoIn case anybody is wondering, this is the attack vector that was used for the equifax hack and is also used in a fair number of remote code execution exploits for java servers.
- latchkey 8y agoThrough the fact that Struts did not implement things correctly and equifax did not upgrade their systems. It also sounds like it was multiple vulnerabilities. Like this one as well: https://www.cvedetails.com/cve/CVE-2017-5638/ https://www.cvedetails.com/cve/CVE-2017-5638/
- sewercake 8y agoWon't this have serious implications for Spark, Hadoop, and other frameworks that distribute workloads across multiple JVM instances?
- zeroxfe 8y agoAFAIK, Hadoop uses protocol buffers for message passing, not Java serialization.
- yzmtf2008 8y agoHadoop uses Avro actually, but the point still stands :)
- erik_seaberg 8y agoHadoop expects your keys and values to implement Writable and serialize themselves (a lot of these are actually hand-written expecting the instance to get reused for each input tuple). There's optional and fairly clumsy glue that makes Avro work in a key or value.
- sitkack 8y agoLast time I dived into details, the communication protocol for HDFS was serialized java objects using the built in serialization mechanism. Edit, they switched to Protocol Buffers in 0.23 https://wiki.apache.org/hadoop/HadoopRpc https://wiki.apache.org/hadoop/HadoopRpc
- slaymaker1907 8y agoSpark uses its own serialization system which while similar to the built in serialization, is designed to give much better performance.
- Scea91 8y agoOnly partly. Spark's RDD API which is still used quite heavily requires external serializer. Most of the time you would use Kryo, but it does not fork for objects larger than 2 GB or sometimes for custom classes that are not explicitly registered as Kryo serializable. In these cases, the changes would break existing code. However, who knows when will Oracle decide to remove the Java serialization API. I expect it will take a few years and the situation will be different on the Spark side then.
- scarface74 8y agoI was trying to understand if C# has the same issue. From what I can tell, as long as you use the default serialization, it seems to be safe. But I can't really tell. https://www.alphabot.com/security/blog/2017/net/How-to-configure-Json.NET-to-create-a-vulnerable-web-API.html https://www.alphabot.com/security/blog/2017/net/How-to-confi...
- dwaite 8y agoI'm rusty on C#, but I believe the equivalent would probably be [Serializable] types being read from a Stream using a BinaryFormatter or SoapFormatter. A malicious stream could include any known types in the system marked as [Serializable], and as they are deserialized any associated static constructor/no argument constructor/property setters could be called. In the JSON case linked, I presume there is a root type given when you are attempting to deserialize a document. However, if one of the properties of that type is ambiguous (say System.Object), and the deserialization algorithm looks for a 'type' property in the JSON with a class name to determine what is instantiated, then there can be all sorts of unintentional types that might be built by the processing of that malicious JSON.
- Const-me 8y agoThey moved away from that. Many newer parts of the .NET framework (e.g. SOAP 1.2 implementation in WCF) use data contract serializers by default. With them, a complete list of known types must be provided to deserializer, it's typically done with [KnownType] attribute.
- TheCoelacanth 8y agoThe biggest cause of vulnerabilities in Java serialization is that the class is part of the serialization format, so an attacker can cause the serialization to produce classes that you aren't expecting. Json.NET seems to allow the same behavior, but has it disabled by default. > In fact the only kind that is not vulnerable is the default: TypeNameHandling.None
- xenadu02 8y agoThis is the entire point of NSSecureCoding as well. You must provide the list of allowed types and any unexpected ones trigger errors.
- ddtaylor 8y agoIn theory this can be solved by using a security policy, but I don't think most are doing anything like that for deserialization.
- mbfg 8y agoall web applications use serialization to do session replication. Now this is safe as the both stream ends are controlled by the developer, and only if they are stupid, (instead of malicious) will there problems. Still if java serialization goes away, this will be the most widespread impacting change in the java ecosphere wrt changing how serialization works.
- koolba 8y agoUsing java serialization for sessions is terrible idea. It’s mildy convenient to start but bites you as soon as you want anything non-java to be able to read your session store.
- rad_gruchalski 8y agoThere are so many great alternatives available. It's a no-brainer. I applaud. [edit] so why the downvote? We have YAML, JSON, protobuf, Thrift, Avro. Yup, these serialize "contents" rather than "structure + contents" but one gets interop with other technologies for free. Every tech mentioned above is so simple to use that removing Java serialization is a no-brainer.
- spockz 8y agoDon’t forget Kryo[1]. Also Avrò messages can contain the schema. [1]: https://github.com/EsotericSoftware/kryo https://github.com/EsotericSoftware/kryo
- rad_gruchalski 8y agoYes! But Kryo docs explicitly mentions some things may have to be handled via Java serialization.
- kibibu 8y agoI suggest you look at the myriad security problems caused by Ruby's handling of YAML and JSON. It's not necessarily the serialization format at fault, but rather developer assumptions.
- willvarfar 8y ago(Here's a post describing some of the problems back in the day with Ruby's YAML and JSON: https://williamedwardscoder.tumblr.com/post/43394068341/rubys-principle-of-too-much-power https://williamedwardscoder.tumblr.com/post/43394068341/ruby... YAML is of course a format that is designed to instantiate objects as described in the source, rather than as checked by the destination, so I wouldn't want the world to adopt YAML instead of Java serialization.)
- imtringued 8y agoYaml is primarily used as hierarchical config format because JSON and XML are too verbose for a human editor.
- needusername 8y agoDoes anybody have a source for this? The publication doesn't doesn't exactly have the best reputation. A lot of things are currently built on top of serialization, from JMX to almost everything in Java EE including servlet sessions.
- boobsbr 8y agoWould this affect network class loading?
- gaius 8y agoOracle has received many reports are received about application servers running on the network with unprotected ports taking serialization streams I’m just not seeing how this is a problem with the language. Leave anything on an unprotected port taking unsanitised input and it’s vulnerable no matter what it is written in.
- hurrrrr 8y agoThe problem is that the API pretends that it already sanitizes user input to a degree. (from the design perspective, not necessarily documentation) Sanitizing the input yourself means not using it. You can't sanitize it.
- wepple 8y ago> Leave anything on an unprotected port taking unsanitised input and it’s vulnerable no matter what it is written in I don’t think you’re comparing apples with apples. Even in a default configuration, the majority of services that show up on a network don’t give you instant code exec. Maybe tomcat servers with default passwords, and a few other things. But (de)serialization isn’t securable at all. You can’t add auth, you can’t WAF it, you can’t fix the underlying vulnerabilities. There is absolutely nothing which has that level of vulnerability and lack of security on a modern network.
- gaius 8y agoThere is absolutely nothing which has that level of vulnerability and lack of security on a modern network. OK, let's say I overwrite part of your system with something evil, knowing that your app will Class.forName() it. Is that a problem with the ClassLoader or is the problem that your perimeter is already compromised anyway?
- wepple 8y agoI don’t think it matters, at all. Java has serialization, and everyone uses it, and it’s a security nightmare. As uncomfortable as it may feel to me, I kinda agree with Oracles approach here: kill serialization, make things more secure. If they have to change core language/libraries, so be it. You were trying to say that anything left on a network socket will be compromised, and that’s simply not true. Most software is pretty solid. Anything in C will have more than it’s fair share of upcoming patches, but unlike Java serialization, you can’t fix a single lib to fix 99% of bugs.. or we would’ve surely done that.
- lolive 8y agoI really hope they choose something like RDF. Which is the only serialization format I know of that handles graph and types natively.
- darkhorn 8y agoUse Lisp, data and code is same thing.
- xg15 8y agoI applaud the general decision, but I wonder what will happen to existing blobs of serialized data. Will there be any migration tools provided? Apart from the horrible security, what annoyed me the most with serialization is the lack of control you have over the process. There doesn't seem to be a way to access serialized data as a simple parse tree or record sequence - you have to construct objects of the actual classes. If only one class is not available or has breaking changes, there is no (built-in) way to access anything inside the blob. This is particularly fun if you want to refactor things. Suddenly package names, class names and names of private fields (!) are part of your public interface. So if we could drop reflection/unsafe-based serialization and instead just got a simple parser/writer for java's binary object graph format, I'd be very happy.
- dnomad 8y agoThere's really no such thing as a "binary object graph format." Objects are not data. Code is not data. It's why serializing objects is so very complicated and dangerous. Serialized "data" is a program that can do everything a normal Java program can do. In the 90s we called this feature "mobile code" and thought it might be the future of distributed computing. Today we call it a "code injection attack" and recognize how enormously dangerous it is. If you want data then define a schema and write/read your data. Serialization should really never be used and certainly not for long-lived data storage. There is one place where serialization becomes useful and that is for storing objects off-heap, IPC, and for short-term storage (ie snapshots like Android's parcelable). For these cases I'd like to see the JVM embrace not just immutable value objects but full-fledged structs that have a well-defined memory layout. You can do this today using off-heap Buffers and interfaces but language support is always good so there's a universal standard that everybody can build upon. Once that's in place there'd be no need to ever use serialization. That said I can't imagine Oracle will simply remove support object serialization. It may be kicked out of the "core" JDK and become an optional module. The classes may be deprecated. But the functionality likely isn't going anywhere in the next ten years. Even if they did remove it nobody should be using the standard object serialization anyways. If you're going to use serialization (and you shouldn't) then you should absolutely be using FST [0]. [0] https://github.com/RuedigerMoeller/fast-serialization https://github.com/RuedigerMoeller/fast-serialization
- jdeca568 8y agoIf i understand correctly, the problem of Java deserialization is 1) gadgets for code execution and maybe 2) DoS. And this is bad because even if it looks secure now, a gadget could be discovered later. The default Java serialization is one of the easiest way to serialize instance of objects but there are many other ways, and many other risky ways among them. It seems to be always the same problem: ClassLoader access. Couldn't there be a way to let the deserializers use a specific ClassLoader? I mean some sort of (Sandboxed)ObjectInputStream that uses a specific ClassLoader defined in the JRE config. The sandboxed contexts could be defined in something like java.security, .policy, to define what it is supposed to know and when/where it is supposed to be used.
- eternalban 8y ago> Serialization was a “horrible mistake” RMI required it. Mr. Reinhold has apparently forgotten the scene in late 90s. CORBA anyone?